Background: Protein inter-residue contact and distance prediction are two key intermediate steps essential to accurate protein structure prediction. Distance prediction comes in two forms: real-valued distances and 'binned' distograms, which are a more finely grained variant of the binary contact prediction problem. The latter has been introduced as a new challenge in the 14th Critical Assessment of Techniques for Protein Structure Prediction (CASP14) 2020 experiment. Despite the recent proliferation of methods for predicting distances, few methods exist for evaluating these predictions. Currently only numerical metrics, which evaluate the entire prediction at once, are used. These give no insight into the structural details of a prediction. For this reason, new methods and tools are needed.
Results: We have developed a web server for evaluating predicted inter-residue distances. Our server, DISTEVAL, accepts predicted contacts, distances, and a true structure as optional inputs to generate informative heatmaps, chord diagrams, and 3D models. All of these outputs facilitate visual and qualitative assessment. The server also evaluates predictions using other metrics such as mean absolute error, root mean squared error, and contact precision.
Conclusions: The visualizations generated by DISTEVAL complement each other and collectively serve as a powerful tool for both quantitative and qualitative assessments of predicted contacts and distances, even in the absence of a true 3D structure.
http://deep.cs.umsl.edu/disteval/
Download from https://github.com/ba-lab/disteval/releases
- Python3
- Numpy
- Scikit-learn
python3 ./disteval.py -hpython3 ./disteval.py -n ./test/1guuA.pdb -c ./test/1guuA.contact.rrExpected output:
Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 1.0, 'count': 9}
min-seq-sep: 12 xL: Top-L {'precision': 1.0, 'count': 9}
min-seq-sep: 12 xL: Top-NC {'precision': 1.0, 'count': 9}
min-seq-sep: 24 xL: Top-L/5 {'precision': 1.0, 'count': 1}
min-seq-sep: 24 xL: Top-L {'precision': 1.0, 'count': 1}
min-seq-sep: 24 xL: Top-NC {'precision': 1.0, 'count': 1}
python3 ./disteval.py -n ./test/1guuA.pdb -d ./test/1guuA.predicted.npyExpected output:
Evaluating distances..
min-seq-sep: 12 xL: Top-L/5 {'mae': 0.9403, 'mse': 1.5143, 'rmse': 1.2306, 'count': 10}
min-seq-sep: 12 xL: Top-L {'mae': 1.7522, 'mse': 5.6841, 'rmse': 2.3841, 'count': 50}
min-seq-sep: 12 xL: Top-NC {'mae': 1.9263, 'mse': 6.6872, 'rmse': 2.586, 'count': 603}
min-seq-sep: 24 xL: Top-L/5 {'mae': 1.8154, 'mse': 4.6469, 'rmse': 2.1557, 'count': 10}
min-seq-sep: 24 xL: Top-L {'mae': 2.1541, 'mse': 8.1816, 'rmse': 2.8603, 'count': 50}
min-seq-sep: 24 xL: Top-NC {'mae': 2.4536, 'mse': 9.6231, 'rmse': 3.1021, 'count': 295}
Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 0.9, 'count': 10}
min-seq-sep: 12 xL: Top-L {'precision': 0.6, 'count': 30}
min-seq-sep: 12 xL: Top-NC {'precision': 0.6, 'count': 30}
min-seq-sep: 24 xL: Top-L/5 {'precision': 0.5, 'count': 10}
min-seq-sep: 24 xL: Top-L {'precision': 0.38462, 'count': 13}
min-seq-sep: 24 xL: Top-NC {'precision': 0.38462, 'count': 13}
python3 ./disteval.py -n ./test/1guuA.pdb -r ./test/1guuA.npz Expected output:
Evaluating distances..
min-seq-sep: 12 xL: Top-L/5 {'mae': 0.5485, 'mse': 0.5375, 'rmse': 0.7331, 'count': 10}
min-seq-sep: 12 xL: Top-L {'mae': 0.6789, 'mse': 0.7678, 'rmse': 0.8762, 'count': 50}
min-seq-sep: 12 xL: Top-NC {'mae': 1.2951, 'mse': 3.8733, 'rmse': 1.9681, 'count': 741}
min-seq-sep: 24 xL: Top-L/5 {'mae': 0.537, 'mse': 0.4237, 'rmse': 0.6509, 'count': 10}
min-seq-sep: 24 xL: Top-L {'mae': 0.6691, 'mse': 0.6725, 'rmse': 0.8201, 'count': 50}
min-seq-sep: 24 xL: Top-NC {'mae': 1.2281, 'mse': 3.2863, 'rmse': 1.8128, 'count': 351}
Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 1.0, 'count': 10}
min-seq-sep: 12 xL: Top-L {'precision': 0.8, 'count': 30}
min-seq-sep: 12 xL: Top-NC {'precision': 0.8, 'count': 30}
min-seq-sep: 24 xL: Top-L/5 {'precision': 1.0, 'count': 10}
min-seq-sep: 24 xL: Top-L {'precision': 0.84615, 'count': 13}
min-seq-sep: 24 xL: Top-NC {'precision': 0.84615, 'count': 13}
wget http://deep.cs.umsl.edu/disteval/static/data/casp14/T1024/RaptorX_RR1
wget http://deep.cs.umsl.edu/disteval/static/data/casp14/casp14_pdbs/T1024.pdb
python3 ./disteval.py -n ./T1024.pdb -c ./RaptorX_RR1Expected output:
Evaluating distances..
min-seq-sep: 12 xL: Top-L/5 {'mae': 1.7837, 'mse': 4.9053, 'rmse': 2.2148, 'count': 78}
min-seq-sep: 12 xL: Top-L {'mae': 2.4797, 'mse': 13.0069, 'rmse': 3.6065, 'count': 392}
min-seq-sep: 12 xL: Top-NC {'mae': 3.6061, 'mse': 16.4059, 'rmse': 4.0504, 'count': 5459}
min-seq-sep: 24 xL: Top-L/5 {'mae': 1.7837, 'mse': 4.9053, 'rmse': 2.2148, 'count': 78}
min-seq-sep: 24 xL: Top-L {'mae': 2.4398, 'mse': 12.8404, 'rmse': 3.5834, 'count': 392}
min-seq-sep: 24 xL: Top-NC {'mae': 3.6114, 'mse': 16.4634, 'rmse': 4.0575, 'count': 4906}
Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 0.9359, 'count': 78}
min-seq-sep: 12 xL: Top-L {'precision': 0.82143, 'count': 392}
min-seq-sep: 12 xL: Top-NC {'precision': 0.68562, 'count': 633}
min-seq-sep: 24 xL: Top-L/5 {'precision': 0.9359, 'count': 78}
min-seq-sep: 24 xL: Top-L {'precision': 0.80357, 'count': 392}
min-seq-sep: 24 xL: Top-NC {'precision': 0.68631, 'count': 577}
- Install csh
sudo apt install csh
- Download 'dssp-2.0.4-linux-amd64' from https://osf.io/qydjv/
chmod +x dssp-2.0.4-linux-amd64
- Download TM-score from https://zhanglab.ccmb.med.umich.edu/TM-score/TMscore.gz
wget https://zhanglab.ccmb.med.umich.edu/TM-score/TMscore.gz gunzip TMscore.gz chmod +x TMscore
- DISTFOLD
- Follow instructions here to download DISTFOLD, an updated version of CONFOLD.
python3 disteval.py -f ./test/1guuA.fasta -n ./test/1guuA.pdb -c ./test/1guuA.contact.rr -s ./test/1guuA.ss -o ./build-1guuA -b
Expected output:
TM-score RMSD GDT-TS MODEL
0.297 10.100 0.385 1guuA_11.pdb
0.320 7.729 0.460 1guuA_8.pdb
...
0.465 3.935 0.630 1guuA_model1.pdb
0.483 5.776 0.600 1guuA_model2.pdb
0.550 4.534 0.665 1guuA_5.pdb
python3 disteval.py -f ./test/1guuA.fasta -n ./test/1guuA.pdb -d ./test/1guuA.predicted.npy -s ./test/1guuA.ss -o ./build-1guuA -b -m 6 -t 12
Expected output:
TM-score RMSD GDT-TS MODEL
0.107 37.610 0.155 extended.pdb
0.630 3.016 0.745 1guuA_11.pdb
...
0.681 2.528 0.785 1guuA_6.pdb
0.681 2.489 0.790 1guuA_9.pdb
python3 disteval.py -f ./test/1guuA.fasta -n ./test/1guuA.pdb -d ./test/1guuA.predicted.npy -s ./test/1guuA.ss -o ./build-1guuA -b -m 2 -t 12
Expected output:
TM-score RMSD GDT-TS MODEL
0.107 37.610 0.155 extended.pdb
0.253 10.230 0.340 1guuA_11.pdb
...
0.681 3.349 0.775 1guuA_13.pdb
0.684 2.330 0.795 1guuA_3.pdb
python3 disteval.py -f ./test/1guuA.fasta -n ./test/1guuA.pdb -o ./build-1guuA -p -b -m 2 -t 18
Expected output:
TM-score RMSD GDT-TS MODEL
0.107 37.610 0.155 extended.pdb
...
0.987 0.265 1.000 1guuA_model2.pdb
0.991 0.214 1.000 1guuA_16.pdb
python3 disteval.py -f ./test/1guuA.fasta -n ./test/1guuA.pdb -r ./test/1guuA.npz -o ./build-1guuA -b -m 2 -t 12
Expected output:
TM-score RMSD GDT-TS MODEL
0.107 37.610 0.155 extended.pdb
0.268 9.724 0.375 1guuA_14.pdb
...
0.876 0.979 0.940 1guuA_model1.pdb
0.880 1.151 0.950 1guuA_16.pdb
2021, "DISTEVAL: a web server for evaluating predicted protein distances", BMC Bioinformatics