Skip to content

Releases: wells-wood-research/timed-design

Models 01-2024

31 Jan 15:20
05d08a1
Compare
Choose a tag to compare

Models for the paper "TIMED-Design: Flexible and Accessible Protein Sequence Design with Convolutional Neural Networks"

Performance Comparison

For detailed performance comparisons, please see the paper.

Macro Recall

Macro-Recall is accuracy averaged per residue - resistant to class imbalance.
boxplot_recall_chargepolar_all

RMSD_100

We sampled 10% of the PDBench dataset and ran it through AlphaFold2 + Amber relaxation. RMSD_100 is a normalised version of RMSD.
boxplot_RMSD_Norm_polarcharge_all

Isoelectric Point Mean Absolute Error (MAE)

Difference between the isoelectric point of the original sequence and the predicted sequence.
boxplot_iso_chargepolar_all

Charge Mean Absolute Error (MAE)

Difference between the overall charge of the original sequence and the predicted sequence.
boxplot_charge_chargepolar_all

Training

All models were trained using the culled PDB set from PISCES cullpdb_pc90_res3.0_R1.0_d200702_chains40583containing over 35K non-redundant protein structures (40K+ chains), with resolutions up to 3.0 Å.

CNN Models

We reimplemented all of the CNN models in the literature as they were all closed-source. The dataset for CNN models was created using aposteriori using the following command:

make-frame-dataset /scratch/datasets/biounit/ -d benchmarking_set.csv -e .pdb1.gz --voxels-per-side 21 --frame-edge-length 21 -g True -p 35 -n benchmark_set -v -r -z -cb True -ae CNOCBCA  --compression_gzip True -o /scratch/timed_dataset/

For Charge and Polar models we used the codecs (-ae) equivalent to CNOCBCAQ and CNOCBCAP, respectively.

GNN Models

The code for training ProteinMPNN with custom training sets is not available. We recreated the steps given to us by the authors and published them here: https://github.com/wells-wood-research/ProteinMPNN_custom_training/tree/main

What's Changed

New Contributors

Full Changelog: modelspublication...publication_01_2024

Full Changelog: publication_01_2024...publication_01_2024

Models 12-2023

14 Dec 13:54
abc6afa
Compare
Choose a tag to compare

Models for the paper "TIMED-Design: Flexible and Accessible Protein Sequence Design with Convolutional Neural Networks"

Performance Comparison

For detailed performance comparisons, please see the paper.

Macro Recall

Macro-Recall is accuracy averaged per residue - resistant to class imbalance.
boxplot_recall_chargepolar_all

RMSD_100

We sampled 10% of the PDBench dataset and ran it through AlphaFold2 + Amber relaxation. RMSD_100 is a normalised version of RMSD.
boxplot_RMSD_Norm_polarcharge_all

Isoelectric Point Mean Absolute Error (MAE)

Difference between the isoelectric point of the original sequence and the predicted sequence.
boxplot_iso_chargepolar_all

Charge Mean Absolute Error (MAE)

Difference between the overall charge of the original sequence and the predicted sequence.
boxplot_charge_chargepolar_all

Training

All models were trained using the culled PDB set from PISCES cullpdb_pc90_res3.0_R1.0_d200702_chains40583containing over 35K non-redundant protein structures (40K+ chains), with resolutions up to 3.0 Å.

CNN Models

We reimplemented all of the CNN models in the literature as they were all closed-source. The dataset for CNN models was created using aposteriori using the following command:

poetry run make-frame-dataset /scratch/datasets/biounit/ -d benchmarking_set.csv -e .pdb1.gz --voxels-per-side 21 --frame-edge-length 21 -g True -p 35 -n benchmark_set -v -r -z -cb True -ae CNOCBCA  --compression_gzip True -o /scratch/timed_dataset/

For Charge and Polar models we used the codecs (-ae) equivalent to CNOCBCAQ and CNOCBCAP, respectively.

GNN Models

The code for training ProteinMPNN with custom training sets is not available. We recreated the steps given to us by the authors and published them here: https://github.com/wells-wood-research/ProteinMPNN_custom_training/tree/main

Models 03-2023

23 Mar 17:24
574d2a3
Compare
Choose a tag to compare

All models were trained using the following dataset settings from aposteriori

poetry run make-frame-dataset /scratch/datasets/biounit/ -d benchmarking_set.csv -e .pdb1.gz --voxels-per-side 21 --frame-edge-length 21 -g True -p 35 -n benchmark_set -v -r -z -cb True -ae CNOCBCA --compression_gzip True -o /scratch/timed_dataset/

We retrained all models with the same dataset and tested on the PDBench benchmark.

Sequence Metrics

Accuracy

accuracy
legend_small

Macro-Recall

Macro-Recall is accuracy averaged per residue - resistant to class imbalance.
macro-recall
legend_small

Charge Mean Absolute Error (MAE)

Difference between the charge of the original sequence and the predicted sequence.
charge
legend_small

Isoelectric Point Mean Absolute Error (MAE)

Difference between the isoelectric point of the original sequence and the predicted sequence.
isoelectric
legend_small

3D Structure Metrics

RMSD

We sampled 10% of the dataset and ran it through AlphaFold2 + Amber relaxation
download
legend_small

TIMED-design pre-release

16 Aug 08:49
a94468e
Compare
Choose a tag to compare
Pre-release

All models except timed use the following:

poetry run make-frame-dataset /scratch/datasets/biounit/ -d benchmarking_set.csv -e .pdb1.gz --voxels-per-side 21 --frame-edge-length 21 -g True -p 35 -n benchmark_set -v -r -z -cb True -ae CNOCBCA  --compression_gzip True -o /scratch/timed_dataset/

Models 2022-04

07 Apr 07:38
Compare
Choose a tag to compare

All models except timed use the following:

poetry run make-frame-dataset /scratch/datasets/biounit/ -d benchmarking_set.csv -e .pdb1.gz --voxels-per-side 21 --frame-edge-length 21 -g True -p 35 -n benchmark_set -v -r -z -cb True -ae CNOCBCA  --compression_gzip True -o /scratch/timed_dataset/

TIMED uses the following settings:

poetry run make-frame-dataset ../../shared/datasets/biounit/ -e .pdb1.gz --pieces-filter-file /home/shared/datasets/pisces/cullpdb_pc90_res3.0_R1.0_d200702_chains40583 --voxels-per-side 21 --frame-edge-length 13 -g False -p 35 -n pisces_expanded -v -r -z -cb True -ae CNOCBCA -b blacklist.csv

Screenshot 2022-04-07 at 09 55 05

Screenshot 2022-04-07 at 09 56 58