Contact prediction task specifies 40 CASP12 test proteins, but the current config downloads 34 CASP14 proteins

[adpartin]

Issue

Table 1 of the PEER paper reports the following ProteinNet split sizes for the contact prediction task:

train 25,299 / valid 224 / test 40

Section 3.3 in the paper states that PEER follows TAPE and evaluates contact prediction on the ProteinNet CASP12 test set. However, a run with the current public config and a clean dataset cache downloads a different test split containing 34 CASP14 proteins.

Why the current config downloads 34

For example, config/single_task/ESM/contact_ESM.yaml sets:

dataset:
  class: ProteinNet

The config resolves ProteinNet through TorchDrug's dataset registry. The current TorchDrug ProteinNet class documents 40 test proteins but pins the following archive:

url = "https://miladeepgraphlearningproteindata.s3.us-east-2.amazonaws.com/data/proteinnet.tar.gz"
md5 = "ab44ab201b1570c0171a2bba9eb4d389"

That archive contains:

train 25299
valid 224
test 34

The ProteinNet source file included in the PyPI 0.2.1 release is identical to the file currently on TorchDrug's master branch (checked 2026-09-06).

Comparison with TAPE

TAPE's download_data.sh downloads ProteinNet from:

http://s3.amazonaws.com/songlabdata/proteindata/data_pytorch/proteinnet.tar.gz

Fetching that file gives MD5:

85c055d47487df2725e422599d9465d9

Comparison of the current TorchDrug archive, the TAPE archive, and the PEER paper:

split	TorchDrug archive	TAPE archive	PEER paper	ID comparison
train	25,299	25,299	25,299	identical
valid	224	224	224	identical
test	34	40	40	0 shared IDs

The train and valid IDs match. The two test splits share no IDs, so this is not the same test set with 6 records removed.

Every TAPE test ID appears in the official CASP12 target list. Every TorchDrug test ID appears in the official CASP14 target list. The paper's documented count of 40 is therefore consistent with the CASP12 test set, while the current config downloads the 34-protein CASP14 test set.

Related

TorchDrug dataset issue: DeepGraphLearning/torchdrug#265

Suggested fix

Could you clarify which test split was used for the published results for the contact-prediction task?

If the 40-protein CASP12 split was used, please update the public reproduction path to download that split. If the 34-protein CASP14 split is intended, please update the repository documentation to identify the different test set and its size.

At present, a run using the public config and current TorchDrug dataset class does not evaluate on the test set described in the paper.