[논문 리뷰] Benchmarking AlphaFold3's protein-protein complex accuracy and machine learning prediction reliability for binding free energy changes upon mutation

JunJie Wee, Guo-Wei Wei|PubMed|2024. 06. 06.

Machine Learning in Bioinformatics참고 문헌 20인용 수 11

한 줄 요약

본 논문은 AlphaFold3가 예측한 단백질-단백질 복합체 구조를 SKEMPI 2.0 돌연변이와 비교 평가하고, AF3가 BFE 변화에 대해 Pearson Rp 0.86을 보이며 PDB 구조에 비해 RMSE가 8.6% 증가하고, 유연한 영역에서의 신뢰도 한계를 강조한다.

ABSTRACT

AlphaFold 3 (AF3), the latest version of protein structure prediction software, goes beyond its predecessors by predicting protein-protein complexes. It could revolutionize drug discovery and protein engineering, marking a major step towards comprehensive, automated protein structure prediction. However, independent validation of AF3's predictions is necessary. Evaluated using the SKEMPI 2.0 database which involves 317 protein-protein complexes and 8338 mutations, AF3 complex structures give rise to a very good Pearson correlation coefficient of 0.86 for predicting protein-protein binding free energy changes upon mutation, slightly less than the 0.88 achieved earlier with the Protein Data Bank (PDB) structures. Nonetheless, AF3 complex structures led to a 8.6% increase in the prediction RMSE compared to original PDB complex structures. Additionally, some of AF3's complex structures have large errors, which were not captured in its ipTM performance metric. Finally, it is found that AF3's complex structures are not reliable for intrinsically flexible regions or domains.

연구 동기 및 목표

• Assess AF3-predicted protein-protein complex accuracy against experimental references in SKEMPI 2.0.
• Evaluate AF3-based predictions of mutation-induced binding free energy (BFE) changes using topology-based MT-TopLap models.
• Compare AF3-based predictions to state-of-the-art methods on the S8338 dataset.
• Identify conditions under which AF3 predictions are more or less reliable, including structural misalignment and flexible regions.

제안 방법

• Use AlphaFold3 to generate 317 protein-protein complex predictions (S8338 subset of SKEMPI 2.0).
• Apply MT-TopLap, a topology-based deep learning model using persistent Laplacian features, to predict mutation-induced BFE changes from AF3 structures.
• Perform 10-fold cross-validation predicting BFE changes with MT-TopLap AF3 and compare Rp and RMSE to baselines (e.g., mCSM-PPI2, TopLapNetGPT, TopNetGBT, etc.).
• Analyze structural alignment via RMSD, ipTM, and pTM to assess alignment vs. standard AF3 confidence metrics.
• Investigate per-residue correlations between B-factors and per-residue RMSD to assess reliability across flexible regions.

Figure 1: A: The cartoon representation of ribonuclease inhibitor-angiogenin complex (PDB ID: 1A4Y). The ribonuclease inhibitor is shown in blue and the angiogenin is in green. The mutation spots of 1A4Y in the S8338 dataset are indicated in red. B: The structural alignment of 1A4Y with its AF3 pred

실험 결과

연구 질문

• RQ1 What is the predictive correlation (Rp) and RMSE when using AF3-predicted PPI complexes to estimate mutation-induced BFE changes?
• RQ2 How does AF3-based structure accuracy (RMSD, ipTM, pTM) relate to BFE prediction reliability?
• RQ3 Do AF3 predictions exhibit systematic weaknesses in intrinsically flexible regions or domains?
• RQ4 How does MT-TopLap perform on AF3-predicted structures relative to established PDB-based benchmarks?
• RQ5 What structural or topological factors best explain failures or discrepancies in AF3-based predictions?

주요 결과

• AF3-predicted PPI structures achieve Rp = 0.86 for BFE-change predictions on SKEMPI 2.0 S8338 mutations, slightly below the 0.88 achieved with PDB references.
• Using AF3 structures increases BFE-change RMSE by 8.6% compared to using original PDB complexes.
• A subset of AF3 complexes show large RMSD misalignments not captured by ipTM, indicating ipTM alone misses some structural errors.
• AF3 predictions are unreliable for intrinsically flexible regions or domains, as high B-factors correlate with higher residue RMSD.
• MT-TopLap AF3 achieves competitive Rp values versus several state-of-the-art topology-based methods in 10-fold CV, albeit with some performance gaps relative to PDB-based inputs.

Figure 2: A: Top 40 protein-protein complexes with poorest RMSD alignment scores. B: The ipTM scores for protein-protein complexes in A. C: The mean absolute prediction error for mutation-induced binding free energy changes of the protein-protein complexes in A. D: Top 40 protein-protein complexes w