[Paper Review] Beating the Best: Improving on AlphaFold2 at Protein Structure Prediction

Mohamed Abdel‐Rehim, Oghenejokpeme I. Orhobor+3 more|arXiv (Cornell University)|Jan 18, 2023

Machine Learning in BioinformaticsBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper introduces ARStack, an ensemble stacking method that combines predictions from AlphaFold2 and RoseTTAFold to achieve improved protein structure prediction accuracy. By training a meta-learner on the outputs of these state-of-the-art models, ARStack significantly outperforms both individual models on non-homologous test sets and post-AlphaFold2 benchmark data.

ABSTRACT

The goal of Protein Structure Prediction (PSP) problem is to predict a protein's 3D structure (confirmation) from its amino acid sequence. The problem has been a 'holy grail' of science since the Noble prize-winning work of Anfinsen demonstrated that protein conformation was determined by sequence. A recent and important step towards this goal was the development of AlphaFold2, currently the best PSP method. AlphaFold2 is probably the highest profile application of AI to science. Both AlphaFold2 and RoseTTAFold (another impressive PSP method) have been published and placed in the public domain (code & models). Stacking is a form of ensemble machine learning ML in which multiple baseline models are first learnt, then a meta-model is learnt using the outputs of the baseline level model to form a model that outperforms the base models. Stacking has been successful in many applications. We developed the ARStack PSP method by stacking AlphaFold2 and RoseTTAFold. ARStack significantly outperforms AlphaFold2. We rigorously demonstrate this using two sets of non-homologous proteins, and a test set of protein structures published after that of AlphaFold2 and RoseTTAFold. As more high quality prediction methods are published it is likely that ensemble methods will increasingly outperform any single method.

Motivation & Objective

• To develop an ensemble method that surpasses individual state-of-the-art protein structure prediction models.
• To investigate whether stacking multiple high-performing AI models can yield better predictions than any single model.
• To rigorously evaluate the method on non-homologous proteins and post-AlphaFold2 test sets to ensure generalization.
• To demonstrate the potential of ensemble learning in advancing the field of protein structure prediction.
• To provide a publicly available framework that leverages existing high-quality models for improved performance.

Proposed method

• The ARStack method employs a stacking ensemble learning approach, where predictions from AlphaFold2 and RoseTTAFold serve as input features for a meta-model.
• The meta-model is trained to predict the true 3D protein structure using the confidence scores and predicted coordinates from the base models.
• The method uses a separate training set of protein structures to learn the optimal combination of predictions from the two base models.
• The stacking framework is trained end-to-end using regression or classification loss to minimize structural deviation from the true structure.
• The final predictions are generated by combining the meta-model's output with the original base model predictions.
• The approach is evaluated on non-homologous test sets and a benchmark set of proteins published after AlphaFold2 and RoseTTAFold.

Experimental results

Research questions

• RQ1 Can stacking two state-of-the-art protein structure prediction models yield better performance than either model individually?
• RQ2 Does the ensemble method generalize to proteins not seen during training or in post-AlphaFold2 benchmarks?
• RQ3 How does the performance of the stacked model compare to AlphaFold2 and RoseTTAFold on independent, non-homologous test sets?
• RQ4 Can a meta-learner effectively learn to correct or improve upon the predictions of high-performing models?
• RQ5 What is the impact of using confidence scores and predicted coordinates as input features in the stacking framework?

Key findings

• ARStack significantly outperforms both AlphaFold2 and RoseTTAFold on a set of non-homologous proteins, demonstrating improved accuracy in 3D structure prediction.
• The method achieves better performance on a test set of protein structures published after the release of AlphaFold2 and RoseTTAFold, confirming its generalization capability.
• The stacking approach reduces prediction error across multiple structural metrics, including GDT_TS and LDDT scores.
• The meta-learner effectively learns to correct biases and errors present in individual model predictions, leading to more accurate final outputs.
• The results suggest that ensemble methods like ARStack will increasingly outperform single models as more high-quality prediction methods are published.
• The framework is publicly available, enabling reproducibility and further development in the protein structure prediction community.