[Paper Review] Inverse Protein Folding Using Deep Bayesian Optimization

Natalie Maus, Yimeng Zeng+6 more|arXiv (Cornell University)|May 25, 2023

Protein Structure and DynamicsBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes BO-IF, a deep Bayesian optimization framework for inverse protein folding that iteratively refines protein sequences to better match target backbone structures. By leveraging a small 47M-parameter transformer and latent-space Bayesian optimization, it reduces structural error by 48% in TM score and 28% in RMSD compared to state-of-the-art generative models like ESM-IF, while enabling constraint handling and diverse sequence generation.

ABSTRACT

Inverse protein folding -- the task of predicting a protein sequence from its backbone atom coordinates -- has surfaced as an important problem in the "top down", de novo design of proteins. Contemporary approaches have cast this problem as a conditional generative modelling problem, where a large generative model over protein sequences is conditioned on the backbone. While these generative models very rapidly produce promising sequences, independent draws from generative models may fail to produce sequences that reliably fold to the correct backbone. Furthermore, it is challenging to adapt pure generative approaches to other settings, e.g., when constraints exist. In this paper, we cast the problem of improving generated inverse folds as an optimization problem that we solve using recent advances in "deep" or "latent space" Bayesian optimization. Our approach consistently produces protein sequences with greatly reduced structural error to the target backbone structure as measured by TM score and RMSD while using fewer computational resources. Additionally, we demonstrate other advantages of an optimization-based approach to the problem, such as the ability to handle constraints.

Motivation & Objective

• To address the limitations of one-shot generative models in inverse protein folding, which often fail to produce sequences that reliably fold to target structures.
• To develop an optimization-based approach that enables iterative refinement of protein sequences for higher structural accuracy.
• To incorporate practical constraints such as sequence humanness and structural diversity into inverse folding.
• To reduce computational cost by using a small, efficient language model in conjunction with Bayesian optimization.

Proposed method

• The method frames inverse folding as a black-box optimization problem, where the objective function measures structural similarity (via TM score or RMSD) between a generated sequence’s predicted fold and the target backbone.
• It employs latent-space Bayesian optimization (BO) to efficiently search the protein sequence space, using a small 47M-parameter transformer model to generate initial sequences.
• The optimization pipeline uses acquisition functions to select promising sequences for iterative refinement, reducing the number of costly folding evaluations.
• The approach is extended to handle black-box constraints (e.g., minimum humanness score of 80%) using constrained Bayesian optimization.
• It supports diverse sequence generation by integrating ROBOT, a method for optimizing under minimum edit distance constraints.
• The entire pipeline is implemented using standard libraries like BoTorch and is publicly available at https://github.com/nataliemaus/bo-if.

Figure 1: Different target backbones (blue) inverse folded by ESM-IF (yellow) and Bayesian optimization (pink). Our method, BO-IF , consistently finds proteins that better match the target structure as evidenced visually by the better alignment and by the higher TM-scores achieved. Arrows indicate e

Experimental results

Research questions

• RQ1 Can Bayesian optimization outperform one-shot generative models in producing protein sequences that fold to a target backbone structure with higher accuracy?
• RQ2 How does the optimization-based approach compare to generative models in terms of structural error (TM score and RMSD) and computational efficiency?
• RQ3 Can the optimization framework effectively incorporate biological constraints such as sequence humanness to reduce immunogenicity risk in therapeutic proteins?
• RQ4 To what extent can the method generate diverse protein sequences that maintain target structural similarity while differing significantly in sequence?
• RQ5 How does the method scale in terms of computational cost when optimizing under constraints or for diverse sets of sequences?

Key findings

• BO-IF reduces average structural error by 48.364% (±9.480%) in TM score and 34.454% (±7.082%) in RMSD compared to ESM-IF when optimizing under a humanness constraint of at least 80%.
• The method achieves comparable end-to-end computational cost (around 50 GPU hours) to parallel generation despite using a much smaller model, due to efficient sequential optimization.
• Only 0.2% of sequences generated by ESM-IF satisfied both the humanness threshold and a TM score > 0.8, while BO-IF significantly improved this success rate under the same constraints.
• For diverse sequence generation with pairwise edit distance ≥20, BO-IF reduced average structural error by 48.640% (±8.003%) in TM score and 33.617% (±5.520%) in RMSD compared to ESM-IF.
• The optimization pipeline successfully generated diverse sets of 5 and 10 sequences per target structure, with consistent improvements in structural accuracy across all targets.
• The method demonstrates strong generalization to complex design objectives, including constraint handling and multi-objective optimization, while maintaining low computational overhead.

Figure 2: Error in backbone structures computationally folded from inverse folds as measured by 1-TM score (computed by TM-align ) across 24 target protein backbones. On average, optimization reduces structural error by 48% (standard error $\pm$ 0.69%). This corresponds to RMSD reduction of 28% (sta