[Paper Review] A Pareto-optimal compositional energy-based model for sampling and optimization of protein sequences

Nataša Tagasovska, Nathan C. Frey+10 more|arXiv (Cornell University)|Oct 19, 2022

vaccines and immunoinformatics approaches4 citations

TL;DR

This paper proposes a Pareto-compositional energy-based model (pcEBM) that enables sampling of protein sequences satisfying multiple conflicting objectives simultaneously by combining multi-objective optimization with compositional energy-based modeling. The method achieves faster convergence and superior coverage of the Pareto front compared to baselines, generating diverse, high-quality antibody sequences optimized for binding affinity, polyreactivity, and viscosity.

ABSTRACT

Deep generative models have emerged as a popular machine learning-based approach for inverse design problems in the life sciences. However, these problems often require sampling new designs that satisfy multiple properties of interest in addition to learning the data distribution. This multi-objective optimization becomes more challenging when properties are independent or orthogonal to each other. In this work, we propose a Pareto-compositional energy-based model (pcEBM), a framework that uses multiple gradient descent for sampling new designs that adhere to various constraints in optimizing distinct properties. We demonstrate its ability to learn non-convex Pareto fronts and generate sequences that simultaneously satisfy multiple desired properties across a series of real-world antibody design tasks.

Motivation & Objective

• To address the challenge of generating protein sequences that satisfy multiple, often conflicting, biological properties such as binding affinity, polyreactivity, and viscosity.
• To develop a generative model that can sample from the Pareto front of optimal trade-offs between these properties, rather than optimizing a single objective.
• To improve sampling efficiency and diversity in multi-objective protein sequence optimization, especially in cases where objectives are orthogonal or non-convex.
• To enable controllable and guided generation of novel antibody sequences that are both functional and developable, reducing failure risks in downstream development.
• To provide a generalizable, modular framework applicable to other molecular design problems involving multiple objectives.

Proposed method

• The pcEBM framework integrates multiple gradient descent steps from individual energy-based models (EBMs) for each objective into a single compositional sampling process.
• It uses a compositional energy function defined as the sum of individual property-specific EBMs, each trained to model a specific biological property of interest.
• The sampling process applies a multi-gradient update rule that simultaneously minimizes energy across all objectives, enabling convergence toward Pareto-optimal solutions.
• The method employs a local descent (LD) strategy to improve coverage of the Pareto front, avoiding premature convergence to suboptimal regions.
• A step size schedule is used to balance convergence speed and stability during sampling, with ablation studies confirming its impact on performance.
• The approach is evaluated using both synthetic and real-world antibody design tasks, including sequence improvement from initial 'seed' sequences with poor properties.

Experimental results

Research questions

• RQ1 Can a compositional energy-based model effectively sample protein sequences that are Pareto-optimal across multiple conflicting objectives such as binding affinity and developability?
• RQ2 How does the proposed pcEBM compare to standard multi-objective baselines in terms of convergence speed, Pareto front coverage, and sequence quality?
• RQ3 To what extent can pcEBM improve the properties of existing antibody sequences, particularly in reducing nonspecificity (BV score), when used as a refinement tool?
• RQ4 Does the integration of local descent (LD) in the sampling process lead to better exploration of the Pareto front compared to standard gradient descent or direct LD?
• RQ5 Can the model generalize across different types of protein design tasks and maintain performance under varying hyperparameters?

Key findings

• pcEBM achieves faster convergence than cEBM, as shown by energy score decay over sampling steps, with a 0.01 step size yielding stable and rapid improvement.
• The method produces a broader and more diverse Pareto front than MGD and cEBM baselines, with significantly improved hyper-volume and similarity to training data.
• pcEBM generates sequences with higher hyper-volume and lower variance in edit distance and objective scores across different step sizes, indicating robustness and stability.
• In improving existing antibody sequences, pcEBM and MGD reduced the average BV score from 0.27 to above 0.9, outperforming cEBM and demonstrating strong performance in reducing nonspecificity.
• t-SNE visualization confirms that pcEBM successfully moves sequences from low-BV (red) regions toward high-BV (blue) regions of the latent space, indicating effective optimization of nonspecificity.
• The use of local descent (LD) in pcEBM leads to better coverage of the Pareto front than direct LD, confirming the advantage of multi-gradient composition in exploration.