[Paper Review] Leveraging Multi-modal Representations to Predict Protein Melting Temperatures

Daiheng Zhang, Yan Zeng+2 more|arXiv (Cornell University)|Dec 5, 2024

Protein Structure and DynamicsBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This paper proposes ESM3-DTm, a multi-modal deep learning framework that leverages protein language models—specifically ESM3, ESM2, and OpenFold—combined with sequence and structural features to predict changes in protein melting temperature ($\Delta T_m$). By fine-tuning the ESM3 backbone on both sequence and structure inputs, the model achieves a new state-of-the-art Pearson correlation coefficient of 0.50 on the s571 test set, demonstrating the superiority of multimodal representations for $\Delta T_m$ prediction.

ABSTRACT

Accurately predicting protein melting temperature changes (Delta Tm) is fundamental for assessing protein stability and guiding protein engineering. Leveraging multi-modal protein representations has shown great promise in capturing the complex relationships among protein sequences, structures, and functions. In this study, we develop models based on powerful protein language models, including ESM-2, ESM-3 and AlphaFold, using various feature extraction methods to enhance prediction accuracy. By utilizing the ESM-3 model, we achieve a new state-of-the-art performance on the s571 test dataset, obtaining a Pearson correlation coefficient (PCC) of 0.50. Furthermore, we conduct a fair evaluation to compare the performance of different protein language models in the Delta Tm prediction task. Our results demonstrate that integrating multi-modal protein representations could advance the prediction of protein melting temperatures.

Motivation & Objective

• To improve the accuracy of predicting changes in protein melting temperature ($\Delta T_m$) using multi-modal protein representations.
• To evaluate the effectiveness of different protein language models—ESM2, ESM3, and OpenFold—in capturing sequence and structural features relevant to $\Delta T_m$.
• To investigate the impact of integrating structural information and fine-tuning strategies on prediction performance.
• To establish a fair benchmark for $\Delta T_m$ prediction by re-splitting the training data to prevent overfitting.
• To compare various regression head architectures for fusing wild-type and mutated protein representations.

Proposed method

• The model uses ESM3 as the primary backbone, jointly processing protein sequences and 3D structures to generate unified embeddings.
• For ESM3 and ESM2 backbones, the method extracts CLS embeddings for global context and mutated-position embeddings for local features.
• For OpenFold, the model uses outputs from the Evoformer and Structure Modules as global embeddings and mutated-position representations as local features.
• Two fusion strategies are applied: outer product and linear combination of wild-type and mutant embeddings to predict $\Delta T_m$.
• End-to-end fine-tuning is performed on all model parameters, with gradient clipping and a OneCycle learning rate schedule.
• An ensemble of the top-performing regression heads is used to improve final prediction robustness.

Experimental results

Research questions

• RQ1 Can multi-modal protein representations that integrate sequence and 3D structure significantly improve $\Delta T_m$ prediction accuracy?
• RQ2 How do different protein language models (ESM2, ESM3, OpenFold) compare in their ability to predict $\Delta T_m$?
• RQ3 Does fine-tuning the backbone network yield better performance than freezing it during training?
• RQ4 Which regression head architecture (e.g., outer product, linear combination) leads to the most accurate $\Delta T_m$ predictions?
• RQ5 What is the impact of data split strategy on model generalization and performance evaluation?

Key findings

• ESM3-DTm achieves a Pearson correlation coefficient (PCC) of 0.50 on the s571 test set, setting a new state-of-the-art for $\Delta T_m$ prediction.
• The model reduces mean absolute error (MAE) to 5.21 and root mean square error (RMSE) to 7.68, outperforming prior methods including GeoDTm-3D and GeoStab.
• Multimodal ESM3-DTm significantly outperforms sequence-only models, with a 0.01 increase in PCC and 0.10 reduction in MAE compared to ESM3-DTm (seq only).
• Fine-tuning the ESM2-650M backbone yields a PCC of 0.48, compared to 0.46 when the backbone is frozen, demonstrating the benefit of full parameter fine-tuning.
• The outer product and linear combination of mutated-position embeddings achieved the highest performance among regression heads, with PCCs of 0.41 and 0.40, respectively.
• OpenFold-DTm performed poorly (PCC=0.35), indicating that while OpenFold is strong for structure prediction, it is less effective for $\Delta T_m$ regression in this setup.