[论文解读] DPLM-2: A Multimodal Diffusion Protein Language Model

Xinyou Wang, Zaixiang Zheng+4 more|arXiv (Cornell University)|Oct 17, 2024

Machine Learning in Bioinformatics被引用 6

一句话总结

DPLM-2 将离散扩散蛋白质语言模型扩展为同时对蛋白质序列和结构进行建模与生成，使用无查找表的结构分词器和多模态训练目标，实现无条件的共同生成以及各种有条件任务。

ABSTRACT

Proteins are essential macromolecules defined by their amino acid sequences, which determine their three-dimensional structures and, consequently, their functions in all living organisms. Therefore, generative protein modeling necessitates a multimodal approach to simultaneously model, understand, and generate both sequences and structures. However, existing methods typically use separate models for each modality, limiting their ability to capture the intricate relationships between sequence and structure. This results in suboptimal performance in tasks that requires joint understanding and generation of both modalities. In this paper, we introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. To enable structural learning with the language model, 3D coordinates are converted to discrete tokens using a lookup-free quantization-based tokenizer. By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. We also implement an efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence-based protein language models. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures eliminating the need for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive performance in various conditional generation tasks, including folding, inverse folding, and scaffolding with multimodal motif inputs, as well as providing structure-aware representations for predictive tasks.

研究动机与目标

• 阐明并解决蛋白质序列与结构统一建模的需求。
• 开发一个学习序列与结构联合分布的多模态蛋白质基础模型。
• 利用结构分词器将三维坐标转换为离散令牌，以用于语言模型学习。
• 以预训练的基于序列的知识进行热身，以增强结构学习。
• 展示带有结构感知表示的无条件共同生成以及多种有条件生成任务。

提出的方法

• 在一个统一框架中扩展离散扩散蛋白质语言模型（DPLM），以处理序列和结构。
• 引入一个无查找表的量化器（LFQ），将三维骨架坐标令牌化为离散的结构令牌。
• 将结构令牌与氨基酸序列拼接，使残基级别的位置与共享编码对齐。
• 应用模态特定的噪声计划和自混合训练策略，以缓解序列扩散中的暴露偏差。
• 使用 LoRA 从预训练的基于序列的 DPLM 高效热身，以在保留预训练参数的同时转移进化知识。

实验结果

研究问题

• RQ1 一个单一的多模态扩散模型是否能够以高保真度同时对蛋白质序列和结构进行建模与生成？
• RQ2 如何在语言模型框架中有效地学习结构信息？
• RQ3 多模态条件对折叠、逆折叠和基序支架任务的好处是什么？
• RQ4 在序列数据上的预训练与数据增强是否能提升多模态生成及多样性？

主要发现

• DPLM-2 实现了兼容的蛋白质序列和三维结构的同时生成，而无需两阶段级联。
• 在实验结构和 AlphaFold 预测结构上训练的模型学习序列和结构的联合、边缘和条件分布。
• DPLM-2 在带有多模态输入的折叠、逆折叠和基序支架任务中展示了具有竞争力的性能。
• 来自 DPLM-2 的结构感知表示提升了预测任务的表现，不仅仅是生成。
• 从预训练的基于序列的 DPLM 开始热身并结合数据增强，显著提高设计性和多样性，尤其是对于较长的蛋白质。