[Paper Review] DPLM-2: A Multimodal Diffusion Protein Language Model

Xinyou Wang, Zaixiang Zheng+4 more|arXiv (Cornell University)|Oct 17, 2024

Machine Learning in Bioinformatics6 citations

TL;DR

DPLM-2 extends a discrete diffusion protein language model to jointly model and generate both protein sequences and structures using a lookup-free structure tokenizer and a multimodal training objective, enabling unconditional co-generation and various conditional tasks.

ABSTRACT

Proteins are essential macromolecules defined by their amino acid sequences, which determine their three-dimensional structures and, consequently, their functions in all living organisms. Therefore, generative protein modeling necessitates a multimodal approach to simultaneously model, understand, and generate both sequences and structures. However, existing methods typically use separate models for each modality, limiting their ability to capture the intricate relationships between sequence and structure. This results in suboptimal performance in tasks that requires joint understanding and generation of both modalities. In this paper, we introduce DPLM-2, a multimodal protein foundation model that extends discrete diffusion protein language model (DPLM) to accommodate both sequences and structures. To enable structural learning with the language model, 3D coordinates are converted to discrete tokens using a lookup-free quantization-based tokenizer. By training on both experimental and high-quality synthetic structures, DPLM-2 learns the joint distribution of sequence and structure, as well as their marginals and conditionals. We also implement an efficient warm-up strategy to exploit the connection between large-scale evolutionary data and structural inductive biases from pre-trained sequence-based protein language models. Empirical evaluation shows that DPLM-2 can simultaneously generate highly compatible amino acid sequences and their corresponding 3D structures eliminating the need for a two-stage generation approach. Moreover, DPLM-2 demonstrates competitive performance in various conditional generation tasks, including folding, inverse folding, and scaffolding with multimodal motif inputs, as well as providing structure-aware representations for predictive tasks.

Motivation & Objective

• Motivate and address the need for unified modeling of protein sequences and structures.
• Develop a multimodal protein foundation model that learns the joint distribution of sequence and structure.
• Leverage a structure tokenizer to convert 3D coordinates into discrete tokens for language-model learning.
• Warm up with pre-trained sequence-based knowledge to enhance structure learning.
• Demonstrate unconditional co-generation and multiple conditional generation tasks with structure-aware representations.

Proposed method

• Extend discrete diffusion protein language model (DPLM) to handle sequences and structures in a unified framework.
• Introduce a lookup-free quantizer (LFQ) to tokenize 3D backbone coordinates into discrete structure tokens.
• Concatenate structure tokens with amino acid sequences, aligning residue-level positions with shared encodings.
• Apply modality-specific noise schedulers and a self-mixup training strategy to mitigate exposure bias in sequence diffusion.
• Implement an efficient warm-up from a pre-trained sequence-based DPLM using LoRA to transfer evolutionary knowledge while preserving pre-trained parameters.

Experimental results

Research questions

• RQ1 Can a single multimodal diffusion model jointly model and generate protein sequences and structures with high fidelity?
• RQ2 How can structure information be learned effectively within a language-model framework?
• RQ3 What are the benefits of multimodal conditioning for folding, inverse folding, and motif scaffolding tasks?
• RQ4 Does pre-training on sequence data and data augmentation improve multimodal generation and diversity?

Key findings

• DPLM-2 achieves simultaneous generation of compatible protein sequences and 3D structures without a two-stage cascade.
• The model trained on experimental and AlphaFold-predicted structures learns joint, marginal, and conditional distributions for sequence and structure.
• DPLM-2 demonstrates competitive performance on folding, inverse folding, and motif-scaffolding tasks with multimodal inputs.
• Structure-aware representations from DPLM-2 improve predictive tasks beyond generation alone.
• A warm-up from a pre-trained sequence-based DPLM plus data augmentation significantly improves designability and diversity, especially for longer proteins.