[论文解读] Pre-Training Protein Encoder via Siamese Sequence-Structure Diffusion Trajectory Prediction

Zuobai Zhang, Minghao Xu+4 more|arXiv (Cornell University)|Jan 28, 2023

Machine Learning in Bioinformatics被引用 11

一句话总结

本文提出 DiffPreT，通过对序列和结构的联合扩散来对蛋白质编码器进行预训练，SiamDiff 用于通过 siamese diffusion trajectories 捕捉构象相关性，在下游任务上的平均排序达到最先进的水平。

ABSTRACT

Self-supervised pre-training methods on proteins have recently gained attention, with most approaches focusing on either protein sequences or structures, neglecting the exploration of their joint distribution, which is crucial for a comprehensive understanding of protein functions by integrating co-evolutionary information and structural characteristics. In this work, inspired by the success of denoising diffusion models in generative tasks, we propose the DiffPreT approach to pre-train a protein encoder by sequence-structure joint diffusion modeling. DiffPreT guides the encoder to recover the native protein sequences and structures from the perturbed ones along the joint diffusion trajectory, which acquires the joint distribution of sequences and structures. Considering the essential protein conformational variations, we enhance DiffPreT by a method called Siamese Diffusion Trajectory Prediction (SiamDiff) to capture the correlation between different conformers of a protein. SiamDiff attains this goal by maximizing the mutual information between representations of diffusion trajectories of structurally-correlated conformers. We study the effectiveness of DiffPreT and SiamDiff on both atom- and residue-level structure-based protein understanding tasks. Experimental results show that the performance of DiffPreT is consistently competitive on all tasks, and SiamDiff achieves new state-of-the-art performance, considering the mean ranks on all tasks. Our implementation is available at https://github.com/DeepGraphLearning/SiamDiff.

研究动机与目标

• 使用扩散建模蛋白质序列与结构的联合分布，以学习有信息量的表征。
• 通过 Siamese 扩散轨迹预测将构象异质性纳入以捕捉构象体之间的相关性。
• 在原子级和残基级的结构任务上评估预训练编码器，以评估泛化性和鲁棒性。
• 证明所提出的方法在多样的下游任务上具备与现有预训练方法相竞争或优于之处。

提出的方法

• DiffPreT 对蛋白质序列和结构执行联合扩散，并训练一个由蛋白质编码器参数化的噪声预测网络。
• 前向扩散向3D 坐标添加高斯噪声，并对序列应用掩码符号扩散；反向使用 SE(3)-不变/协变表示。
• 目标是一个分解为结构损失和序列损失的 ELBO，简化为对结构的噪声回归损失和序列的交叉熵损失。
• SiamDiff 通过扭转扰动生成相关的构象对，并通过相互去噪来最大化它们的扩散轨迹之间的互信息。
• 采用两阶段噪声调度（先大噪声，后小噪声）以捕捉粗粒度和细粒度特征。
• 最终目标结合四个损失项，强制两个构象之间以及两种模态之间的跨轨迹去噪。

实验结果

研究问题

• RQ1 一个联合序列-结构扩散模型是否能够学习同时捕捉共进化序列信息和详细结构特征的表征？
• RQ2 将构象感知的 SiamDiff 纳入是否能在下游蛋白质理解任务上超越 DiffPreT 和现有基线？
• RQ3 两阶段噪声调度如何影响多模态蛋白质表征的学习？
• RQ4 Siamese diffusion 轨迹在跨任务（原子级和残基级）捕捉构象相关性方面的影响如何？

主要发现

• DiffPreT 在原子级和残基级任务上表现具有竞争力。
• SiamDiff 在评估任务上达到新的最先进平均秩。
• 原子级：SiamDiff 平均秩 = 1.2 与 DiffPreT = 2.4。
• 残基级：SiamDiff 平均秩 = 1.2 与 DiffPreT = 3.2。
• 总体而言，SiamDiff 提升了平均秩并展示出跨任务的强泛化能力。