[论文解读] Multi-modal Transfer Learning between Biological Foundation Models

Juan Jose Garau-Luis, Patrick Bordes+10 more|arXiv (Cornell University)|Jun 20, 2024

Machine Learning in Bioinformatics被引用 6

一句话总结

IsoFormer 是首个将 DNA、RNA 和蛋白质编码器连接起来以在不同组织中预测 RNA 转录本剪接异构体表达的多模态模型，通过在模态之间迁移知识达到 state-of-the-art。

ABSTRACT

Biological sequences encode fundamental instructions for the building blocks of life, in the form of DNA, RNA, and proteins. Modeling these sequences is key to understand disease mechanisms and is an active research area in computational biology. Recently, Large Language Models have shown great promise in solving certain biological tasks but current approaches are limited to a single sequence modality (DNA, RNA, or protein). Key problems in genomics intrinsically involve multiple modalities, but it remains unclear how to adapt general-purpose sequence models to those cases. In this work we propose a multi-modal model that connects DNA, RNA, and proteins by leveraging information from different pre-trained modality-specific encoders. We demonstrate its capabilities by applying it to the largely unsolved problem of predicting how multiple RNA transcript isoforms originate from the same gene (i.e. same DNA sequence) and map to different transcription expression levels across various human tissues. We show that our model, dubbed IsoFormer, is able to accurately predict differential transcript expression, outperforming existing methods and leveraging the use of multiple modalities. Our framework also achieves efficient transfer knowledge from the encoders pre-training as well as in between modalities. We open-source our model, paving the way for new multi-modal gene expression approaches.

研究动机与目标

• Motivate and enable multi-modal integration of DNA, RNA, and protein sequences to capture interconnected biological signals.
• Develop a general multi-modal aggregation framework that leverages pre-trained modality-specific encoders.
• Demonstrate that multi-modal learning improves RNA transcript isoform expression prediction across tissues.
• Show transfer learning benefits within and across modalities and assess encoder choices.
• Open-source IsoFormer to foster further multi-modal sequence modeling in biology.

提出的方法

• Use three pre-trained encoders (DNA, RNA, protein) to produce embeddings.
• Introduce a cross-attention-based aggregation module with residual connections to create modality-specific multi-modal embeddings.
• Concatenate per-modality multi-modal embeddings into a single h_multi for downstream prediction.
• Train end-to-end with an IsoFormer head to predict transcript isoform expression across tissues using MSE loss.
• Compare different modality combinations and aggregation strategies, and evaluate transfer from pre-trained encoders.

实验结果

研究问题

• RQ1 Can a three-modality (DNA, RNA, protein) encoder framework improve transcript isoform expression prediction over single-modality models?
• RQ2 Does cross-attention-based aggregation effectively fuse multi-modal biological embeddings and allow end-to-end training?
• RQ3 To what extent do pre-trained encoders contribute to performance, and is inter-modal transfer beneficial?
• RQ4 How does the choice of DNA encoder (e.g., Enformer vs NT) affect transfer to the isoform prediction task?

主要发现

• Best performing model uses three modalities (DNA + RNA + Protein) achieving R^2 = 0.43 ± 0.01 and Spearman = 0.65 ± 0.01.
• Using Enformer as the DNA encoder yields the strongest results (R^2 = 0.53 ± 0.01, Spearman = 0.72 ± 0.00) for IsoFormer.
• Aggregation strategy based on cross-attention outperforms alternatives like Perceiver Resampler variants and C-Abstractor in this task.
• Pre-trained encoders provide substantial gains; all encoders pre-trained yields R^2 = 0.53 and Spearman = 0.71, while none pre-trained yields R^2 = 0.10 and Spearman = 0.31.
• Transfer is observed both intra-modal (pre-training within a modality) and inter-modal (leveraging information across modalities).
• IsoFormer with Enformer DNA encoder demonstrates strong cross-modality transfer and long-range context benefits.