[Paper Review] Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions

Jiayang Chen, Zhigang Hu+10 more|arXiv (Cornell University)|Apr 1, 2022

RNA and protein synthesis mechanisms49 citations

TL;DR

RNA-FM is a foundation model trained on 23 million unannotated non-coding RNA sequences via self-supervised learning to produce embeddings that improve RNA secondary/3D structure and function predictions, including SARS-CoV-2 analyses, with strong cross-task generalization.

ABSTRACT

Non-coding RNA structure and function are essential to understanding various biological processes, such as cell signaling, gene expression, and post-transcriptional regulations. These are all among the core problems in the RNA field. With the rapid growth of sequencing technology, we have accumulated a massive amount of unannotated RNA sequences. On the other hand, expensive experimental observatory results in only limited numbers of annotated data and 3D structures. Hence, it is still challenging to design computational methods for predicting their structures and functions. The lack of annotated data and systematic study causes inferior performance. To resolve the issue, we propose a novel RNA foundation model (RNA-FM) to take advantage of all the 23 million non-coding RNA sequences through self-supervised learning. Within this approach, we discover that the pre-trained RNA-FM could infer sequential and evolutionary information of non-coding RNAs without using any labels. Furthermore, we demonstrate RNA-FM's effectiveness by applying it to the downstream secondary/3D structure prediction, SARS-CoV-2 genome structure and evolution prediction, protein-RNA binding preference modeling, and gene expression regulation modeling. The comprehensive experiments show that the proposed method improves the RNA structural and functional modelling results significantly and consistently. Despite only being trained with unlabelled data, RNA-FM can serve as the foundational model for the field.

Motivation & Objective

• Motivate leveraging massive unannotated ncRNA data to improve structure and function prediction beyond label-dependent models.
• Develop a task-agnostic RNA foundation model (RNA-FM) using self-supervised learning on 23M ncRNA sequences.
• Demonstrate that RNA-FM embeddings capture sequential, structural, and evolutionary information.
• Show that fine-tuning with lightweight heads yields state-of-the-art performance across multiple downstream tasks.

Proposed method

• Build a 12-layer transformer (RNA-FM) based on the BERT architecture.
• Pre-train RNA-FM on 23 million ncRNA sequences from RNAcentral with masked-token reconstruction (self-supervised).
• Represent each sequence as an L x 640 embedding matrix after processing.
• Fine-tune with task-specific heads or use embeddings as features for downstream models.
• Compare against leading secondary-structure predictors and 3D distance/closure tasks across diverse benchmarks.
• Provide a web server and release code/weights for community use.

Experimental results

Research questions

• RQ1 Can a foundation model trained on unlabeled ncRNA data learn representations that capture structural and functional signals?
• RQ2 Do RNA-FM embeddings improve RNA secondary structure prediction, 3D closeness/distance prediction, and RNA-protein/regulatory tasks compared to state-of-the-art methods?
• RQ3 Is there interpretable, evolution-related information embedded in RNA-FM representations?
• RQ4 How well does RNA-FM generalize to regulatory regions and viral genomes (e.g., SARS-CoV-2) across benchmarks?

Key findings

• RNA-FM embeddings organize ncRNA types in embedding space by structure/function properties, indicating learned biological signals.
• RNA-FM achieves higher F1 on secondary structure benchmarks (e.g., ArchiveII600 and bpRNA TS0) than many state-of-the-art methods and can surpass UFold in several settings.
• For 3D closeness, a model using RNA-FM embeddings outperforms models using MSA covariances and PETfold, with transfer learning yielding large gains on small datasets.
• RNA-FM embeddings enable competitive or superior performance in protein-RNA interaction and gene expression regulation modeling, despite being trained only on unannotated RNA sequences.
• RNA-FM supports 3D distance prediction tasks, achieving higher R2 and PMCC and lower MSE when combined with sequence data, and can enable end-to-end differentiable 3D predictions for RNA puzzles.
• RNA-FM embeddings enhance SARS-CoV-2 genome regulatory element prediction and can illustrate evolutionary trends among viral variants.