[论文解读] Scalable Deep Learning for RNA Secondary Structure Prediction

Jörg K. H. Franke, Frederic Runge+1 more|arXiv (Cornell University)|Jul 14, 2023

RNA and protein synthesis mechanisms被引用 9

一句话总结

RNAformer 是一个精简的轴向注意力 Transformer，在潜在空间对 RNA 碱基对邻接进行建模，达到 state-of-the-art 的 TS0 性能，并在同族内外预测中展示学习潜在的生物物理折叠模型。

ABSTRACT

The field of RNA secondary structure prediction has made significant progress with the adoption of deep learning techniques. In this work, we present the RNAformer, a lean deep learning model using axial attention and recycling in the latent space. We gain performance improvements by designing the architecture for modeling the adjacency matrix directly in the latent space and by scaling the size of the model. Our approach achieves state-of-the-art performance on the popular TS0 benchmark dataset and even outperforms methods that use external information. Further, we show experimentally that the RNAformer can learn a biophysical model of the RNA folding process.

研究动机与目标

• Motivate improved de novo RNA secondary structure prediction beyond traditional dynamic programming and earlier DL methods.
• Propose a lean architecture that directly models the RNA pairing adjacency matrix in latent space.
• Demonstrate state-of-the-art performance on TS0 without external information or ensembles.
• Investigate inter-family prediction and the ability to learn a biophysical folding model using RNAfold-derived data from Rfam.

提出的方法

• Use axial attention to model row- and column-wise dependencies in a latent RNA pairing matrix.
• Embed the RNA sequence with two linear embeddings for row and column representations, then combine into a latent adjacency representation.
• Process latent representations through multiple Transformer-like blocks with row-wise and column-wise axial attention plus a 3x3 convolutional network.
• Apply recycling of the latent space (no gradients during intermediate passes) to increase effective depth.
• Mask 50% of unpaired entries in the adjacency matrix during loss computation to address class imbalance in base-pairing predictions.
• Train with AdamW, warm-up + cosine decay, dropout, pre-norm, and residual connections; use a 2D latent space up to 32M parameters; evaluation on intra-family (bpRNA) and inter-family (Rfam-RNAfold) datasets.

实验结果

研究问题

• RQ1 Can a lean axial-attention architecture directly model RNA base-pair adjacency to achieve competitive or superior TS0 performance without external information?
• RQ2 Does latent-space recycling improve performance by simulating deeper models in RNA secondary structure prediction?
• RQ3 Can RNAformer generalize to inter-family predictions and learn a biophysical folding model akin to RNAfold when trained on Rfam-derived data?

主要发现

• RNAformer achieves state-of-the-art TS0 performance, with the largest 32M-parameter model plus recycling reaching 0.728 TS0, 0.733 F1, and 17.2% solved on TS0.
• Increasing model size yields consistent performance gains across intra-family predictions, indicating beneficial inductive bias from the architecture.
• Recycling provides an approximate 1% performance lift over non-recycled variants.
• On inter-family/Rfam-derived data, RNAformer large model attains high intra-family-like accuracy (F1 up to 0.967 on Rfam TS and 0.651 on TS-hard) and can match or exceed RNAfold on TS-hard in some settings, suggesting the model can learn an underlying biophysical folding process.
• RNAformer scales with data size and model capacity, replicating RNAfold behavior more closely as model size grows.