[Paper Review] BEACON: Benchmark for Comprehensive RNA Tasks and Language Models

Yuchen Ren, Zhiyuan Chen+11 more|arXiv (Cornell University)|Jun 14, 2024

RNA and protein synthesis mechanisms7 citations

TL;DR

BEACON introduces the first comprehensive RNA benchmark with 13 tasks across structure, function, and engineering, analyzes various models including RNA foundation models, and identifies single-nucleotide tokenization and ALiBi as effective components, proposing the BEACON-B baseline.

ABSTRACT

RNA plays a pivotal role in translating genetic instructions into functional outcomes, underscoring its importance in biological processes and disease mechanisms. Despite the emergence of numerous deep learning approaches for RNA, particularly universal RNA language models, there remains a significant lack of standardized benchmarks to assess the effectiveness of these methods. In this study, we introduce the first comprehensive RNA benchmark BEACON ( extbf{BE}nchm extbf{A}rk for extbf{CO}mprehensive R extbf{N}A Task and Language Models). First, BEACON comprises 13 distinct tasks derived from extensive previous work covering structural analysis, functional studies, and engineering applications, enabling a comprehensive assessment of the performance of methods on various RNA understanding tasks. Second, we examine a range of models, including traditional approaches like CNNs, as well as advanced RNA foundation models based on language models, offering valuable insights into the task-specific performances of these models. Third, we investigate the vital RNA language model components from the tokenizer and positional encoding aspects. Notably, our findings emphasize the superiority of single nucleotide tokenization and the effectiveness of Attention with Linear Biases (ALiBi) over traditional positional encoding methods. Based on these insights, a simple yet strong baseline called BEACON-B is proposed, which can achieve outstanding performance with limited data and computational resources. The datasets and source code of our benchmark are available at https://github.com/terry-r123/RNABenchmark.

Motivation & Objective

• Establish a comprehensive, standardized benchmark for RNA tasks spanning structure, function, and engineering to enable fair comparisons across methods.
• Systematically evaluate traditional neural models and RNA language models on diverse RNA tasks.
• Investigate RNA language model components (tokenization and positional encoding) to identify effective design choices.
• Propose a strong, efficient baseline BEACON-B leveraging single-nucleotide tokenization and ALiBi for broad applicability.

Proposed method

• Assemble BEACON with 13 tasks drawn from structure, function, and engineering research, totaling 967k RNA sequences.
• Evaluate a spectrum of models from CNNs, ResNets, LSTMs to pretrained RNA language models (RNA-FM, RNABERT, RNA-MSM, SpliceBERT, 3UTRBERT, UTR-LM).
• Conduct ablation studies on tokenization methods (Single Nucleotide, BPE, 6mer, Non-overlap) and positional encodings (APE, ALiBi, RoPE).
• Fine-tune RNA foundation models under identical training settings for fair comparison; compare against naive supervised baselines.
• Develop BEACON-B by combining single-nucleotide tokenization with ALiBi on a BERT backbone for fast, data-efficient performance.

Experimental results

Research questions

• RQ1 How do existing models (CNNs, ResNets, LSTMs, and RNA language models) perform across the 13 BEACON tasks?
• RQ2 What is the impact of tokenization and positional encoding choices on RNA language model performance?
• RQ3 Can a simple baseline like BEACON-B achieve strong results with limited data and compute?
• RQ4 Do pretraining attributes (e.g., ncRNA, 5'/3' UTR) confer task-specific gains across RNA tasks?
• RQ5 Which task types (structure, function, engineering) benefit most from RNA foundation models?

Key findings

• RNA language models improve on 8 of 13 tasks compared to prior task-specific SOTA, demonstrating the value of unsupervised pretraining.
• ResNet and LSTM baselines remain competitive and outperform some language models on several tasks, highlighting persistent strength of traditional architectures.
• Single-nucleotide tokenization consistently outperforms BPE, 6mer, and Non-overlap across most tasks, especially when paired with ALiBi.
• ALiBi positional encoding generally yields better generalization on RNA tasks than RoPE or Absolute Encoding, particularly for shorter sequences.
• Pretraining on RNA attributes (e.g., ncRNA, 5’/3’ UTR) yields task-specific gains, e.g., RNA-FM on ncRNA, SpliceBERT on pre-mRNA, and UTR-LM variants on UTR-related tasks.
• BEACON-B achieves strong performance with low data and compute, offering a fast, open-source baseline for the community.