[Paper Review] Rethinking Performance Measures of RNA Secondary Structure Problems

Frederic Runge, Jörg K. H. Franke+2 more|arXiv (Cornell University)|Dec 4, 2023

RNA and protein synthesis mechanisms4 citations

TL;DR

This paper proposes the Weisfeiler-Lehman (WL) graph kernel as a superior alternative to traditional metrics like F1 score and MCC for evaluating RNA secondary structure predictions. Unlike confusion matrix-based measures that fail to capture structural similarities in dynamic or pseudoknotted RNAs, WL leverages graph-based similarity to accurately reflect biological relevance, demonstrating improved evaluation fidelity and utility in RNA design tasks with measurable performance gains.

ABSTRACT

Accurate RNA secondary structure prediction is vital for understanding cellular regulation and disease mechanisms. Deep learning (DL) methods have surpassed traditional algorithms by predicting complex features like pseudoknots and multi-interacting base pairs. However, traditional distance measures can hardly deal with such tertiary interactions and the currently used evaluation measures (F1 score, MCC) have limitations. We propose the Weisfeiler-Lehman graph kernel (WL) as an alternative metric. Embracing graph-based metrics like WL enables fair and accurate evaluation of RNA structure prediction algorithms. Further, WL provides informative guidance, as demonstrated in an RNA design experiment.

Motivation & Objective

• To address the limitations of traditional evaluation metrics like F1 score and MCC in measuring RNA secondary structure similarity, especially for complex features like pseudoknots and multi-interacting base pairs.
• To demonstrate that confusion matrix-based metrics can produce misleading scores, as shown in cases where biologically similar structures receive zero or negative scores.
• To introduce the Weisfeiler-Lehman (WL) graph kernel as a more biologically meaningful metric that captures structural and sequence-level similarities in RNA structures.
• To evaluate the practical utility of WL in real-world applications, particularly in guiding learning-based RNA design algorithms.
• To explore the potential of using differentiable surrogate models of WL for end-to-end training of deep learning models in RNA structure prediction.

Proposed method

• Represent RNA secondary structures as undirected graphs where nucleotides are nodes and base pairs are edges, enabling graph-based analysis.
• Apply the Weisfeiler-Lehman (WL) graph kernel to compute structural similarity by iteratively refining node labels through neighborhood aggregation.
• Use the WL kernel score as a continuous, informative metric for comparing predicted and ground-truth RNA structures, replacing binary confusion matrix-based measures.
• Integrate WL into the evaluation loop of the libLEARNA RNA design algorithm, replacing Hamming distance to guide evolutionary search.
• Train a graph neural network (GNN)-based surrogate model to approximate WL distance, enabling differentiable guidance during deep learning training.
• Conduct controlled experiments on bi-stable RNAs, bulge migration simulations, and sequence mutation scenarios to compare WL against F1, F1-shift, and MCC.

Figure 1: Bi-stable RNA 20mer. The F1 score and MCC when comparing both folds is $0.0$ and $-0.026$ , respectively. The Weisfeiler-Lehman graph kernel provides a score of $0.25$ .

Experimental results

Research questions

• RQ1 Can traditional evaluation metrics like F1 score and MCC reliably measure similarity between biologically similar RNA secondary structures, especially when structural dynamics or pseudoknots are involved?
• RQ2 Does the Weisfeiler-Lehman graph kernel provide a more accurate and biologically meaningful measure of structural similarity than confusion matrix-based metrics in RNA secondary structure prediction?
• RQ3 Can WL improve the performance of learning-based RNA design algorithms by providing better guidance during optimization?
• RQ4 How does WL respond to biologically relevant changes such as base pair shifts, bulge migrations, and sequence mutations compared to existing metrics?
• RQ5 To what extent can a GNN-based surrogate model of WL enable differentiable training for deep learning models in RNA structure prediction?

Key findings

• The F1 score and MCC produce misleading results on bi-stable RNA 20mers, assigning a score of 0.0 and -0.026 respectively, despite the two conformations sharing identical base pairing patterns.
• The WL kernel correctly identifies structural similarity, yielding a score of 0.25 for the bi-stable RNA 20mer, demonstrating its ability to capture biologically relevant similarities.
• For positional shifts in base pairing, WL scores decrease gradually (0.519 for one shift, 0.469 for two), unlike F1 and F1-shift, which drop to zero, showing WL's sensitivity to structural changes.
• In bulge migration simulations, WL shows a gradual decline in similarity with increasing migration distance, while F1-shift scores remain at 1.0, indicating a failure to distinguish structural differences.
• WL effectively captures sequence-level changes, with scores decreasing proportionally to the number of mutations, a capability absent in F1, MCC, and F1-shift.
• When used to guide the libLEARNA RNA design algorithm, WL-based evaluation (libLEARNA-WL) leads to improved performance compared to Hamming distance-based evaluation, demonstrating practical utility.

Figure 2: Example of structural shift. (Left) We show a 5SrRNA of Drosophila melanogaster (Middle) The same structure shifted by one position. (Right) The same structure shifted by two positions.