[Paper Review] Fair Evaluation of Global Network Aligners

Joseph Crawford, Yihan Sun+1 more|arXiv (Cornell University)|Jul 17, 2014

Bioinformatics and Genomic Networks33 references4 citations

TL;DR

This study fairly evaluates global network aligners MI-GRAAL and GHOST by decoupling their node cost functions (NCF) and alignment strategies (AS), revealing that MI-GRAAL's NCF consistently outperforms GHOST's, while AS performance is data-dependent. The best overall alignment results from combining MI-GRAAL's NCF with GHOST's AS, and the study finds that topological information is crucial for alignment quality, while sequence information weight (α) has minimal impact. Larger neighborhood sizes improve alignment, but the second-largest size often yields superior results with reduced computational cost.

ABSTRACT

Abstract Background Analogous to genomic sequence alignment, biological network alignment identifies conserved regions between networks of different species. Then, function can be transferred from well- to poorly-annotated species between aligned network regions. Network alignment typically encompasses two algorithmic components: node cost function (NCF), which measures similarities between nodes in different networks, and alignment strategy (AS), which uses these similarities to rapidly identify high-scoring alignments. Different methods use both different NCFs and different ASs. Thus, it is unclear whether the superiority of a method comes from its NCF, its AS, or both. We already showed on state-of-the-art methods, MI-GRAAL and IsoRankN, that combining NCF of one method and AS of another method can give a new superior method. Here, we evaluate MI-GRAAL against a newer approach, GHOST, by mixing-and-matching the methods’ NCFs and ASs to potentially further improve alignment quality. While doing so, we approach important questions that have not been asked systematically thus far. First, we ask how much of the NCF information should come from protein sequence data compared to network topology data. Existing methods determine this parameter more-less arbitrarily, which could affect alignment quality. Second, when topological information is used in NCF, we ask how large the size of the neighborhoods of the compared nodes should be. Existing methods assume that the larger the neighborhood size, the better. Results Our findings are as follows. MI-GRAAL’s NCF is superior to GHOST’s NCF, while the performance of the methods’ ASs is data-dependent. Thus, for data on which GHOST’s AS is superior to MI-GRAAL’s AS, the combination of MI-GRAAL’s NCF and GHOST’s AS represents a new superior method. Also, which amount of sequence information is used within NCF does not affect alignment quality, while the inclusion of topological information is crucial for producing good alignments. Finally, larger neighborhood sizes are preferred, but often, it is the second largest size that is superior. Using this size instead of the largest one would decrease computational complexity. Conclusion Taken together, our results represent general recommendations for a fair evaluation of network alignment methods and in particular of two-stage NCF-AS approaches.

Motivation & Objective

• To fairly evaluate the contributions of node cost function (NCF) and alignment strategy (AS) in global network alignment methods, specifically MI-GRAAL and GHOST.
• To determine the optimal balance between sequence and topological information in NCF, addressing the arbitrary parameter selection in existing methods.
• To investigate the impact of neighborhood size in NCF on alignment quality and computational efficiency.
• To provide generalizable recommendations for fair evaluation of global network alignment methods.

Proposed method

• Systematically mixed-and-matched NCFs and ASs from MI-GRAAL and GHOST to isolate component contributions.
• Evaluated alignment quality using seven metrics: NC (ground truth), EC, ICS, S3, LCCS (topological), and GO, EXP (biological).
• Varied the weight of sequence information (α) in NCF across multiple values to assess its impact on alignment performance.
• Tested four different neighborhood sizes (T1–T4) in NCF to determine optimal size for alignment quality and computational cost.
• Used both synthetic networks with known ground truth and real biological networks to evaluate performance across data types.
• Performed statistical correlation analysis between alignment quality measures to assess consistency and reliability of evaluation metrics.

Experimental results

Research questions

• RQ1 To what extent does the superiority of a network aligner stem from its NCF, AS, or both?
• RQ2 How much of the node similarity in NCF should come from protein sequence data versus network topology data?
• RQ3 What is the optimal neighborhood size in the NCF for achieving high-quality alignments without excessive computational cost?
• RQ4 How do different alignment quality measures correlate, especially in synthetic versus real network data?

Key findings

• MI-GRAAL’s NCF significantly outperforms GHOST’s NCF across all tested datasets and evaluation metrics.
• The performance of the alignment strategies (AS) is data-dependent, with GHOST’s AS outperforming MI-GRAAL’s in certain datasets.
• The weight of sequence information (α) in the NCF has no significant impact on alignment quality, indicating that sequence data is not a critical factor in NCF design.
• Incorporating topological information into the NCF is essential for producing high-quality alignments, as its absence leads to poor performance.
• Larger neighborhood sizes generally improve alignment quality, but the second-largest size (T3) often yields superior results compared to the largest size (T4), reducing computational cost.
• For real network data, topological and biological alignment quality measures show weak correlations (r ≤ 0.2), indicating a fundamental disconnect between topological and functional alignment quality in real-world scenarios.