[Paper Review] Evaluating Ontology Matching Systems on Large, Multilingual and Real-world Test Cases

Christian Meilicke, Ondřej Šváb-Zamazal+5 more|arXiv (Cornell University)|Aug 15, 2012

Semantic Web and Ontologies25 references3 citations

TL;DR

This paper evaluates 18 ontology matching systems on large-scale, multilingual, and real-world test cases within the OAEI 2011.5 campaign, using standardized benchmarks and automated evaluation infrastructure. Key contributions include identifying scalability challenges, multilingual limitations, and coherence issues in alignments, with LogMap achieving the most coherent results and GOMMA (with background knowledge) leading in F-measure for most tasks.

ABSTRACT

In the field of ontology matching, the most systematic evaluation of matching systems is established by the Ontology Alignment Evaluation Initiative (OAEI), which is an annual campaign for evaluating ontology matching systems organized by different groups of researchers. In this paper, we report on the results of an intermediary OAEI campaign called OAEI 2011.5. The evaluations of this campaign are divided in five tracks. Three of these tracks are new or have been improved compared to previous OAEI campaigns. Overall, we evaluated 18 matching systems. We discuss lessons learned, in terms of scalability, multilingual issues and the ability do deal with real world cases from different domains.

Motivation & Objective

• To assess the performance of ontology matching systems on large, multilingual, and real-world datasets to reflect real-world applicability.
• To evaluate system scalability across increasing ontology sizes and complexity.
• To examine the impact of multilingual support on matching quality, especially in semantic web applications.
• To analyze alignment coherence and its implications for downstream applications like query processing.
• To establish reproducible, automated evaluation using the SEALS infrastructure for consistent benchmarking.

Proposed method

• Conducted an intermediary OAEI campaign (OAEI 2011.5) using standardized datasets from diverse domains including biomedical, conference, and multilingual ontologies.
• Employed a controlled execution environment via the SEALS project infrastructure to ensure reproducibility and automation of evaluations.
• Evaluated systems using precision, recall, F-measure, runtime, and alignment coherence (via HermiT reasoning on merged ontologies with alignments).
• Used gold-standard alignments for reference, including refined and original UMLS-based alignments, to measure compliance.
• Applied multiple test cases: Benchmark (scalability), MultiFarm (multilingual), Large BioMed (real-world complexity), and others.
• Measured performance across different configurations (e.g., GOMMA with/without background knowledge) and hardware (16 CPUs, 10GB RAM).

Experimental results

Research questions

• RQ1 How do ontology matching systems perform on large-scale, real-world ontologies in terms of precision, recall, and F-measure?
• RQ2 To what extent do systems handle multilingual labels, and how does this affect matching quality across different language types?
• RQ3 What is the impact of ontology size and structural complexity (number of classes and properties) on system scalability and runtime?
• RQ4 How do alignment coherence metrics correlate with precision and recall, and which systems produce coherent alignments suitable for reasoning?
• RQ5 Can computational resources (e.g., multi-core CPUs) significantly improve performance, and which systems effectively utilize them?

Key findings

• GOMMA with background knowledge (GOMMA_bk) achieved the highest F-measure (0.921) on Task 1 and maintained strong performance across tasks, with F-measure decreasing to 0.818 on the largest task (Task 3).
• LogMap produced the most coherent alignments, with no unsatisfiable classes detected in reasoning, while GOMMA_nobk generated highly precise but incoherent alignments due to 100% unsatisfiable classes.
• Only LogMap, CODI, and YAM++ consistently produced coherent alignments across tracks, highlighting a critical gap in alignment quality assessment.
• Runtime increased significantly with ontology size, with GOMMA_bk taking over 18 minutes on the largest task (Task 3), while CSA maintained consistent results across all tasks.
• Multilingual support remains weak: only 3 systems effectively handled multilingual labels in the MultiFarm dataset, indicating a major limitation for multilingual knowledge integration.
• Despite improvements, no system surpassed the baseline LogMapLt in F-measure on the Benchmark and Large BioMed tracks, indicating high difficulty in detecting non-trivial correspondences without precision loss.