[Paper Review] Gollum: A Gold Standard for Large Scale Multi Source Knowledge Graph Matching

Sven Hertling, Heiko Paulheim|arXiv (Cornell University)|Sep 15, 2022

Semantic Web and Ontologies4 citations

TL;DR

Gollum presents a large-scale gold standard for multi-source knowledge graph matching, comprising over 275,000 correspondences across 4,149 knowledge graphs derived from DBpedia-extracted wikis. It enables evaluation of both unsupervised and supervised matching approaches through three distinct splits, including a transitive-closure-resistant test set, and demonstrates high-quality alignment with precision up to 0.985 on confidence-filtered subsets.

ABSTRACT

The number of Knowledge Graphs (KGs) generated with automatic and manual approaches is constantly growing. For an integrated view and usage, an alignment between these KGs is necessary on the schema as well as instance level. While there are approaches that try to tackle this multi source knowledge graph matching problem, large gold standards are missing to evaluate their effectiveness and scalability. We close this gap by presenting Gollum -- a gold standard for large-scale multi source knowledge graph matching with over 275,000 correspondences between 4,149 different KGs. They originate from knowledge graphs derived by applying the DBpedia extraction framework to a large wiki farm. Three variations of the gold standard are made available: (1) a version with all correspondences for evaluating unsupervised matching approaches, and two versions for evaluating supervised matching: (2) one where each KG is contained both in the train and test set, and (3) one where each KG is exclusively contained in the train or the test set.

Motivation & Objective

• To address the lack of large-scale, multi-source gold standards for evaluating knowledge graph matching systems.
• To enable benchmarking of both unsupervised and supervised matching approaches across diverse training and test splits.
• To support the evaluation of scalability and effectiveness in multi-source knowledge graph integration scenarios.
• To provide a high-quality, confidence-adjustable dataset derived from real-world DBpedia-extracted knowledge graphs.
• To facilitate future development of multi-source matching systems by offering training and test sets with controlled data leakage prevention.

Proposed method

• The gold standard is constructed from 4,149 knowledge graphs extracted via the DBpedia framework from a large wiki farm.
• Correspondences are established at the instance, class, and property levels using a combination of manual curation and confidence thresholds based on similarity scores.
• Three versions of the dataset are released: one for unsupervised evaluation, one with overlapping KGs in train and test, and one with exclusive KG splits to prevent transitive closure leakage.
• A hierarchical agglomerative clustering (HAC) strategy is used to guide the incremental merge process in multi-source matching, based on tf-idf vectors of textual content and URI fragments.
• Two 1:1 matchers—Alod2Vec and a string-based matcher—are reused in a multi-source strategy, with postprocessing enforcing 1:1 alignment via confidence sorting.
• The evaluation includes precision, recall, and F1 scores across instance, class, and property matching, with confidence thresholds applied to assess robustness.

Experimental results

Research questions

• RQ1 How effective are existing 1:1 knowledge graph matching systems when adapted to multi-source matching through iterative merging?
• RQ2 What is the impact of data leakage via transitive closure on the reliability of test sets in multi-source knowledge graph matching?
• RQ3 How does the quality of the gold standard vary across different confidence thresholds and matching types (instance, class, property)?
• RQ4 Can supervised training splits improve the performance of unsupervised matching systems in multi-source settings?
• RQ5 What is the scalability and performance trade-off between different matching strategies in large-scale multi-source integration?

Key findings

• The gold standard contains 275,103 correspondences across 4,149 knowledge graphs, with high-quality alignment verified through confidence-based filtering.
• On the highest-confidence subset (conf=1.0), precision reaches 0.985, indicating strong alignment quality.
• The string-based matcher achieves a precision of 0.959 and F1 of 0.795 on instance matching, outperforming Alod2Vec in this category.
• Alod2Vec achieves higher class matching precision (0.842) than the string-based matcher (0.829), indicating better performance on semantic class alignment.
• Recall values remain low (e.g., 0.390 for Alod2Vec on instances), indicating that many true correspondences remain undetected, highlighting room for improvement.
• The runtime for Alod2Vec is significantly higher (six days) than the string-based matcher (two days), primarily due to computationally intensive embedding-based matching.