[Paper Review] Structural Quality Metrics to Evaluate Knowledge Graphs

Sumin Seo, Heeseon Cheon+2 more|arXiv (Cornell University)|Nov 18, 2022

Semantic Web and Ontologies4 citations

TL;DR

This paper proposes six structural quality metrics to evaluate knowledge graph quality by assessing ontology structure and usage, focusing on class and property instantiation, hierarchy, and inheritance. Applied to Wikidata, DBpedia, YAGO, Freebase, Google KG, and Naver's Raftel, the metrics reveal that high-scale knowledge graphs often underutilize their ontologies, while well-structured graphs like Raftel achieve higher quality through refined, actively used class and property definitions.

ABSTRACT

This work presents six structural quality metrics that can measure the quality of knowledge graphs and analyzes five cross-domain knowledge graphs on the web (Wikidata, DBpedia, YAGO, Google Knowledge Graph, Freebase) as well as 'Raftel', Naver's integrated knowledge graph. The 'Good Knowledge Graph' should define detailed classes and properties in its ontology so that knowledge in the real world can be expressed abundantly. Also, instances and RDF triples should use the classes and properties actively. Therefore, we tried to examine the internal quality of knowledge graphs numerically by focusing on the structure of the ontology, which is the schema of knowledge graphs, and the degree of use thereof. As a result of the analysis, it was possible to find the characteristics of a knowledge graph that could not be known only by scale-related indicators such as the number of classes and properties.

Motivation & Objective

• To address the lack of structural quality evaluation in knowledge graphs beyond size and distribution metrics.
• To identify structural factors that determine the quality of knowledge graphs, particularly ontology design and usage.
• To develop quantitative metrics that reflect how well knowledge graphs utilize their ontologies to express real-world knowledge.
• To compare cross-domain knowledge graphs using these metrics to uncover structural strengths and weaknesses.
• To demonstrate that structural quality is a critical, measurable dimension of knowledge graph quality independent of data scale.

Proposed method

• Proposes six structural quality metrics: Instantiated Class Ratio, Instantiated Property Ratio, Class Instantiation, Subclass Property Acquisition, Subclass Property Instantiation, and Inverse Multiple Inheritance.
• Normalizes each metric to a 1–10 scale to enable cross-graph comparison.
• Combines Class Metrics (CM) and Property Metrics (PM) into a weighted average structural quality score, varying weights to reflect different evaluation priorities.
• Applies the metrics to five public knowledge graphs (Wikidata, DBpedia, YAGO, Freebase, Google KG) and one proprietary graph (Raftel).
• Uses RDF triple data and ontology definitions to compute instantiation and inheritance statistics.
• Employs normalization and weighted aggregation to produce a comprehensive, interpretable quality score reflecting structural maturity.

Experimental results

Research questions

• RQ1 What structural characteristics of knowledge graphs contribute to higher quality beyond mere size?
• RQ2 How well do knowledge graphs utilize their ontologies in terms of class and property instantiation?
• RQ3 To what extent does ontology hierarchy (e.g., subclassing) influence the expressiveness and quality of knowledge graphs?
• RQ4 How do different knowledge graphs compare in structural quality when evaluated through ontology usage and design?
• RQ5 Can structural quality metrics reveal hidden quality issues in large-scale knowledge graphs that scale-based metrics miss?

Key findings

• DBpedia achieved the highest structural quality score when property usage (PM) was prioritized, indicating strong property instantiation despite moderate class hierarchy.
• Raftel, based on Wikidata’s ontology, scored highest in Class Metrics (CM), demonstrating superior class-level structuring and instantiation.
• Wikidata, despite having a rich ontology with many classes and properties, showed relatively low instantiation rates, indicating underutilization of its structural potential.
• Freebase had high property counts but low Subclass Property Acquisition and Subclass Property Instantiation due to the absence of a class hierarchy, limiting its structural quality.
• Google Knowledge Graph scored low on most metrics due to limited public access to its ontology and minimal property exposure via API.
• YAGO exhibited low structural quality due to a small number of classes and properties, with weak subclassing and low instantiation, indicating limited expressiveness.