[Paper Review] Using Large Language Models for OntoClean-based Ontology Refinement

Yihang Zhao, Neil Vetter+1 more|arXiv (Cornell University)|Mar 23, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper proposes using large language models (LLMs) like GPT-4 to automate the ontology refinement process via the OntoClean methodology, specifically by labeling classes with metaphysical meta-properties such as Rigidity, Identity, and Unity. Using in-context learning prompting on hierarchical ontology representations, GPT-4 achieved over 95% accuracy across all meta-properties, demonstrating strong feasibility for integrating LLMs into ontology tools via dedicated plugins.

ABSTRACT

This paper explores the integration of Large Language Models (LLMs) such as GPT-3.5 and GPT-4 into the ontology refinement process, specifically focusing on the OntoClean methodology. OntoClean, critical for assessing the metaphysical quality of ontologies, involves a two-step process of assigning meta-properties to classes and verifying a set of constraints. Manually conducting the first step proves difficult in practice, due to the need for philosophical expertise and lack of consensus among ontologists. By employing LLMs with two prompting strategies, the study demonstrates that high accuracy in the labelling process can be achieved. The findings suggest the potential for LLMs to enhance ontology refinement, proposing the development of plugin software for ontology tools to facilitate this integration.

Motivation & Objective

• To address the challenge of manual ontology refinement using the OntoClean methodology, which requires philosophical expertise and lacks consensus.
• To investigate whether large language models (LLMs) can automate the assignment of OntoClean meta-properties to ontology classes.
• To evaluate the effectiveness of different prompting strategies—zero-shot and in-context learning—on LLMs for this task.
• To compare performance across GPT-3.5 and GPT-4 using flat and hierarchical ontology representations.
• To propose a plugin-based integration of LLMs into ontology tools like Protégé for scalable, human-in-the-loop ontology refinement.

Proposed method

• The study employs two prompting strategies: a zero-shot 'bare' prompt and an in-context learning prompt that includes definitions of OntoClean meta-properties.
• Ontology inputs were represented both flatly (as a list of entities) and hierarchically (as a tab-indented tree derived from a spanning tree of the ontology graph).
• The LLMs were prompted to label each class in the ontology with all relevant OntoClean meta-properties based on the provided definitions and structure.
• Experiments were conducted using two benchmark ontologies: the Upper Ontology and the Mini Pizza Ontology.
• Performance was evaluated across 30 trials per configuration, with results aggregated and compared across models, prompting types, and representation styles.
• The hierarchical representation was found to be essential for meaningful results, while flat representations were excluded due to lack of structural dependency.

Experimental results

Research questions

• RQ1 Can large language models accurately assign OntoClean meta-properties to ontology classes with minimal prompting?
• RQ2 How does in-context learning compare to zero-shot prompting in improving LLM accuracy for ontology meta-property labeling?
• RQ3 Does the choice of LLM (GPT-3.5 vs. GPT-4) significantly affect performance in labeling metaphysical properties?
• RQ4 How do flat versus hierarchical ontology representations impact the reliability of LLM-generated labels?
• RQ5 To what extent can LLMs handle abstract ontological concepts like Unity and Identity, which require philosophical reasoning?

Key findings

• GPT-4 achieved an inaccuracy rate of approximately 4% across all OntoClean meta-properties when using in-context learning, indicating over 95% accuracy.
• GPT-3.5 showed suboptimal performance, with inaccuracy rates of 60–70% for Identity and Unity, and 35–40% for Rigidity and Dependence in zero-shot settings.
• In-context learning improved GPT-3.5’s performance on Rigidity and Dependence, reducing inaccuracy to 25–30%, but had no significant effect on Unity.
• Flat ontology representations yielded unusable results due to lack of structural context and were excluded from final analysis.
• GPT-4’s performance on Unity did not improve with in-context learning, suggesting that abstract, dated definitions in OntoClean may hinder model interpretation.
• The results support the feasibility of integrating LLMs into ontology tools via plugins, particularly for high-accuracy, human-in-the-loop refinement workflows.