[Paper Review] OntoChat: a Framework for Conversational Ontology Engineering using Language Models

Boxuan Zhang, Valentina Anita Carriero+5 more|arXiv (Cornell University)|Mar 9, 2024

Natural Language Processing Techniques4 citations

TL;DR

OntoChat is a conversational framework that leverages large language models (LLMs) to automate and streamline ontology engineering tasks, including user story generation, competency question (CQ) extraction, CQ clustering, and ontology testing. It achieved 87.5% accuracy in classifying supported CQs and was well-received by both domain experts and ontology engineers for reducing manual effort and improving requirement elicitation efficiency.

ABSTRACT

Ontology engineering (OE) in large projects poses a number of challenges arising from the heterogeneous backgrounds of the various stakeholders, domain experts, and their complex interactions with ontology designers. This multi-party interaction often creates systematic ambiguities and biases from the elicitation of ontology requirements, which directly affect the design, evaluation and may jeopardise the target reuse. Meanwhile, current OE methodologies strongly rely on manual activities (e.g., interviews, discussion pages). After collecting evidence on the most crucial OE activities, we introduce extbf{OntoChat}, a framework for conversational ontology engineering that supports requirement elicitation, analysis, and testing. By interacting with a conversational agent, users can steer the creation of user stories and the extraction of competency questions, while receiving computational support to analyse the overall requirements and test early versions of the resulting ontologies. We evaluate OntoChat by replicating the engineering of the Music Meta Ontology, and collecting preliminary metrics on the effectiveness of each component from users. We release all code at https://github.com/King-s-Knowledge-Graph-Lab/OntoChat.

Motivation & Objective

• To identify the most labor-intensive and error-prone ontology engineering (OE) activities requiring computational support.
• To design a conversational framework that leverages LLMs to automate requirement elicitation, analysis, and testing in OE.
• To reduce systemic ambiguities and biases arising from manual, multi-party interactions between domain experts and ontology designers.
• To evaluate the framework’s effectiveness in accelerating and improving the quality of ontology development, particularly for real-world use cases like the Music Meta Ontology.
• To release a reproducible, open-source prototype to support community adoption and further research in LLM-augmented ontology engineering.

Proposed method

• Conducted a survey with 24 ontology engineers to rank OE activities by difficulty and manual effort, identifying requirement elicitation and CQ generation as top candidates for automation.
• Built OntoChat as a conversational agent that supports users in creating user stories, extracting CQs, clustering CQs, and testing ontologies via LLM-driven prompts.
• Implemented LLM-based workflows for generating user stories from domain expert input, extracting semantically coherent and scope-aligned CQs, and clustering them into thematic groups.
• Designed a classification module to evaluate ontology testing by distinguishing between supported and adversarial CQs using fine-tuned LLMs.
• Integrated feedback loops where users can refine outputs, enabling iterative improvement of requirements and reducing bias in early-stage specification.
• Deployed an online prototype and evaluated it with 6 domain experts and 8 ontology engineers using questionnaires and a controlled classification task.

Experimental results

Research questions

• RQ1 Which ontology engineering activities are the most in need of computational support?
• RQ2 How can LLMs enable a conversational ontology engineering framework to support requirement elicitation, analysis, and testing?
• RQ3 To what extent can LLMs reduce manual effort and improve consistency in user story and competency question generation?
• RQ4 How effective is the framework in identifying and classifying valid versus invalid competency questions?

Key findings

• 80% of domain experts found the user stories generated by OntoChat to be well-structured and relevant, with 5 out of 6 preferring it over manual curation.
• 62.5% of ontology engineers found the CQ clustering feature intuitive, and 87.5% considered the resulting clusters to represent meaningful groupings of requirements.
• The ontology testing component achieved 87.5% accuracy in classifying supported CQs, with a precision of 88% and recall of 85.7%.
• While 6 out of 8 ontology engineers were satisfied with CQ generation quality, 2 noted issues with entity scope accuracy, indicating room for refinement.
• 50% of domain experts were dissatisfied with the example data generated in user stories, highlighting a need for improved data synthesis in LLM prompts.
• All participants agreed that OntoChat holds strong promise for reducing time and effort in ontology engineering, especially in collaborative, multi-stakeholder settings.