[Paper Review] Agent-OM: Leveraging LLM Agents for Ontology Matching

Zhangcheng Qiang, Weiqing Wang+1 more|arXiv (Cornell University)|Dec 1, 2023

Semantic Web and Ontologies4 citations

TL;DR

This paper proposes Agent-OM, a novel LLM-based framework for ontology matching that uses two Siamese agents with prompt-based tools to enhance retrieval and matching. It achieves state-of-the-art performance on complex and few-shot ontology matching tasks without fine-tuning, leveraging chain-of-thought reasoning, retrieval-augmented generation, and in-context learning.

ABSTRACT

Ontology matching (OM) enables semantic interoperability between different ontologies and resolves their conceptual heterogeneity by aligning related entities. OM systems currently have two prevailing design paradigms: conventional knowledge-based expert systems and newer machine learning-based predictive systems. While large language models (LLMs) and LLM agents have revolutionised data engineering and have been applied creatively in many domains, their potential for OM remains underexplored. This study introduces a novel agent-powered LLM-based design paradigm for OM systems. With consideration of several specific challenges in leveraging LLM agents for OM, we propose a generic framework, namely Agent-OM (Agent for Ontology Matching), consisting of two Siamese agents for retrieval and matching, with a set of OM tools. Our framework is implemented in a proof-of-concept system. Evaluations of three Ontology Alignment Evaluation Initiative (OAEI) tracks over state-of-the-art OM systems show that our system can achieve results very close to the long-standing best performance on simple OM tasks and can significantly improve the performance on complex and few-shot OM tasks.

Motivation & Objective

• To address the underutilization of LLMs and LLM-based agents in ontology matching (OM), a key task in semantic interoperability.
• To overcome limitations of existing OM systems—such as reliance on expert knowledge or large labeled datasets—by leveraging pre-trained LLMs.
• To design a scalable, no-code, natural language-driven framework that enables few-shot and complex OM tasks without model fine-tuning.
• To explore how LLM agents with planning, memory, and tool use can improve OM performance beyond standard prompt-based LLM inference.
• To evaluate the framework on diverse OAEI tracks and demonstrate its superiority in complex and low-resource scenarios.

Proposed method

• Design a dual-agent architecture with two Siamese LLM agents specialized for retrieval and matching, respectively.
• Employ chain-of-thought (CoT) prompting to decompose complex OM tasks into logical reasoning steps.
• Integrate retrieval-augmented generation (RAG) and in-context learning (ICL) to enhance factual grounding and reduce hallucinations.
• Develop a set of prompt-based tools for entity retrieval, semantic matching, and memory storage to enable tool-augmented reasoning.
• Use a generic, reusable prompt engineering pipeline that avoids model fine-tuning and supports dynamic, real-time knowledge injection.
• Implement the framework as a proof-of-concept system and evaluate it across three OAEI tracks using standard OM benchmarks.

Experimental results

Research questions

• RQ1 Can LLM-based agents outperform traditional knowledge-based and machine learning-based OM systems in complex and few-shot ontology matching tasks?
• RQ2 How effective are CoT reasoning and RAG in mitigating LLM hallucinations during ontology matching?
• RQ3 To what extent can a no-code, prompt-based framework achieve high performance without fine-tuning or domain-specific data?
• RQ4 How does the dual-agent (Siamese) design improve retrieval and matching accuracy compared to single-agent LLM approaches?
• RQ5 What role does in-context learning and tool use play in enhancing the scalability and reusability of LLM-based OM systems?

Key findings

• Agent-OM achieves performance very close to the best long-standing OM systems on simple ontology matching tasks, demonstrating strong baseline capability.
• On complex and few-shot OM tasks, Agent-OM significantly outperforms existing state-of-the-art systems, indicating strong generalization and reasoning ability.
• The framework achieves high performance without model fine-tuning, relying instead on prompt engineering, CoT, and RAG for contextual grounding.
• The use of Siamese agents with specialized retrieval and matching roles improves accuracy and task decomposition compared to monolithic LLM approaches.
• The system demonstrates high reusability and scalability, effectively handling low-resource scenarios with only a few seed mappings.
• RAG and in-context learning effectively reduce hallucinations and improve factual consistency, though not entirely eliminated.