[Paper Review] SAC-KG: Exploiting Large Language Models as Skilled Automatic Constructors for Domain Knowledge Graphs

Hanzhu Chen, Xu Shen+4 more|arXiv (Cornell University)|Sep 22, 2024

Advanced Graph Neural Networks4 citations

TL;DR

SAC-KG proposes a novel framework that leverages large language models (LLMs) as skilled automatic constructors to build precise, multi-level domain knowledge graphs (KGs) from raw corpora. By integrating a Generator, Verifier, and Pruner in an iterative, entity-induced tree search, SAC-KG achieves 89.32% precision on a KG with over one million nodes—surpassing state-of-the-art methods by over 20% in precision.

ABSTRACT

Knowledge graphs (KGs) play a pivotal role in knowledge-intensive tasks across specialized domains, where the acquisition of precise and dependable knowledge is crucial. However, existing KG construction methods heavily rely on human intervention to attain qualified KGs, which severely hinders the practical applicability in real-world scenarios. To address this challenge, we propose a general KG construction framework, named SAC-KG, to exploit large language models (LLMs) as Skilled Automatic Constructors for domain Knowledge Graph. SAC-KG effectively involves LLMs as domain experts to generate specialized and precise multi-level KGs. Specifically, SAC-KG consists of three components: Generator, Verifier, and Pruner. For a given entity, Generator produces its relations and tails from raw domain corpora, to construct a specialized single-level KG. Verifier and Pruner then work together to ensure precision by correcting generation errors and determining whether newly produced tails require further iteration for the next-level KG.Experiments demonstrate that SAC-KG automatically constructs a domain KG at the scale of over one million nodes and achieves a precision of 89.32%, leading to a superior performance with over 20% increase in precision rate compared to existing state-of-the-art methods for the KG construction task.

Motivation & Objective

• Address the high human dependency and low scalability of existing domain knowledge graph (KG) construction methods.
• Overcome two major challenges in LLM-based KG construction: contextual noise in input and knowledge hallucination in output.
• Enable fully automatic, precise, and controllable construction of domain-specific KGs using LLMs as skilled automatic constructors.
• Achieve high precision and scalability in constructing large-scale, multi-level domain KGs without requiring manual curation or fine-tuning.

Proposed method

• The Generator retrieves domain-specific corpora and relevant triples from an open-source KG (e.g., DBpedia) to form a contextually enriched input for the LLM.
• The LLM generates entity-induced triples (head-relation-tail) based on the combined input, forming a single-level KG for a given entity.
• The Verifier employs rule-based criteria from RuleHub to detect and classify generation errors, improving factual consistency.
• The Pruner evaluates generated tails and determines whether they should be marked for further iteration (‘growing’) or pruned, enabling multi-level KG expansion.
• The framework uses an iterative, tree-search-like process where entities flagged as ‘growing’ are used as inputs for the next generation step.
• The integration of open KG retrieval and domain corpora retrieval enhances factual grounding and reduces hallucination.

Experimental results

Research questions

• RQ1 Can LLMs be effectively leveraged as skilled automatic constructors to build precise, domain-specific knowledge graphs with minimal human intervention?
• RQ2 How can contextual noise in raw domain corpora be mitigated to improve the quality of LLM-generated triples?
• RQ3 To what extent can error detection and pruning mechanisms reduce knowledge hallucination and improve KG precision?
• RQ4 Can the iterative, multi-level construction process significantly enhance the scalability and factual accuracy of domain KGs compared to single-pass methods?
• RQ5 How does SAC-KG perform across diverse benchmarks, including both domain-specific and open information extraction (OIE) tasks?

Key findings

• SAC-KG constructs a domain-specific knowledge graph with over one million nodes and achieves a precision of 89.32%, significantly outperforming existing state-of-the-art methods.
• The framework improves precision by over 20% compared to prior SOTA methods in domain KG construction, demonstrating superior factual accuracy.
• Ablation studies show that removing any component—especially the pruner or open KG retriever—leads to notable performance degradation, highlighting their critical role.
• In case studies on rice varieties and experts, SAC-KG produces more interpretable and domain-specific triples than rule-based or other LLM-based methods, despite lower recall.
• SAC-KG achieves state-of-the-art performance on standard OIE benchmarks (OIE2016, NYT, WEB, PENN), confirming its generalization and robustness beyond domain-specific tasks.
• The integration of both domain corpora and open KG retrieval significantly enhances generation quality, as shown by ablation results and case analysis.