[Paper Review] AutoKG: Efficient Automated Knowledge Graph Generation for Language Models

Bohan Chen, Andrea L. Bertozzi|arXiv (Cornell University)|Nov 22, 2023

Topic Modeling4 citations

TL;DR

AutoKG proposes a lightweight, non-trainable method for automated knowledge graph (KG) generation by extracting keywords via large language models (LLMs) and computing edge weights using graph Laplace learning. It enhances retrieval-augmented generation through a hybrid search combining semantic similarity and graph-based associations, improving factual consistency and response quality without fine-tuning neural networks.

ABSTRACT

Traditional methods of linking large language models (LLMs) to knowledge bases via the semantic similarity search often fall short of capturing complex relational dynamics. To address these limitations, we introduce AutoKG, a lightweight and efficient approach for automated knowledge graph (KG) construction. For a given knowledge base consisting of text blocks, AutoKG first extracts keywords using a LLM and then evaluates the relationship weight between each pair of keywords using graph Laplace learning. We employ a hybrid search scheme combining vector similarity and graph-based associations to enrich LLM responses. Preliminary experiments demonstrate that AutoKG offers a more comprehensive and interconnected knowledge retrieval mechanism compared to the semantic similarity search, thereby enhancing the capabilities of LLMs in generating more insightful and relevant outputs.

Motivation & Objective

• To address the limitations of semantic similarity search in linking LLMs to knowledge bases, particularly in capturing complex relational dynamics.
• To develop a lightweight, efficient method for automated knowledge graph construction that avoids training or fine-tuning neural networks.
• To enhance LLM reasoning and factual accuracy by integrating a simplified, keyword-based KG with a hybrid search strategy.
• To enable on-the-fly, interpretable, and scalable knowledge retrieval without requiring complex subgraph embeddings or neural adaptation.
• To provide a computationally efficient alternative to existing retrieval-augmented generation (RAG) methods that rely on dense neural retrievers or end-to-end training.

Proposed method

• AutoKG extracts keywords from text blocks in a knowledge base using a pretrained LLM via prompt engineering.
• It constructs a simplified, undirected knowledge graph where nodes are keywords and edges are weighted based on graph Laplace learning to reflect relational strength.
• The method applies graph Laplace learning to compute edge weights by solving a system of linear equations derived from the graph Laplacian matrix, capturing local and global structural relationships.
• A hybrid search strategy combines semantic vector similarity (based on embedding similarity) and graph-based adjacency search to retrieve both semantically relevant and structurally connected information.
• The retrieved keywords and associated text blocks are injected into the LLM prompt to enrich generation, avoiding the need for neural adaptation of the LLM to KG structure.
• The approach is computationally efficient, with theoretical time complexity of O(N log N) for KG construction and O(N) for search, matching standard semantic retrieval in scalability.

Experimental results

Research questions

• RQ1 Can a non-trainable, lightweight knowledge graph construction method improve retrieval-augmented generation in LLMs compared to standard semantic similarity search?
• RQ2 How does the integration of graph-based associations with semantic similarity enhance the factual consistency and relevance of LLM outputs?
• RQ3 To what extent can simplified keyword-based KGs, without entity-specific relations or directed edges, support complex reasoning in LLMs?
• RQ4 What is the computational efficiency of the proposed hybrid search compared to traditional semantic retrieval, especially at scale?
• RQ5 Can AutoKG be effectively applied to real-world knowledge bases without requiring fine-tuning or complex neural adaptation?

Key findings

• AutoKG constructs a knowledge graph with M ≈ 0.1N keywords from N text blocks, significantly reducing complexity while preserving relational structure.
• The hybrid search method achieves an average response time of 0.0310 seconds for retrieving up to 60 text blocks, slightly slower than semantic similarity search (0.0305 seconds), confirming matching time complexity of O(N).
• The method demonstrates comparable efficiency to standard semantic retrieval while enabling richer, more interconnected knowledge access through graph-based associations.
• Preliminary qualitative evaluations show that AutoKG enhances LLM responses by incorporating more diverse and contextually linked information, improving factual relevance and insightfulness.
• The approach avoids model fine-tuning and neural adaptation, making it highly flexible for on-the-fly updates and deployment across diverse knowledge bases.
• The authors identify the need for a more complex, structured dataset to enable rigorous quantitative evaluation of AutoKG’s integration with LLMs in future work.