[Paper Review] Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs

Yifei Zhang, Xintao Wang+4 more|arXiv (Cornell University)|Jun 30, 2024

Topic Modeling4 citations

TL;DR

This paper proposes Chain-of-Knowledge (CoK), a framework that enhances large language models (LLMs) with knowledge reasoning by learning from knowledge graphs (KGs). It introduces KnowReason, a curated dataset generated via rule mining and natural language conversion, and employs a trial-and-error mechanism to mitigate rule overfitting, significantly improving LLM performance on knowledge and general reasoning benchmarks, including OOD generalization and commonsense reasoning tasks.

ABSTRACT

Large Language Models (LLMs) have exhibited impressive proficiency in various natural language processing (NLP) tasks, which involve increasingly complex reasoning. Knowledge reasoning, a primary type of reasoning, aims at deriving new knowledge from existing one.While it has been widely studied in the context of knowledge graphs (KGs), knowledge reasoning in LLMs remains underexplored. In this paper, we introduce Chain-of-Knowledge, a comprehensive framework for knowledge reasoning, including methodologies for both dataset construction and model learning. For dataset construction, we create KnowReason via rule mining on KGs. For model learning, we observe rule overfitting induced by naive training. Hence, we enhance CoK with a trial-and-error mechanism that simulates the human process of internal knowledge exploration. We conduct extensive experiments with KnowReason. Our results show the effectiveness of CoK in refining LLMs in not only knowledge reasoning, but also general reasoning benchmarkms.

Motivation & Objective

• To address the underexplored integration of knowledge reasoning into large language models (LLMs), which is critical for complex reasoning tasks.
• To construct a high-quality, diverse dataset, KnowReason, by mining compositional rules from knowledge graphs and converting them into natural language samples.
• To overcome rule overfitting and hallucination in LLMs during knowledge reasoning training, which limits generalization to unseen rules.
• To improve LLM generalization on out-of-distribution (OOD) reasoning tasks and transferability to other reasoning benchmarks beyond knowledge reasoning.

Proposed method

• Dataset construction via three-stage pipeline: (1) rule mining from knowledge graphs to extract compositional relational patterns; (2) knowledge selection to identify interrelated triples matching mined rules; (3) sample generation to convert triples into natural language reasoning examples.
• Model learning using behavior cloning to fine-tune LLMs on the KnowReason dataset, enabling them to mimic reasoning chains.
• Introduction of a trial-and-error mechanism in CoK (T&E) that simulates human-like internal knowledge exploration, reducing overreliance on specific rules and improving robustness.
• Use of anonymized and regular settings for evaluation: anonymized to prevent data leakage and regular to test real-world applicability with real entities.
• Application of the CoK framework to downstream reasoning tasks, including commonsense reasoning (CSQA), BBH, and ARC, to evaluate transferability.
• Implementation of knowledge probing to ensure model-specific data in regular settings and prevent data leakage during evaluation.

Experimental results

Research questions

• RQ1 Can knowledge reasoning be effectively integrated into LLMs by leveraging structured knowledge from knowledge graphs?
• RQ2 How does rule overfitting affect LLM performance in knowledge reasoning, and can it be mitigated through a human-inspired learning mechanism?
• RQ3 To what extent does the CoK framework improve generalization on out-of-distribution (OOD) reasoning tasks?
• RQ4 Does the CoK framework enhance performance on diverse reasoning benchmarks beyond knowledge reasoning, such as commonsense or symbolic reasoning?

Key findings

• In the anonymized setting, CoK (T&E) achieved 22.22% accuracy on the OOD dataset, outperforming vanilla CoT (0.00%) and ICL-CoK (6.20%), demonstrating improved generalization.
• On the regular setting, CoK (T&E) achieved 22.22% accuracy on the OOD dataset, showing robustness to real-world entity data and reduced rule dependency.
• On commonsense reasoning benchmarks, Mistral-7b + CoK (T&E) improved CSQA accuracy from 66.5% to 68.1%, BBH from 53.2% to 54.7%, and ARC-e from 81.2% to 82.3%, indicating strong transferability.
• Error analysis revealed that in OOD settings, model errors were primarily due to incorrect reasoning paths rather than fact hallucinations, highlighting the importance of path selection in generalization.
• The trial-and-error mechanism in CoK (T&E) significantly reduced rule overfitting, as evidenced by better OOD performance and lower hallucination rates during entity selection.
• The KnowReason dataset, constructed via rule mining and natural language conversion, enables effective training and evaluation of knowledge reasoning in LLMs, with diverse and realistic reasoning chains.