[Paper Review] Chain-of-History Reasoning for Temporal Knowledge Graph Forecasting

Yuwei Xia, Ding Wang+4 more|arXiv (Cornell University)|Feb 22, 2024

Topic Modeling4 citations

TL;DR

This paper proposes Chain-of-History (CoH) reasoning to enhance temporal knowledge graph (TKG) forecasting by enabling large language models (LLMs) to systematically explore high-order historical chains step-by-step, overcoming limitations in handling complex, long-context histories. CoH improves LLM-based TKG prediction accuracy and is designed as a plug-and-play module that fuses LLM reasoning with graph-based models, achieving state-of-the-art performance across multiple datasets and backbones.

ABSTRACT

Temporal Knowledge Graph (TKG) forecasting aims to predict future facts based on given histories. Most recent graph-based models excel at capturing structural information within TKGs but lack semantic comprehension abilities. Nowadays, with the surge of LLMs, the LLM-based TKG prediction model has emerged. However, the existing LLM-based model exhibits three shortcomings: (1) It only focuses on the first-order history for prediction while ignoring high-order historical information, resulting in the provided information for LLMs being extremely limited. (2) LLMs struggle with optimal reasoning performance under heavy historical information loads. (3) For TKG prediction, the temporal reasoning capability of LLM alone is limited. To address the first two challenges, we propose Chain-of-History (CoH) reasoning which explores high-order histories step-by-step, achieving effective utilization of high-order historical information for LLMs on TKG prediction. To address the third issue, we design CoH as a plug-and-play module to enhance the performance of graph-based models for TKG prediction. Extensive experiments on three datasets and backbones demonstrate the effectiveness of CoH.

Motivation & Objective

• To address the limitation of existing LLM-based TKG models that rely only on first-order histories and fail to utilize richer high-order temporal dependencies.
• To mitigate the performance degradation of LLMs under heavy historical information loads by structuring reasoning in a step-by-step, manageable manner.
• To enhance graph-based TKG models by integrating the semantic reasoning strengths of LLMs through a modular, plug-and-play CoH module.
• To improve temporal reasoning in TKG forecasting by combining the structural modeling of GNNs with the semantic comprehension of LLMs.

Proposed method

• CoH reasoning processes high-order historical chains incrementally: in each step, the LLM identifies and reasons over increasingly longer history chains (e.g., first-order, then second-order, etc.).
• At each step, the LLM is prompted to infer the most relevant history chain leading toward the query fact, reducing cognitive load and preventing information overload.
• The final prediction is made based only on the most recently inferred high-order chain, ensuring focused and accurate reasoning.
• CoH is designed as a plug-and-play module that fuses its LLM-generated predictions with those from graph-based models using a learnable weight parameter.
• The method avoids retraining and instead leverages in-context learning with prompt engineering to guide LLMs through structured reasoning paths.
• Prior knowledge is used to improve reasoning, but the authors filter out leakage to ensure that gains stem from semantic understanding, not memorization.

Experimental results

Research questions

• RQ1 Can step-by-step, chain-of-history reasoning improve LLM-based TKG forecasting by effectively utilizing high-order historical dependencies?
• RQ2 Does limiting the context length per reasoning step prevent performance degradation when handling long historical sequences?
• RQ3 Can CoH be effectively integrated as a plug-and-play module to enhance graph-based TKG models without retraining?
• RQ4 How does the fusion weight between CoH and graph-based model predictions affect overall performance?
• RQ5 To what extent does prior knowledge contribute to CoH reasoning, independent of data leakage?

Key findings

• CoH significantly improves MRR performance on ICEWS14, ICEWS18, and ICEWS05-15, outperforming both standalone LLM and graph-based baselines.
• Performance of LLMs alone degrades with increasing history length beyond a threshold, confirming the need for structured reasoning like CoH.
• The optimal fusion weight $w$ favors CoH predictions slightly more than graph-based model outputs, indicating CoH's strong complementary value.
• The hyperparameter $\alpha$, which controls score gap between indexes, has minimal impact on performance, suggesting robustness to hyperparameter tuning.
• Sensitivity analysis confirms that CoH's performance is stable across different $\alpha$ and $w$ values, indicating reliable and consistent behavior.
• The use of prior knowledge improves reasoning, but the gains are primarily due to static semantic knowledge rather than data leakage.