[Paper Review] Large Language Models Can Learn Temporal Reasoning

Siheng Xiong, Ali Payani+2 more|arXiv (Cornell University)|Jan 12, 2024

Topic Modeling4 citations

TL;DR

This paper proposes TempGraph-LLM, a novel framework that enhances large language models (LLMs) for temporal reasoning by first translating text into structured temporal graphs and then applying symbolic reasoning via chain-of-thought bootstrapping and data augmentation. The method achieves state-of-the-art performance on temporal QA benchmarks, with a 79.5% exact match score using Llama2-13B, demonstrating that graph-structured reasoning and symbolic integration significantly improve accuracy and consistency over free-text reasoning.

ABSTRACT

While large language models (LLMs) have demonstrated remarkable reasoning capabilities, they are not without their flaws and inaccuracies. Recent studies have introduced various methods to mitigate these limitations. Temporal reasoning (TR), in particular, presents a significant challenge for LLMs due to its reliance on diverse temporal concepts and intricate temporal logic. In this paper, we propose TG-LLM, a novel framework towards language-based TR. Instead of reasoning over the original context, we adopt a latent representation, temporal graph (TG) that enhances the learning of TR. A synthetic dataset (TGQA), which is fully controllable and requires minimal supervision, is constructed for fine-tuning LLMs on this text-to-TG translation task. We confirmed in experiments that the capability of TG translation learned on our dataset can be transferred to other TR tasks and benchmarks. On top of that, we teach LLM to perform deliberate reasoning over the TGs via Chain-of-Thought (CoT) bootstrapping and graph data augmentation. We observed that those strategies, which maintain a balance between usefulness and diversity, bring more reliable CoTs and final results than the vanilla CoT distillation.

Motivation & Objective

• To address the limitations of existing LLMs in performing reliable temporal reasoning, which requires integrated knowledge of time, logic, and commonsense.
• To develop a framework that explicitly models temporal relationships through structured temporal graphs, improving reasoning accuracy beyond free-text chain-of-thought prompting.
• To create a fully controllable, minimal-supervision synthetic dataset for pre-training LLMs on text-to-temporal-graph alignment.
• To integrate symbolic reasoning with LLMs using bootstrapped reasoning steps and data augmentation to enhance reliability and performance on complex temporal tasks.

Proposed method

• The framework first translates natural language stories into temporal graphs using a rule-based pipeline, with events linked by temporal relations (start/end times, durations, order).
• A synthetic dataset, TempGraph QA, is constructed using templates on YAGO11k to generate aligned (story, tempgraph, question, answer) samples with high control and minimal supervision.
• The model is pre-trained on this synthetic dataset to improve temporal graph extraction, enabling transfer to downstream tasks.
• Symbolic reasoning is guided via chain-of-thought bootstrapping, generating reliable intermediate reasoning steps for supervised fine-tuning.
• Two data augmentation strategies are introduced to mitigate data scarcity in neuro-symbolic reasoning tasks.
• External knowledge (mathematical and commonsense) is integrated to enhance reasoning capabilities during inference.

Figure 1: Our TempGraph-LLM framework has two steps: text-to-tempgraph translation and neuro-symbolic reasoning.

Experimental results

Research questions

• RQ1 Can large language models be effectively trained to extract structured temporal graphs from natural language text with minimal supervision?
• RQ2 Does symbolic reasoning with structured intermediate steps improve the reliability and accuracy of LLMs in temporal reasoning compared to free-text chain-of-thought prompting?
• RQ3 To what extent does pre-training on a synthetic, fully controllable temporal QA dataset improve zero-shot or few-shot performance on downstream temporal reasoning tasks?
• RQ4 How do data augmentation and external knowledge integration affect the performance of LLMs in complex neuro-symbolic reasoning tasks?
• RQ5 Can a graph-based, neuro-symbolic framework generalize across diverse temporal reasoning benchmarks and outperform standard prompting and fine-tuning methods?

Key findings

• The TempGraph-LLM framework achieves a 79.5% exact match score on the TempGraph QA benchmark using Llama2-13B, outperforming baseline fine-tuned models by 14.9 percentage points.
• Incorporating symbolic reasoning with bootstrapped chain-of-thought steps improves performance by 5.9 percentage points over free-text CoT, demonstrating greater consistency and reliability.
• Data augmentation improves performance by 4.0 percentage points (from 75.3% to 79.7% EM) when combined with symbolic reasoning, highlighting its importance in low-data regimes.
• Pre-training on the synthetic TempGraph QA dataset transfers well to other tasks, showing that temporal graph learning generalizes beyond the training distribution.
• The integration of external knowledge (mathematical and commonsense) boosts performance by 2.1 percentage points (from 75.3% to 77.6% EM) when combined with symbolic reasoning.
• Ablation studies confirm that both the temporal graph representation and symbolic reasoning are critical: removing either leads to significant performance drops.

Figure 2: Performance comparison between different strategies of CoT generation on TempGraph QA.