[Paper Review] TRAM: Benchmarking Temporal Reasoning for Large Language Models

Yuqing Wang, Yun Zhao|arXiv (Cornell University)|Oct 2, 2023

Topic Modeling4 citations

TL;DR

TRAM is a comprehensive benchmark of 10 temporal reasoning tasks—spanning duration, frequency, arithmetic, causality, and temporal relations—comprising 526.1k multiple-choice questions derived from diverse sources. Evaluated across LLMs like GPT-4 and Llama2, it reveals that even state-of-the-art models fall short of human performance, with GPT-4 achieving 87.8% average accuracy, highlighting persistent challenges in understanding implicit temporal cues.

ABSTRACT

Reasoning about time is essential for understanding the nuances of events described in natural language. Previous research on this topic has been limited in scope, characterized by a lack of standardized benchmarks that would allow for consistent evaluations across different studies. In this paper, we introduce TRAM, a temporal reasoning benchmark composed of ten datasets, encompassing various temporal aspects of events such as order, arithmetic, frequency, and duration, designed to facilitate a comprehensive evaluation of the TeR capabilities of large language models (LLMs). We evaluate popular LLMs like GPT-4 and Llama2 in zero-shot and few-shot scenarios, and establish baselines with BERT-based and domain-specific models. Our findings indicate that the best-performing model lags significantly behind human performance. It is our aspiration that TRAM will spur further progress in enhancing the TeR capabilities of LLMs.

Motivation & Objective

• To address the lack of a unified, standardized benchmark for evaluating temporal reasoning in large language models (LLMs).
• To comprehensively assess LLMs across foundational and advanced temporal reasoning capabilities, including duration, frequency, arithmetic, causality, and temporal relations.
• To establish a benchmark that supports consistent evaluation across diverse LLM architectures and prompting strategies (zero-shot, few-shot, chain-of-thought).
• To identify persistent limitations in current LLMs, especially in interpreting implicit temporal cues and nuanced temporal narratives.

Proposed method

• TRAM integrates ten distinct temporal reasoning tasks derived from existing datasets, human-crafted templates, web sources, and programmatic generation.
• Each task is structured as a multiple-choice question to enable standardized, zero-shot evaluation of LLMs.
• Questions are derived from diverse sources: TimeBank and TempEval-3 for temporal relations, SNLI/MNLI for temporal NLI, COPA-inspired for causality, and ROCStories/SCT for storytelling.
• Expert annotations and programmatic generation are used to produce answers, ensuring high-quality and consistent labeling.
• The benchmark includes 38 subtasks and 526.1k questions, covering explicit and implicit temporal expressions across various contexts.
• Models are evaluated using zero-shot, few-shot, and chain-of-thought prompting, with BERT-style models fine-tuned on limited data.

Experimental results

Research questions

• RQ1 How well do current large language models perform on a unified, comprehensive benchmark of temporal reasoning tasks?
• RQ2 What are the relative strengths and weaknesses of different LLM architectures (e.g., GPT-4, Llama2, PaLM2) in understanding temporal duration, frequency, and arithmetic?
• RQ3 To what extent do LLMs struggle with implicit temporal cues, such as causal relationships or contextual time references?
• RQ4 How does chain-of-thought prompting affect performance on complex temporal reasoning tasks compared to standard prompting?
• RQ5 How far do state-of-the-art LLMs fall short of human-level performance in temporal reasoning?

Key findings

• GPT-4 achieves the highest average accuracy of 87.8% across TRAM tasks, outperforming other LLMs and BERT-based models.
• Despite strong performance, GPT-4 still lags behind human performance by approximately 10%, indicating significant room for improvement.
• Models consistently struggle with implicit temporal cues, such as temporal causality and nuanced event sequencing, even in advanced models.
• Performance disparities exist across task types, with arithmetic and causality tasks posing particular challenges.
• Manual error analysis reveals that models often misinterpret temporal relations when cues are implicit or context-dependent.
• The benchmark exposes that current LLMs lack robust understanding of temporal narratives and event-based reasoning, especially in real-world, context-rich scenarios.