[Paper Review] TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

Qiang Ning, Hao Wu+4 more|arXiv (Cornell University)|May 1, 2020

Topic Modeling39 references4 citations

TL;DR

TORQUE is a new reading comprehension dataset comprising 3.2K news snippets and 21K human-annotated questions targeting temporal ordering between events. It evaluates models' ability to infer event sequences using natural language questions, with RoBERTa-large achieving 51% exact-match—30% below human performance, highlighting a critical gap in temporal reasoning for NLP systems.

ABSTRACT

A critical part of reading is being able to understand the temporal relationships between events described in a passage of text, even when those relationships are not explicitly stated. However, current machine reading comprehension benchmarks have practically no questions that test temporal phenomena, so systems trained on these benchmarks have no capacity to answer questions such as "what happened before/after [some event]?" We introduce TORQUE, a new English reading comprehension benchmark built on 3.2k news snippets with 21k human-generated questions querying temporal relationships. Results show that RoBERTa-large achieves an exact-match score of 51% on the test set of TORQUE, about 30% behind human performance.

Motivation & Objective

• Address the lack of temporal reasoning evaluation in existing reading comprehension benchmarks, which predominantly focus on coreference and entity relations.
• Develop a large-scale, natural language question-answering dataset that tests systems' ability to infer temporal order between events, even when relationships are implicit.
• Create a scalable, high-quality crowdsourcing pipeline to generate questions with contrasting temporal keywords (e.g., before/after/during) to test model sensitivity to subtle linguistic cues.
• Provide a benchmark that captures complex temporal phenomena such as event duration, multiple event sequences, and temporal scope beyond sentence boundaries.
• Enable future research into improving neural models' temporal reasoning by offering a public dataset and codebase for reproducible evaluation.

Proposed method

• Collect 3.2K news snippets from the TempEval3 dataset, focusing on narratives with multiple events and temporal dynamics.
• Use crowd workers to annotate events in the text using a defined event schema, identifying predicates and their arguments as events.
• Generate 21K human-written questions querying temporal relations (e.g., 'what happened before X?') using natural language, with a focus on temporal keywords like before, after, during.
• Include contrastive question pairs (e.g., changing 'before' to 'after') to test model robustness to small lexical changes.
• Validate 20% of the data with additional crowd workers to form a test set, ensuring high-quality annotations and reducing data collection artifacts.
• Fine-tune state-of-the-art models like RoBERTa-large on the dataset and evaluate using exact-match accuracy on the test set.

Experimental results

Research questions

• RQ1 Can existing reading comprehension models accurately infer temporal order between events when relationships are implicit or require cross-sentence reasoning?
• RQ2 How sensitive are current models to subtle changes in temporal keywords (e.g., 'before' vs. 'after') in natural language questions?
• RQ3 To what extent do models fail to understand event duration, ongoing states, or temporal scope beyond the immediate context?
• RQ4 How does the performance of neural models on temporal reasoning compare to human performance, and what are the key failure modes?
• RQ5 Can a crowdsourcing pipeline reliably generate high-quality, diverse, and contrastive temporal reasoning questions at scale?

Key findings

• RoBERTa-large achieves an exact-match score of 51% on the Torque test set, indicating a significant performance gap compared to human performance.
• Human performance on the test set is approximately 81% exact-match, showing that the task is challenging and not yet solved by current models.
• Models trained on SQuAD fail to generalize to temporal reasoning, as shown by their inability to correctly answer contrastive questions like 'what happened before' vs. 'what happened after'.
• The performance gap persists even when fine-tuning on Torque, suggesting that current architectures lack sufficient inductive bias for temporal reasoning.
• The dataset reveals that models often fail to identify the correct temporal event sequence when multiple events are present or when temporal cues are embedded in complex sentence structures.
• The contrastive question design effectively exposes model brittleness, as small changes in question wording (e.g., 'before' to 'after') lead to drastically different answers in model outputs.