[Paper Review] Temporal Reasoning on Implicit Events from Distant Supervision

Ben Zhou, Kyle Richardson+4 more|arXiv (Cornell University)|Oct 24, 2020

Topic Modeling47 references4 citations

TL;DR

This paper introduces Tracie, a new dataset for evaluating temporal reasoning on implicit events—events not explicitly stated but inferable through commonsense reasoning. The authors propose SymTime, a neuro-symbolic model that combines distant supervision from large-scale text with symbolic reasoning over start times and durations to infer end times, achieving 78.9% accuracy on Tracie, a 5% improvement over strong baselines and 11% in zero-shot settings.

ABSTRACT

We propose TRACIE, a novel temporal reasoning dataset that evaluates the degree to which systems understand implicit events -- events that are not mentioned explicitly in natural language text but can be inferred from it. This introduces a new challenge in temporal reasoning research, where prior work has focused on explicitly mentioned events. Human readers can infer implicit events via commonsense reasoning, resulting in a more comprehensive understanding of the situation and, consequently, better reasoning about time. We find, however, that state-of-the-art models struggle when predicting temporal relationships between implicit and explicit events. To address this, we propose a neuro-symbolic temporal reasoning model, SYMTIME, which exploits distant supervision signals from large-scale text and uses temporal rules to combine start times and durations to infer end times. SYMTIME outperforms strong baseline systems on TRACIE by 5%, and by 11% in a zero prior knowledge training setting. Our approach also generalizes to other temporal reasoning tasks, as evidenced by a gain of 1%-9% on MATRES, an explicit event benchmark.

Motivation & Objective

• Address the gap in temporal reasoning research by focusing on implicit events, which are not explicitly mentioned but can be inferred from context.
• Develop a benchmark dataset, Tracie, to evaluate systems' ability to reason about both start and end times of implicit events.
• Improve generalization in low-resource settings by leveraging distant supervision from large-scale text to pre-train models on temporal patterns.
• Design a neuro-symbolic model that combines learned predictions of start-time distances and event durations to infer end-time relations using symbolic logic.
• Demonstrate robustness and transferability of the approach on both implicit-event and explicit-event temporal reasoning benchmarks.

Proposed method

• Construct Tracie, a dataset of 5,400 human-curated instances in a multi-premise textual entailment format, focusing on temporal relations between implicit and explicit events.
• Apply distant supervision by extracting temporal patterns—specifically, start-time distances and durations—from large-scale text across paragraph-level windows, not just sentence-level.
• Train a pattern-based model, PtnTime, using these distant signals to improve pre-training for temporal reasoning, especially on start-time prediction.
• Develop SymTime, a neuro-symbolic model that decomposes end-time reasoning into two components: predicting relative start-time distances and estimating event durations.
• Use symbolic computation based on Allen interval algebra to combine predictions: end time = start time + duration, enabling logical inference of temporal order.
• Train SymTime end-to-end with task-specific fine-tuning, but also evaluate it in zero-shot and low-resource settings to assess generalization.

Experimental results

Research questions

• RQ1 Can existing pre-trained language models effectively reason about the start and end times of implicit events?
• RQ2 To what extent can distant supervision from large-scale text improve temporal reasoning on implicit events, especially in low-resource settings?
• RQ3 Can symbolic reasoning over start-time distances and durations improve end-time inference beyond end-to-end neural models?
• RQ4 How does the integration of symbolic temporal rules enhance model robustness and generalization across different data distributions?
• RQ5 Does the proposed method generalize to other temporal reasoning benchmarks involving only explicit events?

Key findings

• The T5-Large model fine-tuned on Tracie achieves only 67.9% binary accuracy, indicating significant limitations in reasoning about implicit events.
• PtnTime, a model pre-trained with distant supervision on paragraph-level temporal patterns, improves accuracy to 76.6% on Tracie, a 9% gain over standard T5-Large.
• SymTime, which combines PtnTime with a duration model and applies symbolic reasoning, achieves 78.9% accuracy on Tracie, a 5% improvement over strong baselines.
• In a zero-prior-knowledge training setting, SymTime improves by 11% over baselines, demonstrating strong generalization from distant supervision.
• SymTime achieves gains of 1% to 9% on MATRES, a benchmark for explicit events, showing transferability to other temporal reasoning tasks.
• The model shows greater robustness to input distribution shifts, with only a 4% performance drop in the no-story setting, compared to a 10% drop for T5-Large.