[Paper Review] Effective Distant Supervision for Temporal Relation Extraction

Xinyu Zhao, Shih-Ting Lin+1 more|arXiv (Cornell University)|Oct 24, 2020

Topic Modeling22 references4 citations

TL;DR

This paper proposes a novel distant supervision method for temporal relation extraction that automatically collects and labels event pairs using explicit temporal cues (e.g., 'before', 'after', timexes), then masks these cues to force models to learn implicit contextual signals. A pre-trained Transformer model fine-tuned on this masked data achieves strong zero-shot and few-shot transfer performance on human-annotated benchmarks, with RoBERTa reaching 79.8 F1 on MATRES, demonstrating effective generalization from synthetic, distantly-labeled data.

ABSTRACT

A principal barrier to training temporal relation extraction models in new domains is the lack of varied, high quality examples and the challenge of collecting more. We present a method of automatically collecting distantly-supervised examples of temporal relations. We scrape and automatically label event pairs where the temporal relations are made explicit in text, then mask out those explicit cues, forcing a model trained on this data to learn other signals. We demonstrate that a pre-trained Transformer model is able to transfer from the weakly labeled examples to human-annotated benchmarks in both zero-shot and few-shot settings, and that the masking scheme is important in improving generalization.

Motivation & Objective

• To address the scarcity of high-quality, diverse training data for temporal relation extraction in new domains.
• To develop an automatic method for generating distantly-labeled examples without relying on external knowledge bases.
• To improve model generalization by masking explicit temporal indicators, forcing reliance on contextual cues.
• To evaluate whether pre-trained models can effectively transfer from automatically collected, masked examples to human-annotated benchmarks.
• To compare the effectiveness of different distant supervision signal sources—discourse connectives vs. timex-anchored events.

Proposed method

• Collect event pairs from English Gigaword using heuristic rules to identify explicit temporal cues, such as discourse connectives (e.g., 'before', 'after') and timexes (e.g., 'yesterday', 'now').
• Automatically label event pairs based on these cues, creating a large-scale, distantly-supervised dataset.
• Apply a masking strategy that hides the explicit temporal indicators (e.g., 'before', 'after', '2020') in the input text, forcing the model to infer relations from context alone.
• Fine-tune a pre-trained RoBERTa model on the masked examples using a classification head that combines event embeddings via concatenation, elementwise multiplication, and subtraction.
• Use the model to perform inference on human-annotated benchmarks like MATRES in zero-shot and few-shot settings.
• Evaluate performance using F1 scores and compare with baselines and the majority class.

Experimental results

Research questions

• RQ1 Can a pre-trained Transformer model effectively generalize from automatically collected, distantly-labeled temporal relation examples to human-annotated benchmarks?
• RQ2 How does masking explicit temporal cues affect model generalization and performance on downstream tasks?
• RQ3 Which source of distant supervision—discourse connectives or timex-anchored events—produces better generalization in low-resource settings?
• RQ4 Does the distribution of events in the distant data (e.g., verb types, semantics) affect transfer performance to human-annotated datasets?
• RQ5 Can the proposed label-and-mask paradigm be used to create effective training data for other NLP tasks beyond temporal relation extraction?

Key findings

• Masking explicit temporal cues during training significantly improves generalization on the MATRES benchmark, especially when using timex-anchored examples.
• The model achieves an F1 score of 79.8 on MATRES using RoBERTa, matching state-of-the-art performance from prior local models.
• The DistantTimex method, which uses timex-anchored events, outperforms the BeforeAfter method based on discourse connectives, particularly in zero-shot and few-shot transfer.
• The event distribution in the DistantTimex dataset is more diverse and semantically complex than in the BeforeAfter dataset, which may contribute to better generalization.
• The model trained on masked DistantTimex data generalizes better than the majority baseline, indicating it learns meaningful, implicit temporal cues rather than relying on trivial signals.
• The approach enables effective zero-shot and few-shot transfer to human-annotated benchmarks, demonstrating the viability of synthetic data for low-resource temporal relation extraction.