[Paper Review] Improving Temporal Relation Extraction with a Globally Acquired Statistical Resource

Ning Qiang, Hao Wu+2 more|arXiv (Cornell University)|Apr 17, 2018

Data Management and Algorithms30 references4 citations

TL;DR

This paper introduces TemProb, a probabilistic knowledge base derived from 20 years of New York Times articles, which captures the typical temporal order between events. By integrating this globally acquired statistical resource into existing temporal relation extraction systems, the authors achieve significant performance improvements on the TimeBank-Dense benchmark, outperforming state-of-the-art methods even without gold event features.

ABSTRACT

Extracting temporal relations (before, after, overlapping, etc.) is a key aspect of understanding events described in natural language. We argue that this task would gain from the availability of a resource that provides prior knowledge in the form of the temporal order that events usually follow. This paper develops such a resource -- a probabilistic knowledge base acquired in the news domain -- by extracting temporal relations between events from the New York Times (NYT) articles over a 20-year span (1987--2007). We show that existing temporal extraction systems can be improved via this resource. As a byproduct, we also show that interesting statistics can be retrieved from this resource, which can potentially benefit other time-aware tasks. The proposed system and resource are both publicly available.

Motivation & Objective

• To address the lack of prior knowledge about typical event temporal order in temporal relation extraction systems.
• To construct a large-scale, probabilistic knowledge base of event temporal relations from a 20-year news corpus.
• To demonstrate that integrating global statistical knowledge improves the performance of existing temporal relation extraction systems.
• To provide a publicly available resource and system that benefits time-aware NLP tasks.

Proposed method

• Extracted temporal relations between events from over 1 million NYT articles (1987–2007) using an existing temporal relation extraction system.
• Constructed TemProb, a probabilistic knowledge base that records the frequency of before/after/overlapping relations between event pairs.
• Integrated TemProb into existing temporal relation extraction systems as a global prior, either locally or globally, to refine predictions.
• Applied the knowledge base in both local and global inference settings, using conditional random fields with prior probabilities from TemProb.
• Validated the method on the TimeBank-Dense (TBDense) benchmark, comparing performance with state-of-the-art systems.
• Performed naive augmentation of predictions on the full TBDense dataset by combining outputs from the proposed system and Ning et al. (2017).

Experimental results

Research questions

• RQ1 Can a globally acquired statistical resource of typical event temporal order improve temporal relation extraction?
• RQ2 How effective is prior knowledge about event sequence patterns in reducing ambiguity when event content is missing?
• RQ3 To what extent can TemProb enhance performance when gold event features (e.g., tense, aspect) are unavailable?
• RQ4 Can the integration of global statistical knowledge lead to state-of-the-art results on benchmark datasets?

Key findings

• The proposed system achieved an F1 score of 55.5 on the partial TimeBank-Dense test set, outperforming both CAEVO (47.6 F1) and Ning et al. (2017) (51.5 F1).
• Even without gold event features (tense, aspect, modality, polarity), the proposed system outperformed CAEVO and Ning et al. (2017) in both precision and F1.
• The McNemar’s test confirmed the improvement was statistically significant (p < 0.0005).
• Naive augmentation of the proposed system’s predictions with those from Ning et al. (2017) yielded a new state-of-the-art F1 score of 49.2 on the complete TBDense dataset.
• The awareness F1 score of the proposed system on the complete TBDense set reached 49.2, surpassing both CAEVO (45.7) and Ning et al. (2017) (48.5).
• TemProb demonstrated utility beyond the primary task, enabling retrieval of meaningful temporal statistics that could benefit other time-aware NLP applications.