[Paper Review] An Improved Neural Baseline for Temporal Relation Extraction

Ning Qiang, Sanjay Subramanian+1 more|arXiv (Cornell University)|Sep 1, 2019

Topic Modeling45 references4 citations

TL;DR

This paper proposes a strong neural baseline for temporal relation extraction using contextualized embeddings, a Siamese encoder for temporal common sense knowledge, and integer linear programming for global inference. It achieves a 10% absolute F1 improvement (25% error reduction) over the prior state-of-the-art on two benchmark datasets, demonstrating that model performance is limited more by data quality than architecture.

ABSTRACT

Determining temporal relations (e.g., before or after) between events has been a challenging natural language understanding task, partly due to the difficulty to generate large amounts of high-quality training data. Consequently, neural approaches have not been widely used on it, or showed only moderate improvements. This paper proposes a new neural system that achieves about 10% absolute improvement in accuracy over the previous best system (25% error reduction) on two benchmark datasets. The proposed system is trained on the state-of-the-art MATRES dataset and applies contextualized word embeddings, a Siamese encoder of a temporal common sense knowledge base, and global inference via integer linear programming (ILP). We suggest that the new approach could serve as a strong baseline for future research in this area.

Motivation & Objective

• To address the limited performance of neural models in temporal relation extraction due to low-quality training data.
• To evaluate whether neural architectures or data quality is the primary bottleneck in this task.
• To establish a strong, publicly available neural baseline for future research in temporal relation extraction.
• To investigate the impact of different word embedding techniques and knowledge injection on model performance.

Proposed method

• Uses a Bi-LSTM encoder with contextualized word embeddings (ELMo, BERT) to represent sentence pairs containing events.
• Employs a Siamese neural network to encode temporal common sense knowledge from TemProb, injecting prior knowledge about typical event orderings.
• Applies integer linear programming (ILP) for global inference to enforce transitivity constraints across predicted temporal relations.
• Combines event representation with position-aware encoding and concatenation strategies to improve modeling of relative event order.
• Trains end-to-end on the MATRES dataset, which features higher annotation quality and inter-annotator agreement than prior datasets.
• Uses a three-metric evaluation framework including accuracy, precision/recall/F1, and awareness score to comprehensively assess performance.

Experimental results

Research questions

• RQ1 Does the performance gap between neural and feature-based models in temporal relation extraction stem from model architecture or data quality?
• RQ2 How do different contextualized word embeddings (e.g., BERT, ELMo) compare in performance for temporal relation extraction?
• RQ3 To what extent can injecting external temporal common sense knowledge improve neural model performance?
• RQ4 Can global inference via integer linear programming improve consistency and accuracy of temporal relation predictions?
• RQ5 Is the proposed system a strong, generalizable baseline for future work in temporal relation extraction?

Key findings

• The proposed system achieves a 10% absolute improvement in F1 score and 25% error reduction over the previous state-of-the-art CogCompTime system on the MATRES dataset.
• Contextualized embeddings (ELMo and BERT) significantly outperform static embeddings (word2vec, GloVe, FastText), with p < 0.001 in McNemar’s test.
• The addition of a common sense encoder (CSE) improves performance over baseline concatenation by p < 0.001, regardless of whether ELMo or BERT is used.
• The model achieves an F1 score of 78.4 on MATRES and 74.9 on TCR when using BERT with the full pipeline, significantly outperforming CogCompTime.
• The study confirms that high-quality data (like MATRES) enables neural models to outperform previous feature-based systems, indicating data quality is a key bottleneck.
• No significant performance difference was found between ELMo and BERT, despite BERT’s dominance in other NLP tasks.