[Paper Review] TempEval-3: Evaluating Events, Time Expressions, and Temporal Relations

Naushad UzZaman, Héctor Llorens+4 more|arXiv (Cornell University)|Jun 22, 2012

Constraint Satisfaction and Optimization7 references85 citations

TL;DR

This paper proposes TempEval-3, a large-scale shared evaluation task for temporal information processing in NLP, integrating a 500K-token silver-standard corpus and 100K-token gold-standard dataset. It advances prior work by unifying event, temporal expression, and temporal relation extraction into a single end-to-end task using TimeML standards, with a novel temporal awareness score for holistic system evaluation.

ABSTRACT

We describe the TempEval-3 task which is currently in preparation for the SemEval-2013 evaluation exercise. The aim of TempEval is to advance research on temporal information processing. TempEval-3 follows on from previous TempEval events, incorporating: a three-part task structure covering event, temporal expression and temporal relation extraction; a larger dataset; and single overall task quality scores.

Motivation & Objective

• To advance research in temporal information processing by creating a large-scale, standardized evaluation framework for temporal relation extraction.
• To explore the utility of combining automatically generated 'silver' standard data with human-annotated 'gold' standard data for training and evaluation.
• To unify event, temporal expression, and temporal relation extraction into a single, integrated task to better reflect real-world NLP challenges.
• To improve system evaluation by introducing a temporal awareness score that combines performance across all three subtasks.
• To provide a larger, more diverse dataset than previous TempEval iterations, enabling more robust model training and evaluation.

Proposed method

• The dataset comprises 500K tokens of automatically annotated 'silver' data generated using state-of-the-art systems (TIPSem, TIPSem-B, TRIOS) trained on TimeBank and AQUAINT.
• A merging algorithm combines outputs from three systems using weighted voting (TIPSem: 0.36, TIPSem-B: 0.32, TRIOS: 0.32) to produce a consensus annotation.
• A portion of the silver data is human-reviewed and released as additional gold-standard training data.
• The evaluation uses a graph-based metric that accounts for temporal closure, improving precision and recall for relation annotations.
• A new 'temporal awareness score' aggregates F1-scores across event, temporal expression, and relation extraction tasks into a single holistic metric.
• All data follows the ISO-TimeML schema, with standardized elements: DOCID, DCT, TITLE, TEXT, TIMEX3, EVENT, and TLINK tags.

Experimental results

Research questions

• RQ1 Can large-scale automatically generated silver-standard corpora effectively support training and evaluation in temporal relation extraction, reducing reliance on costly human annotation?
• RQ2 How does the integration of silver and gold-standard data impact system performance on event, temporal expression, and temporal relation extraction tasks?
• RQ3 To what extent does a unified end-to-end task structure (extracting events, temporal expressions, and relations together) improve system performance compared to isolated subtasks?
• RQ4 How effective is the proposed temporal awareness score in capturing overall system capability across multiple temporal NLP components?
• RQ5 Does using the full set of TimeML temporal relation types (e.g., before, after, includes, during) lead to more nuanced and realistic evaluation than coarse-grained relation sets?

Key findings

• The TempEval-3 dataset includes approximately 500K tokens of automatically generated silver-standard data and 100K tokens of gold-standard data, significantly expanding the scale of prior TempEval evaluations.
• The silver-standard data was generated by merging outputs from three state-of-the-art systems (TIPSem, TIPSem-B, TRIOS) using a weighted voting strategy with optimal weights determined empirically.
• A portion of the silver data is human-reviewed and released as additional gold-standard training data, enhancing data quality without sacrificing scale.
• The evaluation framework introduces a graph-based metric that accounts for temporal closure, improving the robustness of relation annotation evaluation.
• The temporal awareness score provides a unified performance metric that combines F1-scores from event, temporal expression, and relation extraction tasks, enabling holistic system comparison.
• The task structure requires participants to perform event and temporal expression extraction as prerequisites for relation annotation, reflecting real-world system pipelines.