[Paper Review] ManTIME: Temporal expression identification and normalization in the TempEval-3 challenge

Michele Filannino, Gavin Brown+1 more|arXiv (Cornell University)|Apr 30, 2013

Natural Language Processing TechniquesComputer Science7 references17 citations

TL;DR

ManTIME is a CRF-based system with a post-processing pipeline for temporal expression identification and normalization in the TempEval-3 challenge. It achieves 0.90 F1 in identification and 0.84 type accuracy in normalization, with WordNet features harming performance and silver data failing to improve results despite extensive use.

ABSTRACT

This paper describes a temporal expression identification and normalization system, ManTIME, developed for the TempEval-3 challenge. The identification phase combines the use of conditional random fields along with a post-processing identification pipeline, whereas the normalization phase is carried out using NorMA, an open-source rule-based temporal normalizer. We investigate the performance variation with respect to different feature types. Specifically, we show that the use of WordNet-based features in the identification task negatively affects the overall performance, and that there is no statistically significant difference in using gazetteers, shallow parsing and propositional noun phrases labels on top of the morphological features. On the test data, the best run achieved 0.95 (P), 0.85 (R) and 0.90 (F1) in the identification phase. Normalization accuracies are 0.84 (type attribute) and 0.77 (value attribute). Surprisingly, the use of the silver data (alone or in addition to the gold annotated ones) does not improve the performance.

Motivation & Objective

• To develop a robust system for temporal expression identification and normalization in the TempEval-3 challenge.
• To evaluate the impact of various feature types—morphological, syntactic, gazetteers, and WordNet—on identification performance.
• To assess whether silver-standard data improve system performance compared to gold-annotated data alone.
• To investigate the effectiveness of a post-processing pipeline in enhancing CRF-based identification results.
• To evaluate normalization accuracy using an updated rule-based system, NorMA.

Proposed method

• The identification phase uses Linear Conditional Random Fields (CRF) with a BIO labeling scheme and a custom factor graph topology.
• Features include morphological (e.g., word form, lemma, pattern, regular expressions), syntactic (chunks, propositional noun phrases), gazetteers (e.g., cities, nationalities), and WordNet-based features (e.g., senses, hypernyms, hyponyms).
• A post-processing pipeline applies three modules: probabilistic correction, BIO fixer (corrects invalid label transitions), and threshold-based label switching (uses prior probabilities >0.87 to override CRF predictions).
• The normalization phase uses NorMA, an open-source rule-based normalizer, to map temporal expressions to TimeML format for type and value attributes.
• Model selection and hyperparameter tuning used 5×10-fold cross-validation on a mixed training set of gold and silver data, with seed=490 for shuffling.
• The final system was evaluated on the TempEval-3 test set using F1, precision, recall, and value/type accuracy metrics.

Experimental results

Research questions

• RQ1 Does the inclusion of WordNet-based features improve temporal expression identification performance in a CRF-based system?
• RQ2 Do syntactic and gazetteer features significantly enhance identification beyond morphological features?
• RQ3 Can silver-standard data from the TE3Silver corpus improve system performance when used in training?
• RQ4 To what extent does a post-processing pipeline enhance precision and recall in temporal expression identification?
• RQ5 What is the impact of training data composition (gold vs. silver) on normalization accuracy for temporal type and value attributes?

Key findings

• The best-performing run achieved 0.95 precision, 0.85 recall, and 0.90 F1 in the identification phase using gold-annotated data and the post-processing pipeline.
• The use of WordNet-based features significantly degraded identification performance, with a p-value of 0.0054 from ANOVA, indicating a statistically significant negative impact.
• Syntactic and gazetteer features did not yield a statistically significant performance difference compared to morphological features alone, suggesting limited added value.
• Normalization achieved 0.84 type accuracy and 0.77 value accuracy, indicating challenges in handling novel lexical patterns and value interpretation.
• Silver data did not improve performance when used alone or in combination with gold data, despite being 7x larger, suggesting potential noise or distributional mismatch.
• The post-processing pipeline significantly improved F1 scores by 2.27% (p=3.51×10−23), with the highest gains observed on gold-annotated data, demonstrating its effectiveness in refining CRF predictions.