[Paper Review] Event Detection with Neural Networks: A Rigorous Empirical Evaluation

J. Walker Orr, Prasad Tadepalli+1 more|arXiv (Cornell University)|Aug 26, 2018

Topic Modeling18 references4 citations

TL;DR

This paper proposes a novel DAG-GRU model that integrates syntactic dependency information and temporal context via an attention mechanism for event detection. Empirical evaluation on ACE2005 shows the model is competitive with state-of-the-art methods, but performance is highly sensitive to random initialization and data splits, suggesting the need to move beyond single-split evaluations for reliable model comparison.

ABSTRACT

Detecting events and classifying them into predefined types is an important step in knowledge extraction from natural language texts. While the neural network models have generally led the state-of-the-art, the differences in performance between different architectures have not been rigorously studied. In this paper we present a novel GRU-based model that combines syntactic information along with temporal structure through an attention mechanism. We show that it is competitive with other neural network architectures through empirical evaluations under different random initializations and training-validation-test splits of ACE2005 dataset.

Motivation & Objective

• To develop a neural network model that effectively integrates syntactic and temporal information for event detection.
• To evaluate the performance of the proposed DAG-GRU model against existing architectures under varying random initializations and data splits.
• To investigate the impact of data split variations and model initialization on performance stability and ranking of models.
• To challenge the validity of single-split evaluations in event detection by quantifying variability in model performance.
• To advocate for more robust evaluation protocols in future research, emphasizing reproducibility and statistical reliability.

Proposed method

• The DAG-GRU model extends standard GRUs by operating on a dependency parse graph, using attention to aggregate syntactic dependencies.
• It applies bidirectional GRU encoding over the dependency graph structure, allowing context and syntactic relationships to inform hidden state representations.
• An attention mechanism dynamically weights contributions from syntactically related words, enhancing representation learning for event triggers.
• The model is trained end-to-end on the ACE2005 dataset using standard train/dev/test splits and evaluated via multi-initialization and multi-split experiments.
• Performance is assessed using macro-F1 scores across event types, with model selection based on development set performance.
• A bootstrap resampling method and randomized cross-validation across 10 splits are used to quantify variance and confidence intervals.

Experimental results

Research questions

• RQ1 How does the proposed DAG-GRU model perform compared to existing state-of-the-art models in event detection under standard evaluation protocols?
• RQ2 How sensitive is model performance to random weight initialization across multiple runs?
• RQ3 How does performance vary across different random train/validation/test splits of the ACE2005 dataset?
• RQ4 To what extent does the inclusion of syntactic information via attention improve detection accuracy, especially in non-newswire domains?
• RQ5 Does the standard single-split evaluation protocol lead to misleading model rankings due to high variance?

Key findings

• The DAG-GRU model achieves competitive performance, with a mean F1 score of 69.2% across 20 random initializations, outperforming several baselines.
• Model performance varies significantly across random initializations, with standard deviation reaching up to 0.96% for DAG-GRU B, indicating high sensitivity.
• When evaluated across 10 randomized data splits, performance variance increased substantially, with standard deviations rising to 2.63% for DAG-GRU B, indicating high instability.
• The standard ACE2005 split, which uses only newswire documents for testing, leads to overly optimistic performance estimates due to domain mismatch with training data.
• The ranking of models based on the standard split differs from rankings based on average performance across multiple splits, undermining the reliability of single-split evaluations.
• The study demonstrates that variance from data splits and initialization exceeds typical performance gains from architectural improvements, highlighting the need for more robust evaluation practices.