[Paper Review] Scale Up Event Extraction Learning via Automatic Training Data Generation

Ying Zeng, Yansong Feng+5 more|arXiv (Cornell University)|Dec 11, 2017

Topic Modeling21 references3 citations

TL;DR

This paper proposes a distant supervision approach that automatically generates high-quality training data for event extraction by leveraging structured knowledge bases like Freebase and Wikipedia tables, eliminating the need for manual annotation. By using key arguments instead of explicit triggers, the method scales training data from thousands to hundreds of thousands of instances, enabling a neural model with post-inference to achieve state-of-the-art performance in multi-typed event detection with minimal human effort.

ABSTRACT

The task of event extraction has long been investigated in a supervised learning paradigm, which is bound by the number and the quality of the training instances. Existing training data must be manually generated through a combination of expert domain knowledge and extensive human involvement. However, due to drastic efforts required in annotating text, the resultant datasets are usually small, which severally affects the quality of the learned model, making it hard to generalize. Our work develops an automatic approach for generating training data for event extraction. Our approach allows us to scale up event extraction training instances from thousands to hundreds of thousands, and it does this at a much lower cost than a manual approach. We achieve this by employing distant supervision to automatically create event annotations from unlabelled text using existing structured knowledge bases or tables.We then develop a neural network model with post inference to transfer the knowledge extracted from structured knowledge bases to automatically annotate typed events with corresponding arguments in text.We evaluate our approach by using the knowledge extracted from Freebase to label texts from Wikipedia articles. Experimental results show that our approach can generate a large number of high quality training instances. We show that this large volume of training data not only leads to a better event extractor, but also allows us to detect multiple typed events.

Motivation & Objective

• To address the bottleneck of small, manually annotated training datasets in supervised event extraction.
• To eliminate the need for human-annotated event triggers by relying on key arguments from structured knowledge bases.
• To scale up training data for event extraction from thousands to hundreds of thousands of instances using distant supervision.
• To enable effective multi-typed event detection by leveraging large-scale automatically generated training data.
• To develop a neural network model with post-inference that does not require explicit trigger identification.

Proposed method

• Leverage structured knowledge bases (e.g., Freebase, Wikipedia tables) to identify key arguments that define event types.
• Apply distant supervision to automatically label unstructured text (e.g., Wikipedia articles) using these key arguments as proxies for event types.
• Design heuristics to identify the most informative key arguments from structured tables that best characterize event types.
• Develop a neural network architecture (BLSTM-CRF-ILP) that performs event extraction without relying on explicit event triggers.
• Use linear integer programming (ILP) in post-inference to refine predictions and ensure argument consistency.
• Generate training data by matching text spans to structured table entries based on named entity and temporal/syntactic clues.

Experimental results

Research questions

• RQ1 Can key arguments from structured knowledge bases be used to infer event types without explicit trigger annotations?
• RQ2 To what extent can distant supervision using structured tables generate high-quality training data for event extraction?
• RQ3 Can a neural model trained on automatically generated data detect multiple event types effectively?
• RQ4 How does the performance of the proposed method compare to human-annotated datasets in terms of F1 scores for event type classification and argument detection?
• RQ5 Can the model generalize to diverse event types such as business acquisitions, Olympic wins, and awards, using only table-based supervision?

Key findings

• The proposed method generated a training dataset 14 times larger than the widely used ACE Challenge dataset, constructed in hours rather than years.
• The automatically generated training data achieved F1 scores comparable to manually annotated data, with 87.0% F1 on acquisition events, 77.2% on Olympics events, and 95.0% on awards events.
• The model achieved 72.0% F1 in key argument detection (KAD) and 69.6% in all argument detection (AAD) on the TBWiki dataset, demonstrating strong generalization.
• The approach enabled effective multi-typed event detection, a capability rarely supported by prior methods.
• The use of key arguments instead of triggers allowed the model to detect events even in the absence of explicit event triggers, such as in sentences lacking a 'bankrupt' or 'filed' verb.
• The integration of ILP post-inference significantly improved argument consistency and overall extraction quality.