[Paper Review] Rapid Customization for Event Extraction

Yee Seng Chan, Joshua Fasching+2 more|arXiv (Cornell University)|Sep 20, 2018

Topic Modeling12 references4 citations

TL;DR

This paper presents a minimally supervised system for rapidly customizing event extraction to new event types with under 10 minutes of human effort per type. It uses word embeddings and WordNet to expand initial triggers, applies distant supervision for automatic labeling, and leverages ACE-annotated arguments to train a generalizable argument extractor, achieving strong performance on 67 novel event types with minimal human intervention.

ABSTRACT

We present a system for rapidly customizing event extraction capability to find new event types and their arguments. The system allows a user to find, expand and filter event triggers for a new event type by exploring an unannotated corpus. The system will then automatically generate mention-level event annotation automatically, and train a Neural Network model for finding the corresponding event. Additionally, the system uses the ACE corpus to train an argument model for extracting Actor, Place, and Time arguments for any event types, including ones not seen in its training data. Experiments show that with less than 10 minutes of human effort per event type, the system achieves good performance for 67 novel event types. The code, documentation, and a demonstration video will be released as open source on github.com.

Motivation & Objective

• To reduce the human labor required to extend event extraction systems to new event types.
• To enable rapid customization of event extractors using minimal human input and automated expansion of triggers.
• To train a generalizable argument extraction model that works across novel event types using existing ACE annotations.
• To develop a user interface that accelerates trigger selection and filtering for new event types.
• To demonstrate practical utility of the system on a large set of 67 previously unseen event types.

Proposed method

• Users provide a few initial trigger words for a new event type.
• The system expands these triggers using word embeddings (high cosine similarity) and WordNet (hyponyms).
• Expanded triggers are filtered by the user to retain only relevant terms, forming the final trigger set.
• Distant supervision is applied to an unannotated corpus to automatically generate training examples for event triggers.
• A convolutional neural network (CNN) is trained on the automatically generated trigger examples for event detection.
• A separate argument model is trained on ACE-annotated data to extract Actor, Place, and Time arguments for any event type, including unseen ones.

Experimental results

Research questions

• RQ1 Can event extraction be rapidly customized for new event types with minimal human annotation?
• RQ2 How effective is trigger expansion via word embeddings and WordNet in improving coverage and accuracy?
• RQ3 Can a model trained on ACE data generalize to extract arguments for event types not present in the ACE training data?
• RQ4 To what extent does distant supervision combined with human pruning improve trigger detection performance?
• RQ5 How does the user interface contribute to reducing time and effort in the customization pipeline?

Key findings

• With less than 10 minutes of human effort per event type, the system achieves strong performance on 67 novel event types.
• The trigger model trained on distant supervision with human pruning (T_h) outperforms the model trained on only distant supervision (T_d), indicating that pruning improves generalization.
• The argument model (A_n) achieves an overall F1 score close to that of the model trained on the same data with gold triggers, demonstrating strong generalization to unseen event types.
• Human-validated predictions on 100 test arguments (62 Actor, 7 Place, 10 Time) achieved a precision of 0.79, confirming high reliability of the argument extraction component.
• The system's performance remains strong even when applied to event types not present in the ACE training data, validating the transferability of the argument model.
• The user interface significantly improves time efficiency by enabling interactive filtering and reassignment of triggers across event types.