[Paper Review] Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning

Oscar Sainz, Itziar González-Dios+3 more|arXiv (Cornell University)|May 3, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a textual entailment-based framework for Event Argument Extraction (EAE) that reduces annotation needs by 90% in zero-shot and few-shot settings through multi-source pre-training on multiple NLI datasets. By recasting EAE as entailment with verbalized templates, the method achieves state-of-the-art performance on ACE and WikiEvents while minimizing schema dependency and enabling cross-domain transfer with minimal human effort.

ABSTRACT

Recent work has shown that NLP tasks such as Relation Extraction (RE) can be recasted as Textual Entailment tasks using verbalizations, with strong performance in zero-shot and few-shot settings thanks to pre-trained entailment models. The fact that relations in current RE datasets are easily verbalized casts doubts on whether entailment would be effective in more complex tasks. In this work we show that entailment is also effective in Event Argument Extraction (EAE), reducing the need of manual annotation to 50% and 20% in ACE and WikiEvents respectively, while achieving the same performance as with full training. More importantly, we show that recasting EAE as entailment alleviates the dependency on schemas, which has been a road-block for transferring annotations between domains. Thanks to the entailment, the multi-source transfer between ACE and WikiEvents further reduces annotation down to 10% and 5% (respectively) of the full training without transfer. Our analysis shows that the key to good results is the use of several entailment datasets to pre-train the entailment model. Similar to previous approaches, our method requires a small amount of effort for manual verbalization: only less than 15 minutes per event argument type is needed, and comparable results can be achieved with users with different level of expertise.

Motivation & Objective

• To reduce the high annotation cost of Event Argument Extraction (EAE) by leveraging textual entailment as a reformulation of the task.
• To minimize dependency on rigid, domain-specific schemas by enabling transfer between different event schemas such as ACE and WikiEvents.
• To evaluate the effectiveness of multi-source NLI pre-training (beyond just MNLI) for improving entailment-based EAE performance.
• To quantify the time and effort trade-off between writing verbalization templates and manual annotation, identifying an optimal balance.

Proposed method

• Reformulate EAE as a textual entailment task by creating natural language hypotheses (verbalizations) for each event argument type, using templates that structure input as (event trigger, candidate argument, role type).
• Fine-tune a pre-trained entailment model on multiple NLI datasets—such as MNLI, SNLI, FEVER, and ANLI—to improve generalization and robustness over single-dataset pre-training.
• Use the entailment model to predict the most likely argument role by scoring the entailment probability between a premise (sentence context) and a hypothesis (template with candidate argument).
• Enable zero-shot and few-shot transfer between ACE and WikiEvents by reusing templates and pre-trained models without re-annotating data, reducing annotation needs to just 5–10% of full training.
• Apply multi-source learning by jointly pre-training on diverse NLI datasets to enhance zero-shot and few-shot generalization, outperforming models fine-tuned only on MNLI.
• Use a human-in-the-loop approach for template creation, with minimal domain expertise required, and validate performance using both expert and novice template writers.

Experimental results

Research questions

• RQ1 Can textual entailment be effectively applied to Event Argument Extraction, a more complex task than traditional relation extraction, with strong zero-shot and few-shot performance?
• RQ2 Does multi-source pre-training on multiple NLI datasets significantly improve entailment-based EAE performance compared to single-dataset fine-tuning (e.g., MNLI only)?
• RQ3 To what extent can annotation effort be reduced by shifting from manual data annotation to template writing, and what is the optimal allocation of time between the two?
• RQ4 Can cross-schema transfer between ACE and WikiEvents be achieved without manual re-annotation, and how much does it reduce overall annotation cost?
• RQ5 Does the quality of templates depend on the annotator’s expertise, or can non-experts produce effective verbalizations with comparable performance?

Key findings

• The entailment-based EAE system achieves state-of-the-art performance on both ACE and WikiEvents, matching full-supervision results with only 20% of the training data in few-shot settings.
• Using multiple NLI datasets for pre-training (e.g., MNLI, SNLI, FEVER, ANLI) leads to significantly better performance than using MNLI alone, especially in zero-shot and few-shot regimes.
• Template creation requires less than 15 minutes per event argument type and yields comparable results even when written by novice users, indicating low expertise barrier.
• The method reduces annotation needs to just 10% of the full training data when transferring from ACE to WikiEvents and 5% when transferring from WikiEvents to ACE, with no additional manual labeling.
• Annotating 5% of ACE data takes approximately 9 hours and yields 37.5 F1, while spending 5 hours on template creation yields 40.6 F1 in zero-shot, demonstrating that template writing is more efficient than annotation.
• The performance curve shows diminishing returns from extended template writing, suggesting a sweet spot where 5 hours of template creation and 9 hours of annotation yield optimal results (F1 = 57).