[Paper Review] Document-level Event-based Extraction Using Generative Template-filling Transformers.

Xinya Du, Alexander M. Rush+1 more|arXiv (Cornell University)|Aug 21, 2020

Topic Modeling28 references13 citations

TL;DR

This paper proposes a generative transformer-based encoder-decoder framework for document-level event-based extraction that models context across sentence boundaries, implicitly captures coreference relations, and respects template structure. Evaluated on MUC-4, it achieves state-of-the-art performance by leveraging cross-sentence context and implicit linguistic knowledge, outperforming prior sentence-level methods.

ABSTRACT

We revisit the classic information extraction problem of document-level template filling. We argue that sentence-level approaches are ill-suited to the task and introduce a generative transformer-based encoder-decoder framework that is designed to model context at the document level: it can make extraction decisions across sentence boundaries; is \emph{implicitly} aware of noun phrase coreference structure, and has the capacity to respect cross-role dependencies in the template structure. We evaluate our approach on the MUC-4 dataset, and show that our model performs substantially better than prior work. We also show that our modeling choices contribute to model performance, e.g., by implicitly capturing linguistic knowledge such as recognizing coreferent entity mentions. Our code for the evaluation script and models will be open-sourced at https://github.com/xinyadu/doc_event_entity for reproduction purposes.

Motivation & Objective

• To address the limitations of sentence-level information extraction in capturing document-level context.
• To develop a method that can make extraction decisions across sentence boundaries.
• To implicitly model noun phrase coreference and cross-role dependencies in event templates.
• To improve template-filling performance by leveraging long-range context and structural awareness.
• To demonstrate that generative modeling can outperform traditional sequence labeling on document-level event extraction.

Proposed method

• Uses a transformer-based encoder-decoder architecture to generate event templates directly from document text.
• Leverages the autoregressive generation process to model dependencies across sentences and roles in the template.
• Employs a sequence-to-sequence framework that conditions on the full document context for event extraction.
• Implicitly captures coreference by learning mention representations during training without explicit coreference resolution modules.
• Models cross-role dependencies in templates through the autoregressive generation of slot values in a structured order.
• Trains end-to-end using a sequence generation objective, allowing the model to learn contextual and structural patterns.

Experimental results

Research questions

• RQ1 Can a generative sequence-to-sequence model outperform traditional sentence-level extraction on document-level event extraction?
• RQ2 To what extent can a generative model implicitly learn coreference resolution without explicit modeling?
• RQ3 How effective is the model at capturing cross-role dependencies in event templates?
• RQ4 Does modeling context at the document level significantly improve extraction performance?
• RQ5 What contribution do long-range context and structural awareness make to the final results?

Key findings

• The proposed model achieves state-of-the-art performance on the MUC-4 dataset, significantly outperforming prior sentence-level approaches.
• The model implicitly learns to recognize coreferent entity mentions without requiring explicit coreference resolution components.
• Cross-sentence context modeling leads to substantial improvements in event extraction accuracy.
• The generative framework effectively respects template structure and enforces cross-role dependencies during generation.
• Ablation studies confirm that document-level context and autoregressive generation are key contributors to performance gains.
• The model's performance is robust and generalizable, with results validated through open-sourced evaluation scripts and models.