[Paper Review] A Unified Generative Framework for Various NER Subtasks

Hang Yan, Tao Gui+4 more|arXiv (Cornell University)|Jun 2, 2021

Topic Modeling64 references4 citations

TL;DR

This paper proposes a unified generative framework for flat, nested, and discontinuous NER subtasks by formulating NER as entity span sequence generation using a sequence-to-sequence model with a pointer mechanism. It leverages the pre-trained BART model and three entity representation strategies (Word, BPE, Span), achieving state-of-the-art or near-state-of-the-art performance across eight English NER datasets without task-specific tagging schemas or span enumeration.

ABSTRACT

Named Entity Recognition (NER) is the task of identifying spans that represent entities in sentences. Whether the entity spans are nested or discontinuous, the NER task can be categorized into the flat NER, nested NER, and discontinuous NER subtasks. These subtasks have been mainly solved by the token-level sequence labelling or span-level classification. However, these solutions can hardly tackle the three kinds of NER subtasks concurrently. To that end, we propose to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework. Based on our unified framework, we can leverage the pre-trained Seq2Seq model to solve all three kinds of NER subtasks without the special design of the tagging schema or ways to enumerate spans. We exploit three types of entity representations to linearize entities into a sequence. Our proposed framework is easy-to-implement and achieves state-of-the-art (SoTA) or near SoTA performance on eight English NER datasets, including two flat NER datasets, three nested NER datasets, and three discontinuous NER datasets.

Motivation & Objective

• To address the lack of a unified approach for flat, nested, and discontinuous NER subtasks that can be solved with a single framework.
• To eliminate the need for task-specific tagging schemas or span enumeration in NER models.
• To explore the effectiveness of pre-trained sequence-to-sequence models like BART for diverse NER subtasks.
• To evaluate the impact of different entity representation strategies (Word, BPE, Span) on model performance.
• To demonstrate that a single generative model can handle overlapping, discontinuous, and crossing entity structures.

Proposed method

• Formulate NER as a sequence-to-sequence generation task, where the model generates a sequence of entity pointer indices.
• Use a pointer network within a pre-trained BART model to generate entity spans directly from input sentences.
• Propose three entity representation methods: Word (word-level), BPE (subword-level), and Span (start-end index pairs) to linearize entities into sequences.
• Train the model end-to-end using cross-entropy loss on the generated entity index sequences.
• Incorporate pre-trained BART to enhance contextual representation learning and improve generalization.
• Apply inference decoding with beam search to generate the final list of entity spans.

Experimental results

Research questions

• RQ1 Can a single generative sequence-to-sequence model effectively handle flat, nested, and discontinuous NER subtasks without architectural or tagging schema modifications?
• RQ2 How do different entity representation strategies (Word, BPE, Span) affect performance across diverse NER datasets?
• RQ3 To what extent can pre-trained Seq2Seq models like BART be leveraged to improve performance on multiple NER subtasks?
• RQ4 Does the model maintain strong recall for rare or complex structures like discontinuous and crossing entities?
• RQ5 How does the order of entity prediction in the output sequence affect recall performance?

Key findings

• The proposed unified generative framework achieves state-of-the-art or near-state-of-the-art performance on eight diverse English NER datasets, including two flat, three nested, and three discontinuous NER datasets.
• The 'Span' representation outperforms others on the Genia dataset, likely due to better alignment with BPE tokenization and reduced fragmentation of entity spans.
• The model achieves 55.0 F1 on discontinuous entities in ShARe13 and 52.7 F1 in ShARe14, outperforming prior methods on these rare but complex cases.
• Invalid predictions (e.g., non-contiguous or malformed indices) occur in less than 1% of cases, and excluding them during evaluation does not compromise overall performance.
• For flat and discontinuous NER, later-occurring entities in the output sequence are recalled with higher probability, suggesting lower inter-entity dependency.
• In nested NER, the recall curve is more complex, likely due to hierarchical dependencies where outer entities depend on correctly predicted inner entities.