[Paper Review] Learning to Extract Structured Entities Using Language Models

Haolun Wu, Ye Yuan+5 more|arXiv (Cornell University)|Feb 6, 2024

Data Quality and Management4 citations

TL;DR

This paper introduces Structured Entity Extraction (SEE), a novel task combining named-entity recognition, property extraction, and relation extraction under a predefined schema. The authors propose MuSEE, a multi-stage LLM-based model that improves both effectiveness and efficiency by decomposing extraction into parallel stages, achieving state-of-the-art performance on benchmark datasets with superior robustness and reduced hallucinations, as validated by quantitative and human evaluations.

ABSTRACT

Recent advances in machine learning have significantly impacted the field of information extraction, with Language Models (LMs) playing a pivotal role in extracting structured information from unstructured text. Prior works typically represent information extraction as triplet-centric and use classical metrics such as precision and recall for evaluation. We reformulate the task to be entity-centric, enabling the use of diverse metrics that can provide more insights from various perspectives. We contribute to the field by introducing Structured Entity Extraction and proposing the Approximate Entity Set OverlaP (AESOP) metric, designed to appropriately assess model performance. Later, we introduce a new Multistage Structured Entity Extraction (MuSEE) model that harnesses the power of LMs for enhanced effectiveness and efficiency by decomposing the extraction task into multiple stages. Quantitative and human side-by-side evaluations confirm that our model outperforms baselines, offering promising directions for future advancements in structured entity extraction. Our source code and datasets are available at https://github.com/microsoft/Structured-Entity-Extraction.

Motivation & Objective

• To formalize and define the task of Structured Entity Extraction (SEE) as a unified challenge combining NER, property extraction, relation extraction, and coreference resolution.
• To address the lack of appropriate evaluation metrics for SEE by proposing the Approximate Entity Set Overlap (AESOP) metric with multiple variants.
• To improve the effectiveness and efficiency of structured entity extraction using large language models through a novel multi-stage architecture.
• To evaluate model robustness to perturbations and reduce hallucinations by analyzing grounding to input text.
• To validate performance through quantitative benchmarks and human side-by-side evaluations.

Proposed method

• Formalize SEE as a closed information extraction task where all entity types and property keys are constrained by a predefined schema.
• Propose the AESOP metric, a family of set-based similarity measures tailored for comparing predicted and ground-truth structured entity sets, including variants for precision, recall, and max-f1.
• Design MuSEE, a multi-stage LLM-based model that decomposes entity extraction into sequential stages: entity recognition, property extraction, and relation prediction, enabling parallel processing and reduced token generation.
• Optimize inference efficiency by minimizing redundant token generation through stage-wise decomposition and focused prediction.
• Implement a grounding check by perturbing property values in test data using category-based replacement, measuring performance drop to assess reliance on input text vs. model knowledge.
• Use few-shot prompting with instruction tuning to guide the LLM in generating structured JSON outputs aligned with the schema.

Figure 1: Illustration of the structured entity extraction task. Given a text description as well as some predefined schema containing all the candidates of entity types and property keys, our task aims to output a structured json for all entities in the text with their information.

Experimental results

Research questions

• RQ1 How can structured entity extraction be formally defined as a unified NLP task combining NER, property extraction, and relation extraction under a schema?
• RQ2 What evaluation metric best captures the quality of structured entity extraction when comparing predicted and ground-truth sets of entities with properties?
• RQ3 Can a multi-stage LLM-based architecture improve both effectiveness and efficiency in structured entity extraction compared to end-to-end baselines?
• RQ4 To what extent do LLM-based models hallucinate or rely on pre-trained knowledge rather than grounding to input text?
• RQ5 How does model performance vary across different property types, especially rare or sparse ones?

Key findings

• MuSEE outperforms all baselines on the Wikidata-based dataset, achieving the highest AESOP-MultiProp-Max score, with a 61.75% preference rate in human completeness evaluation.
• MuSEE shows the smallest performance drop on perturbed data, indicating stronger grounding to input text and reduced hallucination compared to other models.
• On property-level analysis, MuSEE consistently outperforms baselines across all properties, with notable improvements on frequent properties like 'type' and 'entity name', and strong performance on rare ones like 'capital'.
• All models, including MuSEE, show lower performance on sparse properties such as 'place of death' and 'occupation', suggesting data sparsity as a key challenge.
• Human evaluation confirms MuSEE's superiority in correctness (59.32% preference) and hallucination reduction (57.13% preference), aligning with quantitative results.
• The multi-stage design of MuSEE improves inference efficiency by reducing token generation while maintaining high accuracy, demonstrating a favorable effectiveness-efficiency trade-off.

Figure 2: The pipeline of our proposed MuSEE model, which is built on an encoder-decoder architecture. The input text only needs to be encoded once. The decoder is shared for all the following three stages. All predictions within each stage can be processed in parallel, and teacher forcing enables p