[Paper Review] STAR: Boosting Low-Resource Information Extraction by Structure-to-Text Data Generation with Large Language Models

Mingyu Derek Ma, Xiaoxuan Wang+4 more|arXiv (Cornell University)|May 24, 2023

Topic Modeling4 citations

TL;DR

STAR proposes a structure-to-text data generation framework that first synthesizes target information extraction structures using few-shot demonstrations, then generates natural language passages via LLMs with instruction tuning and self-reflection for error correction. It significantly boosts low-resource event and relation extraction performance, outperforming human-curated data in quality and effectiveness.

ABSTRACT

Information extraction tasks such as event extraction require an in-depth understanding of the output structure and sub-task dependencies. They heavily rely on task-specific training data in the form of (passage, target structure) pairs to obtain reasonable performance. However, obtaining such data through human annotation is costly, leading to a pressing need for low-resource information extraction approaches that require minimal human labeling for real-world applications. Fine-tuning supervised models with synthesized training data would be a generalizable method, but the existing data generation methods either still rely on large-scale ground-truth data or cannot be applied to complicated IE tasks due to their poor performance. To address these challenges, we propose STAR, a data generation method that leverages Large Language Models (LLMs) to synthesize data instances given limited seed demonstrations, thereby boosting low-resource information extraction performance. Our approach involves generating target structures (Y) followed by generating passages (X), all accomplished with the aid of LLMs. We design fine-grained step-by-step instructions to obtain the initial data instances. We further reduce errors and improve data quality through self-reflection error identification and self-refinement with iterative revision. Our experiments show that the data generated by STAR significantly improve the performance of low-resource event extraction and relation extraction tasks, even surpassing the effectiveness of human-curated data. Human assessment of the data quality shows STAR-generated data exhibits higher passage quality and better align with the task definitions compared with the human-curated data.

Motivation & Objective

• To address the challenge of low-resource information extraction where annotated training data is scarce and costly to produce.
• To develop a scalable, generalizable data generation method that does not rely on existing gold-standard data for synthesis.
• To improve the performance of supervised IE models in low-resource settings through high-quality synthetic training data.
• To ensure generated data maintains structural correctness, linguistic quality, and alignment with task definitions.
• To outperform human-curated data in both data quality and downstream model performance.

Proposed method

• STAR generates target structures (Y) first from seed demonstrations using few-shot prompting, enabling control over diversity and distribution of event types, triggers, and argument roles.
• It then uses instruction-tuned LLMs to generate natural language passages (X) that contain the specified target structure, guided by fine-grained, task-specific instructions.
• Self-reflection is applied by prompting the LLM to identify quality issues in generated passages using automatically created reflection questions.
• Hindsight feedback via template-based revision is applied iteratively to correct errors and improve data quality.
• The pipeline is designed as an inverse generation process (Y → X), shifting from difficult structure induction to easier conditional text generation, where LLMs excel.
• The method supports both event extraction and relation extraction by adapting the structure and instruction templates accordingly.

Figure 1: The Star inverse data generation strategy using event extraction task as an example. We first generate target structures from valid trigger and argument candidates. Then we prompt the LLM with task instructions from different task granularities to generate the initial passage $X_{0}$ conta

Experimental results

Research questions

• RQ1 Can LLM-based data generation that starts from structure synthesis produce high-quality training data for low-resource IE tasks?
• RQ2 Does the inverse generation strategy (Y → X) outperform traditional methods that rely on existing gold data for data synthesis?
• RQ3 Can self-reflection and iterative refinement significantly reduce errors in synthetic IE data?
• RQ4 To what extent can STAR-generated data improve downstream IE model performance compared to human-curated data?
• RQ5 How does the quality of STAR-generated data compare to human-curated data in terms of linguistic correctness and task alignment?

Key findings

• STAR-generated data improved low-resource event extraction performance by 5.4 F1 points over zero-shot baseline when using SURE prompting.
• The method achieved a 3.6 F1 point improvement in relation extraction over the zero-shot baseline using GenPT prompting.
• Human evaluation showed STAR-generated data scored 96% on grammaticality, 79% on informativeness, and 95% on commonsense, outperforming human-curated data in passage quality.
• STAR-generated data achieved 100% accuracy in capturing argument spans and 98% in correctly assigning argument roles, matching or exceeding human performance.
• The generated data demonstrated superior alignment with task definitions, particularly in trigger and event type consistency (99% accuracy).
• In multiple evaluations, STAR-generated data surpassed the performance of human-curated data in downstream IE models, even when the latter was used as a benchmark.

Figure 2: Event extraction performance (F1, %) when the EE models are trained on $N$ augmented training data on top of 10 data points ( $k=10$ ) for each event type. We observe that performance gain brought by Star -generated data is magnified as the data augmentation scales up with a larger $N$ , a