[Paper Review] Information Extraction in Low-Resource Scenarios: Survey and Perspective

Shumin Deng, Ma, Yubo+3 more|arXiv (Cornell University)|Feb 16, 2022

Topic Modeling4 citations

TL;DR

This survey provides a comprehensive review of low-resource information extraction (IE) methods, categorizing traditional and LLM-based approaches into fine-grained paradigms. It empirically evaluates LLM-based techniques, finding that fine-tuned LMs and in-context learning with GPT-family models are most effective, with optimal solutions being task-dependent, offering practical insights for both research and industry applications.

ABSTRACT

Information Extraction (IE) seeks to derive structured information from unstructured texts, often facing challenges in low-resource scenarios due to data scarcity and unseen classes. This paper presents a review of neural approaches to low-resource IE from \emph{traditional} and \emph{LLM-based} perspectives, systematically categorizing them into a fine-grained taxonomy. Then we conduct empirical study on LLM-based methods compared with previous state-of-the-art models, and discover that (1) well-tuned LMs are still predominant; (2) tuning open-resource LLMs and ICL with GPT family is promising in general; (3) the optimal LLM-based technical solution for low-resource IE can be task-dependent. In addition, we discuss low-resource IE with LLMs, highlight promising applications, and outline potential research directions. This survey aims to foster understanding of this field, inspire new ideas, and encourage widespread applications in both academia and industry.

Motivation & Objective

• To systematically categorize low-resource IE methods into traditional and LLM-based paradigms for better understanding and comparison.
• To empirically evaluate the performance of LLM-based methods against state-of-the-art traditional models in low-resource settings.
• To identify the most effective technical solutions for low-resource IE, considering task-specific performance and tuning strategies.
• To highlight promising applications and outline future research directions in robust, faithful, and adaptable low-resource IE systems.
• To guide researchers and practitioners in selecting optimal methods for real-world low-resource IE applications.

Proposed method

• Categorizing traditional low-resource IE into three paradigms: exploiting higher-resource data, developing stronger data-efficient models, and optimizing data and models jointly.
• Classifying LLM-based approaches into direct inference without tuning (e.g., prompting with demonstrations) and model specialization with fine-tuning.
• Conducting an empirical study comparing LLM-based and traditional SOTA models on low-resource IE benchmarks.
• Using in-context learning (ICL) with GPT-family models and fine-tuning open-source LLMs to evaluate performance across tasks.
• Evaluating model performance using standard low-resource IE benchmarks and analyzing generalization under data scarcity and unseen classes.
• Proposing future directions such as active learning, synthetic data generation, and human-in-the-loop feedback for improved robustness and interpretability.

Experimental results

Research questions

• RQ1 Which LLM-based method—direct inference or fine-tuning—performs best in low-resource IE scenarios?
• RQ2 How do well-tuned large language models compare to previous state-of-the-art models in low-resource IE tasks?
• RQ3 To what extent can in-context learning with GPT-family models generalize across diverse low-resource IE tasks?
• RQ4 What are the key factors influencing the optimal selection of low-resource IE techniques, and how do they vary by task?
• RQ5 How can LLMs be effectively leveraged not only as extractors but also as auxiliary tools for data generation or sample reranking?

Key findings

• Well-tuned large language models remain the most effective approach for low-resource information extraction, outperforming many previous SOTA models.
• In-context learning with GPT-family models shows strong promise, particularly when combined with effective prompt design and few-shot demonstrations.
• Fine-tuning open-source LLMs yields better performance than zero-shot prompting in most low-resource IE tasks, indicating the value of task-specific adaptation.
• The optimal LLM-based solution for low-resource IE is highly task-dependent, with no single method universally superior across all scenarios.
• LLMs are less effective as primary extractors in low-resource settings due to schema complexity and limited few-shot generalization, but excel as rerankers for hard samples.
• LLMs can significantly enhance smaller models by assisting in correcting and re-ranking challenging predictions, suggesting collaborative modeling as a viable strategy.