[Paper Review] MetaIE: Distilling a Meta Model from LLM for All Kinds of Information Extraction Tasks

Letian Peng, Zilong Wang+3 more|arXiv (Cornell University)|Mar 30, 2024

Data Quality and ManagementDecision Sciences3 citations

TL;DR

MetaIE proposes a framework to distill a meta-model from large language models (LLMs) for zero-shot and few-shot information extraction (IE) across diverse tasks. By using a symbolic distillation process on a synthetic dataset of label-to-span instructions generated from LLMs, MetaIE enables small models to generalize effectively across 13 IE datasets, outperforming baselines from vanilla pre-training, multi-task fine-tuning, and single-task distillation.

ABSTRACT

Information extraction (IE) is a fundamental area in natural language processing where prompting large language models (LLMs), even with in-context examples, cannot defeat small LMs tuned on very small IE datasets. We observe that IE tasks, such as named entity recognition and relation extraction, all focus on extracting important information, which can be formalized as a label-to-span matching. In this paper, we propose a novel framework MetaIE to build a small LM as meta-model by learning to extract "important information", i.e., the meta-understanding of IE, so that this meta-model can be adapted to all kind of IE tasks effectively and efficiently. Specifically, MetaIE obtains the small LM via a symbolic distillation from an LLM following the label-to-span scheme. We construct the distillation dataset via sampling sentences from language model pre-training datasets (e.g., OpenWebText in our implementation) and prompting an LLM to identify the typed spans of "important information". We evaluate the meta-model under the few-shot adaptation setting. Extensive results on 13 datasets from 6 IE tasks confirm that MetaIE can offer a better starting point for few-shot tuning on IE datasets and outperform other meta-models from (1) vanilla language model pre-training, (2) multi-IE-task pre-training with human annotations, and (3) single-IE-task symbolic distillation from LLM. Moreover, we provide comprehensive analyses of MetaIE, such as the size of the distillation dataset, the meta-model architecture, and the size of the meta-model.

Motivation & Objective

• To address the inefficiency and poor few-shot performance of LLMs in information extraction tasks despite their broad capabilities.
• To develop a unified, small language model meta-model that generalizes across diverse IE tasks without task-specific fine-tuning.
• To explore whether LLMs can be distilled into a meta-understanding of 'important information' extraction via a label-to-span formalization.
• To evaluate the effectiveness of symbolic distillation from LLM-generated data compared to human-annotated or task-specific pre-training.
• To identify optimal model and data scale boundaries for distilling meta-understanding of IE.

Proposed method

• Construct a synthetic distillation dataset by sampling 100,000 sentences from OpenWebText and prompting GPT-3.5-turbo to identify typed spans of 'important information' for various IE labels.
• Formalize all IE tasks as a label-to-span instruction: given a label l, extract a span from the input text, enabling a unified distillation scheme.
• Perform symbolic distillation from the LLM-generated dataset to train a small student model using standard fine-tuning with cross-entropy loss.
• Support multiple model architectures: sequence labeling (e.g., RoBERTa), seq2seq (e.g., BART, T5), and causal LM (e.g., GPT2, OPT), with consistent label-span formatting.
• Use few-shot fine-tuning on downstream IE datasets to evaluate transferability and generalization of the meta-model.
• Conduct ablation studies on distillation data size, student model size, and architecture to analyze scaling behavior and performance trade-offs.

Experimental results

Research questions

• RQ1 Can a small language model be effectively distilled from an LLM to serve as a universal meta-model for all IE tasks?
• RQ2 Does symbolic distillation using LLM-generated label-to-span pairs outperform human-annotated multi-task pre-training or vanilla pre-training?
• RQ3 What is the optimal size of the distillation dataset and student model for achieving meta-understanding of IE?
• RQ4 How does the choice of model architecture (e.g., sequence labeling vs. seq2seq vs. causal LM) affect distillation effectiveness?
• RQ5 Does the meta-model generalize well to out-of-distribution IE datasets, such as SRL, ABSA, and ASTE?

Key findings

• MetaIE outperforms meta-models from vanilla pre-training, multi-IE-task human-annotated pre-training, and single-IE-task symbolic distillation across 13 datasets from 6 IE tasks in few-shot settings.
• RoBERTa-based sequence labeling models achieve the best performance among all architectures, outperforming BERT, BART, T5, GPT2, and OPT with similar parameter counts.
• A distillation dataset of 80K–160K sentences is sufficient to endow the student model with meta-understanding of IE, with even 10K examples providing significant improvement.
• The model’s performance scales with parameter count, with larger models generalizing better on complex or domain-specific tasks.
• MetaIE achieves strong few-shot transfer performance, often matching or exceeding task-specific distillation on in-distribution datasets, demonstrating its universality.
• The success of MetaIE is not due to LLMs alone but stems from the label-to-span formalization, as human-annotation-based distillation from LLM outputs performed poorly.