[Paper Review] Unified Text Structuralization with Instruction-tuned Language Models

Xuanfan Ni, Piji Li+1 more|arXiv (Cornell University)|Mar 27, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a unified, instruction-tuned approach to text structuralization using large language models (LLMs), where prefix and suffix instructions guide the model to extract diverse information structures (e.g., entities, relations, events) without task-specific fine-tuning. Experiments show state-of-the-art performance across multilingual and cross-domain datasets, with strong zero-shot generalization and low-cost dataset creation potential in low-resource domains like law and finance.

ABSTRACT

Text structuralization is one of the important fields of natural language processing (NLP) consists of information extraction (IE) and structure formalization. However, current studies of text structuralization suffer from a shortage of manually annotated high-quality datasets from different domains and languages, which require specialized professional knowledge. In addition, most IE methods are designed for a specific type of structured data, e.g., entities, relations, and events, making them hard to generalize to others. In this work, we propose a simple and efficient approach to instruct large language model (LLM) to extract a variety of structures from texts. More concretely, we add a prefix and a suffix instruction to indicate the desired IE task and structure type, respectively, before feeding the text into a LLM. Experiments on two LLMs show that this approach can enable language models to perform comparable with other state-of-the-art methods on datasets of a variety of languages and knowledge, and can generalize to other IE sub-tasks via changing the content of instruction. Another benefit of our approach is that it can help researchers to build datasets in low-source and domain-specific scenarios, e.g., fields in finance and law, with low cost.

Motivation & Objective

• To address the scarcity of high-quality, multilingual, and domain-specific annotated datasets for text structuralization.
• To overcome the limitations of task-specific models that lack generalization across IE sub-tasks like NER, RE, and EE.
• To enable zero-shot and few-shot information extraction using instruction-tuned LLMs without fine-tuning.
• To reduce the cost and effort of building domain-specific datasets in low-resource fields such as finance and law.
• To evaluate the robustness and generalization of instruction-based prompting across different LLMs and instruction variations.

Proposed method

• Inject a prefix instruction specifying the desired IE task (e.g., 'Extract named entities') before the input text.
• Attach a suffix instruction specifying the target structure type (e.g., 'Output in key-value pairs') after the input text.
• Feed the combined input (prefix + text + suffix) into an instruction-tuned LLM for end-to-end text structuralization.
• Use zero-shot or few-shot prompting without task-specific fine-tuning to enable generalization across diverse IE sub-tasks.
• Evaluate model performance using F1-score on key-value pairs and structure degree metrics across multiple datasets and languages.
• Conduct ablation studies on instruction position and wording to assess model sensitivity and understanding.

Experimental results

Research questions

• RQ1 Can instruction-tuned LLMs perform zero-shot text structuralization across diverse IE tasks without task-specific fine-tuning?
• RQ2 How does the placement and wording of prefix and suffix instructions affect model performance and generalization?
• RQ3 To what extent can this unified prompting approach generalize to low-resource and domain-specific scenarios such as finance and law?
• RQ4 How do different decoding strategies (e.g., greedy vs. top-p sampling) impact output quality across varying model scales?
• RQ5 How does model scale influence error rates and instruction comprehension in text structuralization tasks?

Key findings

• Instruction-tuned LLMs, particularly GPT-3 175B and FLAN-T5-XXL, achieve competitive F1-scores on NER, RE, and EE tasks without any fine-tuning, matching or approaching SOTA performance.
• The method generalizes effectively across languages and domains, including low-resource settings such as medical and financial texts, enabling low-cost dataset creation.
• Ablation studies show that both prefix and suffix instructions are essential—removing either causes a significant drop in performance, indicating their critical role in guiding the model.
• The model is robust to instruction wording variations (e.g., 'extract' vs. 'get', 'information' vs. 'data'), suggesting it understands intent rather than relying on rigid keywords.
• Larger models like GPT-3 175B show fewer errors and better instruction comprehension, with no duplicate or hallucinated outputs observed, unlike smaller models.
• Decoding strategy has minimal impact on large models (e.g., FLAN-T5-XXL), indicating high stability and reliability across inference settings.