[Paper Review] ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT

Wei Xiang, Xingyu Cui+10 more|arXiv (Cornell University)|Feb 20, 2023

Topic Modeling143 citations

TL;DR

ChatIE reframes zero-shot IE as a multi-turn QA task with a two-stage framework on ChatGPT, extracting entity relations, entities, and events without fine-tuning and achieving strong results across several datasets.

ABSTRACT

Zero-shot information extraction (IE) aims to build IE systems from the unannotated text. It is challenging due to involving little human intervention. Challenging but worthwhile, zero-shot IE reduces the time and effort that data labeling takes. Recent efforts on large language models (LLMs, e.g., GPT-3, ChatGPT) show promising performance on zero-shot settings, thus inspiring us to explore prompt-based methods. In this work, we ask whether strong IE models can be constructed by directly prompting LLMs. Specifically, we transform the zero-shot IE task into a multi-turn question-answering problem with a two-stage framework (ChatIE). With the power of ChatGPT, we extensively evaluate our framework on three IE tasks: entity-relation triple extract, named entity recognition, and event extraction. Empirical results on six datasets across two languages show that ChatIE achieves impressive performance and even surpasses some full-shot models on several datasets (e.g., NYT11-HRL). We believe that our work could shed light on building IE models with limited resources.

Motivation & Objective

• Motivate zero-shot information extraction to reduce labeling effort and time.
• Explore whether large language models can form a unified, prompting-based IE system without fine-tuning.
• Develop a two-stage, chat-based pipeline that decomposes complex IE tasks into simpler sub-tasks.

Proposed method

• Transform zero-shot IE into a multi-turn question-answering process using ChatGPT.
• Stage I identifies existing element types (e.g., entity types, relations, event types) in a sentence.
• Stage II performs chained extraction for the identified types, allowing dependencies between sub-tasks.
• Use task-specific templates to generate prompts and require ChatGPT responses in table or list form for structured output.
• Compose Turn-level outputs into final structured data (RE triples, NER entities, EE arguments).
• Evaluate on six datasets across English and Chinese for RE, NER, and EE.

Experimental results

Research questions

• RQ1 Can a two-stage, chat-based prompting framework enable zero-shot IE across multiple IE tasks?
• RQ2 How does ChatIE perform on RE, NER, and EE compared with full-shot and few-shot baselines across diverse datasets?
• RQ3 What is the impact of decomposing IE into multi-turn sub-tasks on extraction accuracy and robustness?

Key findings

• Vanilla ChatGPT with a single-turn prompt underperforms on IE tasks.
• ChatIE’s two-stage framework yields substantial improvements, averaging about 18.98 percentage points in F1 across six datasets.
• ChatIE matches or surpasses several full-shot models on multiple datasets (e.g., NYT11-HRL, DuIE2.0, DuEE1.0) without any fine-tuning.
• On some baselines, ChatIE outperforms supervised models like FCM and MultiR on NYT11-HRL, highlighting reduced data labeling needs.
• ChatIE provides competitive results with less computation since it avoids parameter updates or finetuning.