[Paper Review] GPT3-to-plan: Extracting plans from text using GPT-3

Alberto Olmo Hernandez, Sarath Sreedharan+1 more|arXiv (Cornell University)|Jun 14, 2021

Natural Language Processing TechniquesComputer Science9 references27 citations

TL;DR

The paper investigates using GPT-3 to extract structured action sequences from natural language texts and compares its performance to task-specific plan extraction methods across three datasets.

ABSTRACT

Operations in many essential industries including finance and banking are often characterized by the need to perform repetitive sequential tasks. Despite their criticality to the business, workflows are rarely fully automated or even formally specified, though there may exist a number of natural language documents describing these procedures for the employees of the company. Plan extraction methods provide us with the possibility of extracting structure plans from such natural language descriptions of the plans/workflows, which could then be leveraged by an automated system. In this paper, we investigate the utility of generalized language models in performing such extractions directly from such texts. Such models have already been shown to be quite effective in multiple translation tasks, and our initial results seem to point to their effectiveness also in the context of plan extractions. Particularly, we show that GPT-3 is able to generate plan extraction results that are comparable to many of the current state of the art plan extraction methods.

Motivation & Objective

• Assess whether a general-purpose language model (GPT-3) can extract plan-like action sequences from natural language texts.
• Compare GPT-3's performance to task-specific plan extraction methods on standard datasets.
• Evaluate how GPT-3's few-shot learning capabilities affect action-name and argument extraction.
• Examine GPT-3's ability to infer plan ordering from text and discuss practical implications for planning applications.

Proposed method

• Use GPT-3 (four engines: Davinci, Curie, Babbage, Ada) with few-shot exemplars to map natural language descriptions to a structured plan representation.
• Provide training examples as natural language text labeled with a functional plan notation; test text follows with an expected plan output after specific tags.
• Evaluate using precision, recall, and F1 against ground-truth annotations following essential/exclusive/optional action typing.
• Compare GPT-3 outputs to state-of-the-art task-specific models (EAD, BLCC, STFC, EASDRL, cEASDRL) on three datasets (WHS, WHG, CT).
• Experiment with 1–4-shot prompting to assess few-shot learning effects on F1 scores.

Experimental results

Research questions

• RQ1 Can GPT-3 extract structured action sequences from natural language descriptions of procedures?
• RQ2 How does GPT-3 compare to state-of-the-art task-specific plan extraction methods on standard datasets?
• RQ3 Do general language models like GPT-3 capture the ordering of actions in plans from text?
• RQ4 What is the impact of few-shot prompting on GPT-3's plan extraction performance?
• RQ5 Are there limitations of GPT-3 in argument extraction and more diverse plan descriptions?

Key findings

• GPT-3, especially the Davinci engine, achieves competitive F1 scores against state-of-the-art task-specific extractors on action names.
• Davinci outperforms several task-specific models in action-name extraction across WHS, WHG, and CT datasets.
• GPT-3 shows weaker performance on argument extraction compared to specialized methods, but still above random baseline.
• The model demonstrates some ability to infer plan ordering from text, an area where prior methods often assume fixed sentence order.
• Increasing few-shot examples generally improves performance for larger engines, with Davinci achieving over 80% F1 on WHS.
• For the best-known configuration, Davinci yields action-name F1 scores of 86.32 (WHS), 58.14 (CT), 43.36 (WHG) and action-argument F1 scores of 22.90 (WHS), 29.63 (CT), 22.25 (WHG).
• Curie and smaller engines show diminishing returns and require more examples to approach Davinci’s performance.