[Paper Review] Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning

Saibo Geng, Martin Josifosky+2 more|arXiv (Cornell University)|May 23, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces Grammar-Constrained Decoding (GCD), a method that uses formal grammars to guide large language models (LLMs) in generating structured outputs without finetuning. By integrating an incremental parser with LLMs, GCD ensures syntactically valid and vocabulary-compliant outputs, achieving performance comparable to finetuned models on tasks like information extraction, entity disambiguation, and parsing—especially effective in low-data or zero-shot settings.

ABSTRACT

Despite their impressive performance, large language models (LMs) still struggle with reliably generating complex output structures when not finetuned to follow the required output format exactly. To address this issue, grammar-constrained decoding (GCD) can be used to control the generation of LMs, guaranteeing that the output follows a given structure. Most existing GCD methods are, however, limited to specific tasks, such as parsing or code generation. In this work, we demonstrate that formal grammars can describe the output space for a much wider range of tasks and argue that GCD can serve as a unified framework for structured NLP tasks in general. For increased flexibility, we introduce input-dependent grammars, which allow the grammar to depend on the input and thus enable the generation of different output structures for different inputs. We then empirically demonstrate the power and flexibility of GCD-enhanced LMs on (1) information extraction, (2) entity disambiguation, and (3) constituency parsing. Our results indicate that grammar-constrained LMs substantially outperform unconstrained LMs or even beat task-specific finetuned models. Grammar constraints thus hold great promise for harnessing off-the-shelf LMs for a wide range of structured NLP tasks, especially where training data is scarce or finetuning is expensive. Code and data: https://github.com/epfl-dlab/GCD.

Motivation & Objective

• Address the challenge of LLMs generating unreliable or invalid structured outputs due to vast output vocabularies and lack of structural constraints.
• Overcome the limitations of task-specific finetuning, which is costly and requires labeled data for each new task.
• Develop a unified, flexible framework for structured NLP tasks using formal grammars that can be dynamically adapted to input context.
• Enable few-shot and zero-shot performance on structured tasks by constraining decoding to only valid output sequences.
• Demonstrate that grammar constraints can outperform unconstrained LLMs and even rival finetuned models without retraining.

Proposed method

• Define formal grammars to describe the valid output space for structured NLP tasks, such as subject-relation-object triplets in closed information extraction.
• Introduce input-dependent grammars that dynamically adapt the grammar rules based on the input, enabling variable output structures per input.
• Integrate an incremental parser as a completion engine that, at each decoding step, determines only the valid next tokens according to the current grammar and prefix.
• Combine the LLM’s autoregressive generation with the grammar-aware completion engine to ensure all generated sequences are structurally valid.
• Use a push-down automaton or parser-based approach to efficiently validate and constrain the decoding process in real time.
• Deploy the framework with off-the-shelf LLMs, avoiding the need for task-specific finetuning or complex constraint engineering.

Figure 1: Grammar-constrained decoding (GCD), applied to the task of closed information extraction, where the goal is to extract a list $y$ of subject–relation–object triplets from the input text $x$ . Subjects and objects are constrained to be Wikidata entities, relations to be a Wikidata relation.

Experimental results

Research questions

• RQ1 Can formal grammars be used to define the valid output space across a wide range of structured NLP tasks without task-specific finetuning?
• RQ2 How does grammar-constrained decoding improve few-shot performance of LLMs on structured tasks compared to unconstrained generation?
• RQ3 To what extent can input-dependent grammars extend the applicability of GCD to tasks with variable or input-sensitive output structures?
• RQ4 Can GCD achieve performance on par with or better than finetuned models in low-data regimes?
• RQ5 What are the practical trade-offs in latency and scalability when applying complex grammars to large-scale LLM decoding?

Key findings

• Grammar-constrained decoding significantly improves the reliability and accuracy of LLMs in generating structured outputs across diverse tasks, including information extraction, entity disambiguation, and constituency parsing.
• GCD-enhanced LLMs outperform unconstrained LLMs on all evaluated tasks, even surpassing some finetuned models in few-shot settings.
• The use of input-dependent grammars enables the framework to handle tasks where valid outputs depend on input context, such as entity disambiguation and entity linking.
• Performance gains are most pronounced in low-data or zero-shot scenarios, where the vast output vocabulary otherwise overwhelms few-shot prompting.
• While GCD increases decoding latency due to the parsing overhead, the performance gains justify the cost, especially when finetuning is infeasible.
• The method is most effective with larger LLMs and when grammars are as restrictive as possible, highlighting the importance of grammar design in the framework’s success.

Figure 2: Formal grammars for 14 structured NLP tasks, highlighting the general applicability of grammar-constrained decoding. All 14 grammars are context-free (mostly regular). * marks input-dependent grammars. Inputs $x=\langle x_{0},\dots,x_{n-1}\rangle$ are sequences of lexical units (e.g., word