[Paper Review] PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models

Torsten Scholak, Nathan Schucher+1 more|arXiv (Cornell University)|Sep 10, 2021

Natural Language Processing Techniques4 citations

TL;DR

PICARD is a novel, lightweight method for constrained auto-regressive decoding in large language models that uses incremental parsing to reject invalid tokens during generation, significantly improving output correctness. It enables fine-tuned T5 models to achieve state-of-the-art performance on text-to-SQL benchmarks like Spider and CoSQL without architectural changes or large beam sizes.

ABSTRACT

Large pre-trained language models for textual data have an unconstrained output space; at each decoding step, they can produce any of 10,000s of sub-word tokens. When fine-tuned to target constrained formal languages like SQL, these models often generate invalid code, rendering it unusable. We propose PICARD (code and trained models available at https://github.com/ElementAI/picard), a method for constraining auto-regressive decoders of language models through incremental parsing. PICARD helps to find valid output sequences by rejecting inadmissible tokens at each decoding step. On the challenging Spider and CoSQL text-to-SQL translation tasks, we show that PICARD transforms fine-tuned T5 models with passable performance into state-of-the-art solutions.

Motivation & Objective

• Address the challenge of generating syntactically and semantically valid formal language outputs (e.g., SQL) from pre-trained language models.
• Overcome the limitations of existing constrained decoding methods that require custom vocabularies, model architectures, or prohibitively large beam sizes.
• Enable high-accuracy, correct SQL generation from standard pre-trained language model decoders without retraining or fine-tuning.
• Provide a plug-in solution compatible with existing beam search and greedy decoding algorithms for auto-regressive models.

Proposed method

• Introduces incremental parsing using monadic combinators to validate partial token sequences at each decoding step.
• Applies a series of validation checks—lexing, parsing without guards, and parsing with guards—based on SQL schema and grammar rules.
• Restricts decoding to the top-k highest-probability tokens per step and assigns a score of -∞ to invalid tokens, effectively filtering them out.
• Supports four operation modes: off, lexing, parsing without guards, and parsing with guards, with increasing validation strength.
• Integrates seamlessly with standard auto-regressive decoding by modifying token scores without altering model architecture or vocabulary.
• Uses schema information (table and column names) to validate identifiers and ensure referential correctness during parsing.

Experimental results

Research questions

• RQ1 Can incremental parsing be used to improve the correctness of auto-regressive decoding in pre-trained language models without architectural modifications?
• RQ2 How does the integration of parsing constraints at each decoding step compare to post-hoc filtering of beam hypotheses in terms of performance and efficiency?
• RQ3 To what extent does Picard improve the performance of standard T5 models on text-to-SQL tasks like Spider and CoSQL?
• RQ4 What is the impact of different validation modes (lexing, parsing with/without guards) and beam size on final output quality?
• RQ5 Can Picard enable smaller T5 models to achieve state-of-the-art results comparable to larger models without Picard?

Key findings

• With Picard enabled, a T5-Base model outperforms a T5-Large model without Picard on the Spider dataset, demonstrating significant efficiency gains.
• A T5-3B model with Picard achieves state-of-the-art performance on Spider, reaching 56.9% question match accuracy and 24.2% interaction match accuracy on the development set.
• On CoSQL, Picard boosts the T5-3B model to 54.6% question match accuracy and 23.7% interaction match accuracy on the test set, matching or exceeding prior SOTA systems.
• Performance gains from Picard are most pronounced when moving from beam size 1 to 2, with diminishing returns beyond beam size 4.
• Incremental validation (at each step) is significantly more effective than finalizing-only validation, especially at smaller beam sizes.
• The number of top-k tokens processed by Picard has a modest impact on performance, with the effect being smallest for larger models like T5-3B.