[Paper Review] Generating Sequences by Learning to Self-Correct

Sean Welleck, Ximing Lu+5 more|arXiv (Cornell University)|Oct 31, 2022

Software Engineering Research28 citations

TL;DR

Self-Correction decomposes sequence generation into a base generator and a separate corrector that iteratively improves outputs, achieving improvements across multiple tasks without updating the base model. The corrector can be smaller than the base generator and can leverage explicit feedback.

ABSTRACT

Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content. Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs. Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation. We present Self-Correction, an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations. To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations. We show that Self-Correction improves upon the base generator in three diverse generation tasks - mathematical program synthesis, lexically-constrained generation, and toxicity control - even when the corrector is much smaller than the base generator.

Motivation & Objective

• Motivate the need for iterative refinement in sequence generation to satisfy semantic constraints.
• Propose a two-module framework: a base generator and a separate corrector that can be applied iteratively.
• Develop an online training procedure for the corrector that uses scalar or natural language feedback.
• Demonstrate cross-task effectiveness and transferability of the corrector to other generators and settings.

Proposed method

• Decompose p(y|x) into p0(y0|x) times pθ(y|y0,x) to form a self-corrector with multiple correction steps.
• Train the corrector via self-corrective learning that builds value-improving pairs from a pool of generator outputs and corrections.
• Use a scalar value function v(·) to guide improvements, with optional feedback f(·) guiding corrections.
• Form and sample value-improving pairs with a pairing and sampling strategy that balances improvement and similarity.
• Infer final outputs by decoding y0 from the generator, then iteratively decoding y(t+1) from the corrector.

Experimental results

Research questions

• RQ1 Can a separate corrector trained online improve the quality of a base generator across diverse tasks?
• RQ2 Is the correction mechanism transferable when swapping in a larger or different generator?
• RQ3 Does explicit natural language feedback further enhance correction quality?
• RQ4 What is the impact of training dynamics (pairing, proximity, exploration) on corrector effectiveness?

Key findings

• Self-Correction improves the base generator on mathematical program synthesis, lexically-constrained generation, and toxicity control.
• A corrector that is smaller than the base generator can still achieve substantial gains.
• The corrector can transfer improvements to a larger generator and even when the generator is swapped at test time.
• Explicit natural language feedback further guides corrections and yields performance gains across tasks.
• Multiple corrections and the proposed training dynamics contribute to stronger improvements, with exploration aiding performance.