[Paper Review] Training Language Models with Language Feedback

Jérémy Scheurer, Jon Ander Campos+4 more|arXiv (Cornell University)|Apr 29, 2022

Topic Modeling11 citations

TL;DR

This paper proposes a three-step method to fine-tune language models using natural language feedback instead of pairwise comparisons, enabling models to generate higher-quality outputs by refining initial generations based on human feedback. Using only 100 human-written feedback examples, the approach achieves human-level summarization performance with GPT-3 (175B parameters), demonstrating that large models can effectively incorporate detailed feedback to improve output quality.

ABSTRACT

Pretrained language models often do not perform tasks in ways that are in line with our preferences, e.g., generating offensive text or factually incorrect summaries. Recent work approaches the above issue by learning from a simple form of human evaluation: comparisons between pairs of model-generated task outputs. Comparison feedback conveys limited information about human preferences per human evaluation. Here, we propose to learn from natural language feedback, which conveys more information per human evaluation. We learn from language feedback on model outputs using a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input. In synthetic experiments, we first evaluate whether language models accurately incorporate feedback to produce refinements, finding that only large language models (175B parameters) do so. Using only 100 samples of human-written feedback, our learning algorithm finetunes a GPT-3 model to roughly human-level summarization ability.

Motivation & Objective

• To address the limitation of pairwise comparison feedback in preference learning, which conveys minimal information per human evaluation.
• To explore whether natural language feedback—richer in detail—can improve language model alignment with human preferences.
• To develop a scalable, non-reinforcement learning method for fine-tuning language models using human-written feedback.
• To evaluate whether large language models can accurately interpret and apply complex feedback to refine outputs.

Proposed method

• Condition a large language model on input, initial output, and human-written feedback to generate multiple refinements.
• Score each refinement using embedding-based similarity to the feedback and select the one with the highest score.
• Fine-tune a language model on the selected refinements to maximize likelihood of generating them given the input.
• Use a three-step pipeline: refinement generation, feedback-aware selection, and supervised fine-tuning.
• Leverage instruction-following capabilities of large models to interpret and act on natural language feedback.
• Avoid reinforcement learning or auxiliary losses, enabling direct application of natural language feedback.

Experimental results

Research questions

• RQ1 Can large language models accurately incorporate detailed natural language feedback to refine their outputs?
• RQ2 How does performance in feedback incorporation vary across model sizes, especially in controlled synthetic tasks?
• RQ3 Can a small number of human-written feedback examples (e.g., 100) lead to human-level performance in summarization tasks?
• RQ4 How does feedback-aware refinement selection improve output quality compared to baseline or no-feedback methods?
• RQ5 To what extent do LM-generated refinements reflect the intent and content of human feedback?

Key findings

• Only the largest GPT-3 model (175B parameters) accurately incorporates feedback in a synthetic offensive word-removal task, with a refinement accuracy of 38.5%.
• Smaller models (e.g., Ada, Babbage, Curie) showed significantly lower accuracy, with values below 2.5% on average.
• With only 100 human-written feedback samples, the method achieved summarization quality preferred by human evaluators in 51% of cases, approaching human reference performance.
• The refinement generation step successfully incorporated feedback, especially when selecting the highest-similarity output from multiple candidates.
• The method outperformed both baseline model outputs and feedback-agnostic refinements, as shown in qualitative examples where feedback-informed refinements included critical details omitted in initial summaries.
• Analysis confirmed that embedding-based similarity effectively identified refinements that aligned with feedback, validating the selection strategy.