[Paper Review] LLM Critics Help Catch LLM Bugs

Nat McAleese, Rai Michael Pokorny+4 more|arXiv (Cornell University)|Jun 28, 2024

Law, AI, and Intellectual Property8 citations

TL;DR

The paper trains RLHF-tuned LLM critics (CriticGPT) to critique model-written code, improving bug detection and often outperforming human reviewers; combining humans with critics reduces hallucinations and nitpicks.

ABSTRACT

Reinforcement learning from human feedback (RLHF) is fundamentally limited by the capacity of humans to correctly evaluate model output. To improve human evaluation ability and overcome that limitation this work trains "critic" models that help humans to more accurately evaluate model-written code. These critics are themselves LLMs trained with RLHF to write natural language feedback highlighting problems in code from real-world assistant tasks. On code containing naturally occurring LLM errors model-written critiques are preferred over human critiques in 63% of cases, and human evaluation finds that models catch more bugs than human contractors paid for code review. We further confirm that our fine-tuned LLM critics can successfully identify hundreds of errors in ChatGPT training data rated as "flawless", even though the majority of those tasks are non-code tasks and thus out-of-distribution for the critic model. Critics can have limitations of their own, including hallucinated bugs that could mislead humans into making mistakes they might have otherwise avoided, but human-machine teams of critics and contractors catch similar numbers of bugs to LLM critics while hallucinating less than LLMs alone.

Motivation & Objective

• Address the fundamental limitation of human evaluation in RLHF for large language models.
• Develop scalable oversight by training LLM critics to generate natural language critiques of code.
• Evaluate CriticGPT against human critiques and analyze human–machine collaboration effects.
• Introduce an inference-time sampling method (FSBS) to balance comprehensive critiques with hallucination risk.

Proposed method

• Train autoregressive critic policies that take (question, answer) pairs and output a plain text critique.
• Use RLHF with a reward model trained from contractor-rated critiques to optimize the critic policy (PPO).
• Incorporate an adversarial tampering step where contractors insert subtle bugs to create high-quality evaluation data.
• Apply Force Sampling Beam Search (FSBS) to constrain sampling and select critiques that balance length, highlights, and accuracy.
• Evaluate critiques via contractor ratings on comprehensiveness, critique-bug inclusion (CBI), nitpicks, and overall helpfulness.

Experimental results

Research questions

• RQ1 Can RLHF-trained LLM critics improve the accuracy and usefulness of human evaluations for model-written code?
• RQ2 How do CriticGPT critiques compare to human and ChatGPT critiques in detecting inserted bugs?
• RQ3 What are the tradeoffs between comprehensiveness and hallucinations in LLM critiques, and can FSBS navigate these tradeoffs?
• RQ4 Do human–machine teams (Human+CriticGPT) outperform either humans or critics alone in generating high-quality critiques?
• RQ5 Do critic-guided evaluations generalize to non-code tasks and real-world data distributions?

Key findings

• CriticGPT critiques are substantially preferred over ChatGPT and human critiques for code with inserted bugs.
• CriticGPT catches more inserted bugs than human contractors and outperforms representative humans paid for code review.
• Human+CriticGPT teams write more comprehensive critiques and reduce hallucinations compared to either alone.
• FSBS allows trading off comprehensiveness and hallucinations, enabling Pareto-front choices for critique quality.
• Training on adversarial tampering data yields higher-quality critiques than training without tampering.