[Paper Review] Debating with More Persuasive LLMs Leads to More Truthful Answers

Akbir Khan, John Hughes+8 more|arXiv (Cornell University)|Feb 9, 2024

Artificial Intelligence in Law9 citations

TL;DR

The paper shows that debating stronger expert LLMs against each other, judged by weaker non-experts (including humans), yields more truthful answers; optimizing debaters for persuasiveness improves judge accuracy without ground-truth labels.

ABSTRACT

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

Motivation & Objective

• Motivate scalable oversight as models become more capable than humans.
• Evaluate information-asymmetric debate protocols where non-experts judge expert debaters.
• Develop and validate metrics for debater and judge performance without ground-truth labels.
• Assess whether persuasiveness optimization improves truthfulness in debates.
• Extend analyses to both language model and human judges across multiple protocols.

Proposed method

• Implement three debate protocols: Debate, Interactive Debate, and Consultancy as baselines.
• Use information-asymmetric setup with debaters having access to underlying text and judges (non-experts) without text.
• Operate on the QuALITY reading-com comprehension task with Project Gutenberg texts and a fixed three-round limit.
• Introduce quote verification to ensure evidence trustworthiness (verified/unverified quotes).
• Optimize debaters for persuasiveness via best-of-N sampling and critique-and-refinement at inference time.
• Evaluate with LLM judges (GPT-4-Turbo, GPT-3.5-Turbo, Claude variants) and human judges; report judge accuracy and calibration.
• Develop unsupervised metrics (win rate, Elo ratings) to assess debaters and judge performance without ground-truth labels.

Experimental results

Research questions

• RQ1 Can weaker models reliably supervise stronger models in a debate setting to identify truthful answers?
• RQ2 Does optimizing debaters for persuasiveness improve a non-expert judge’s ability to identify the truth?
• RQ3 How do language-model judges compare to human judges in evaluating debated arguments?
• RQ4 What are the effects of different debate protocols ( Debate vs Interactive Debate vs Consultancy) on accuracy and calibration?
• RQ5 Do results generalize across different base models and judge types?

Key findings

• Non-expert judges achieve 88% accuracy (humans) and 76% accuracy (LLMs) with debate; naive baselines are 60% and 48%, respectively.
• Optimizing debaters for persuasiveness improves judge ability to identify the truth in debates.
• Humans achieve higher accuracy and better calibration with debate than consultancy; confidence-based rejection maintains higher accuracy in debate.
• More capable judges yield higher accuracies and larger distinctions among debaters; strong debaters improve debate outcomes across judge types.
• Optimized consultants can harm judge accuracy, particularly when persuasiveness targets incorrect answers.
• Across protocols, debate outperforms consultancy in both human and LLM judge evaluations; non-expert interaction does not improve accuracy in static vs interactive setups.