[Paper Review] Improving alignment of dialogue agents via targeted human judgements

Amelia Glaese, Nat McAleese+32 more|arXiv (Cornell University)|Sep 28, 2022

Topic Modeling131 citations

TL;DR

Sparrow is an information-seeking dialogue agent trained with targeted human judgments and inline evidence, achieving higher preference and lower rule-violation rates than baselines, and providing evidence for factual claims 78% of the time.

ABSTRACT

We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.

Motivation & Objective

• Break down dialogue quality into natural language rules to guide human judgments and improve targeted reward modeling.
• Use reinforcement learning from human feedback (RLHF) to optimize for both preference (helpfulness) and rule compliance (harmlessness).
• Incorporate inline web evidence to improve factual correctness and verifiability.
• Analyze distributional properties and biases arising from rule-based mitigations and RLHF in dialogue systems.

Proposed method

• Define a set of rule-based constraints for helpfulness and correctness and collect per-rule human judgments.
• Generate dialogue turns by prompting a Chinchilla-70B base model with a detailed dialogue prompt including a Search Query and Search Result for evidence-enabled responses.
• Collect two types of human data: per-turn response preferences and adversarial probing to test rule violations.
• Train two reward models: a Preference Reward Model (for overall response quality) and a Rule Violation Reward Model (for rule compliance).
• Use reranking at test time to select among multiple candidate responses based on reward scores; train with RL (A2C) to optimize the combined objective.
• Incorporate evidence by retrieving web fragments and displaying them alongside model claims, and train evidence-grounding signals to assess plausibility and support.
• Freeze most of the base model during fine-tuning to share representations across reward models and policies; apply instruction-tuning-style objectives for the Rule RM.

Experimental results

Research questions

• RQ1 Can breaking down dialogue safety into per-rule judgments improve the efficiency and effectiveness of human feedback for RLHF?
• RQ2 Does integrating inline evidence improve factual correctness and verifiability of responses?
• RQ3 What is the impact of RLHF with targeted rules and evidence on user-perceived helpfulness and model safety, including resilience to adversarial probing?
• RQ4 What distributional biases or harms arise when enforcing rule-based safety via RLHF and evidence grounding?

Key findings

• Sparrow is preferred to prompted baselines and demonstrates higher resilience to adversarial probing, with rule violations occurring in only 8% of targeted probes.
• Evidence-enabled responses are plausible and supported by the provided web fragments 78% of the time.
• Reranking and RLHF combining preference signals and rule compliance reduces rule violations while maintaining high preference, compared to baselines.
• Providing inline evidence improves verifiability and trust, though distributional biases and harms can be amplified by safety interventions.
• There is a trade-off between optimizing for user preference and reducing rule violations, consistent with prior findings on safety versus usefulness.