[Paper Review] RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment

Kevin Yang, Dan Klein+3 more|arXiv (Cornell University)|Jul 24, 2023

Topic Modeling4 citations

TL;DR

RLCD proposes a novel method for aligning language models without human feedback by generating synthetic preference pairs through contrastive prompting: it uses positive and negative prompts to produce contrasting outputs, automatically labeling the positive output as preferred. This approach reduces label noise and improves alignment performance over RLAIF and context distillation baselines on harmlessness, helpfulness, and story generation tasks across 7B and 30B model scales.

ABSTRACT

We propose Reinforcement Learning from Contrastive Distillation (RLCD), a method for aligning language models to follow principles expressed in natural language (e.g., to be more harmless) without using human feedback. RLCD creates preference pairs from two contrasting model outputs, one using a positive prompt designed to encourage following the given principles, and one using a negative prompt designed to encourage violating them. Using two different prompts causes model outputs to be more differentiated on average, resulting in cleaner preference labels in the absence of human annotations. We then use the preference pairs to train a preference model, which is in turn used to improve a base unaligned language model via reinforcement learning. Empirically, RLCD outperforms RLAIF (Bai et al., 2022b) and context distillation (Huang et al., 2022) baselines across three diverse alignment tasks--harmlessness, helpfulness, and story outline generation--and when using both 7B and 30B model scales for simulating preference data.

Motivation & Objective

• To address the high cost and scalability limitations of collecting human-annotated preference data for language model alignment.
• To improve the quality of synthetic preference data used in reinforcement learning by reducing label noise.
• To combine the strengths of RLAIF (pairwise preferences) and context distillation (directional attribute control) in a single framework.
• To enable effective alignment of large language models using only LLM-generated preference signals.
• To demonstrate that contrastive prompt engineering can yield higher-quality preference data than standard RLAIF or context distillation

Proposed method

• Generate two contrasting prompts: a positive prompt $p_{+}$ encouraging a desired attribute (e.g., helpfulness), and a negative prompt $p_{-}$ discouraging it.
• Use the base language model to generate outputs $(o_{+}, o_{-})$ from $p_{+}$ and $p_{-}$, respectively.
• Automatically label $o_{+}$ as preferred and $o_{-}$ as non-preferred, forming synthetic pairwise preference data.
• Train a preference model on the generated synthetic preference pairs to predict relative quality.
• Fine-tune the target language model via PPO using the preference model’s reward signal.
• Use the same model for both generating preference pairs and for alignment, enabling scalable, feedback-free training.

Experimental results

Research questions

• RQ1 Can synthetic preference pairs generated via contrastive prompting outperform standard RLAIF in terms of alignment quality?
• RQ2 Does using both positive and negative prompts improve the signal-to-noise ratio in preference data compared to standard RLAIF?
• RQ3 Can contrastive distillation achieve better alignment than context distillation while preserving the benefits of pairwise RL?
• RQ4 How does RLCD perform across diverse alignment tasks such as harmlessness, helpfulness, and creative generation?
• RQ5 Does the method scale effectively to larger model sizes (e.g., 30B) without requiring human feedback?

Key findings

• RLCD outperforms RLAIF and context distillation baselines in human evaluations across all three alignment tasks: harmlessness, helpfulness, and story outline generation.
• On the 7B model scale, RLCD achieves superior performance in pairwise comparisons, indicating higher-quality preference signals.
• On the 30B model scale, RLCD maintains or exceeds the performance of baselines, demonstrating scalability.
• The method reduces label noise by ensuring greater quality contrast between $o_{+}$ and $o_{-}$, leading to more reliable preference modeling.
• The use of both positive and negative prompts provides richer supervision than context distillation, which only uses positive signals.
• The approach enables effective alignment without human feedback, making it suitable for large-scale, cost-effective LLM alignment.