[Paper Review] OpenAssistant Conversations -- Democratizing Large Language Model Alignment

Andreas Köpf, Yannic Kilcher+16 more|arXiv (Cornell University)|Apr 14, 2023

Topic Modeling134 citations

TL;DR

The paper releases OpenAssistant Conversations, a large, multilingual, human-generated dataset for aligning LLMs, and demonstrates its usefulness by training and evaluating SFT, RM, and RLHF models on open architectures.

ABSTRACT

Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers. Models trained on OpenAssistant Conversations show consistent improvements on standard benchmarks over respective base models. We release our code and data under a fully permissive licence.

Motivation & Objective

• Democratize access to high-quality human feedback data for LLM alignment by releasing a large, open dataset.
• Assess the safety, ethics, and moderation implications of the dataset.
• Demonstrate the utility of the dataset by training and evaluating SFT, reward models, and RLHF-based models on open architectures.

Proposed method

• Release of OpenAssistant Conversations dataset containing 161,443 messages, 66,497 trees, 35 languages, and 461,292 quality ratings.
• Data collection via crowd-sourcing with five task types: prompting, replying as assistant, replying as prompter, labeling, and ranking.
• Tree state machine governs creation, growth, and completion of conversation trees; moderation and quality control via rewards, leaderboards, and human review.
• Training of SFT models, reward models (RM), and RLHF models on open models such as Pythia, Falcon, and LLaMA; evaluation on standard benchmarks.
• Comparison against baselines shows OpenAssistant Conversations-enabled models outperform some baseline counterparts on several benchmarks.

Figure 1: An example CT of depth 4 containing 12 messages. Any path from the root prompt to a node is a valid thread.

Experimental results

Research questions

• RQ1 Can a large, open, human-annotated conversation dataset drive improvements in alignment-based fine-tuning of LLMs?
• RQ2 How do SFT, RM, and RLHF models trained with OpenAssistant Conversations perform on standard NLP benchmarks compared to baselines?
• RQ3 What are the safety, toxicity, and moderation characteristics of such a dataset, and how well do automated toxicity measures align with human judgments?
• RQ4 What biases or limitations arise from a volunteer-driven, open dataset collection process, and how do these affect model behavior?
• RQ5 How should the data and models be released to maximize reproducibility and responsible research?

Key findings

• OpenAssistant Conversations contains 161,443 messages (91,829 prompter; 69,614 assistant) across 66,497 trees in 35 languages, with 461,292 quality ratings.
• Models trained with this data show consistent improvements over respective base models on standard benchmarks.
• RLHF improves some benchmarks relative to SFT but not uniformly across all tasks, indicating dataset and evaluation nuances.
• Automated toxicity measures (Detoxify) correlate with human labels to a degree, and moderation significantly reduces toxic content, though non-toxic messages can still be removed for other reasons.
• The dataset and models are released under permissive licenses, enabling open research while recognizing biases and safety considerations.
• User studies and demographic analyses reveal contributor homogeneity in some dimensions and potential biases due to uneven participation.

Figure 2: Left: Relative share of the most frequent languages in the dataset. Right: Distribution of contributions among users.