[Paper Review] Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Stephen Casper, Xander Davies+30 more|arXiv (Cornell University)|Jul 27, 2023

Software Reliability and Analysis ResearchComputer Science89 citations

TL;DR

The paper surveys open problems and fundamental limitations of RLHF across human feedback, reward modeling, and policy optimization, and discusses broader safety and governance implications.

ABSTRACT

Reinforcement learning from human feedback (RLHF) is a technique for training AI systems to align with human goals. RLHF has emerged as the central method used to finetune state-of-the-art large language models (LLMs). Despite this popularity, there has been relatively little public work systematizing its flaws. In this paper, we (1) survey open problems and fundamental limitations of RLHF and related methods; (2) overview techniques to understand, improve, and complement RLHF in practice; and (3) propose auditing and disclosure standards to improve societal oversight of RLHF systems. Our work emphasizes the limitations of RLHF and highlights the importance of a multi-faceted approach to the development of safer AI systems.

Motivation & Objective

• Taxonomize concrete challenges in RLHF across human feedback, reward modeling, and policy optimization.
• Differentiate tractable issues from fundamental limitations that require approaches beyond RLHF.
• Discuss integrating RLHF within a broader safety framework and governance considerations.

Proposed method

• Review and categorize existing challenges in RLHF into three main areas: feedback collection, reward modeling, and policy optimization.
• Summarize fundamental limitations and tractable improvements, with distinctions explained in Appendix B.
• Synthesize how RLHF-focused approaches fit within broader safety and oversight strategies.

Figure 1: (Top) Reinforcement Learning from Human Feedback. Gray, rounded boxes correspond to outputs (e.g., text), and colored diamonds correspond to evaluations. (Bottom) Our taxonomy for challenges with RLHF. We divide challenges with RLHF into three main types: challenges with obtaining quality

Experimental results

Research questions

• RQ1 What are the primary categories of challenges in RLHF (feedback, reward model, policy) and which are fundamental versus tractable within RLHF?
• RQ2 How can RLHF be complemented by additional safety measures and governance practices to improve societal oversight?
• RQ3 What are the implications of misaligned humans, data quality, and reward hacking for RLHF deployment?
• RQ4 What auditing and disclosure standards could improve transparency and accountability of RLHF systems?

Key findings

• RLHF faces challenges at the stages of feedback collection, reward modeling, and policy optimization.
• Many limitations are fundamental and would require approaches beyond RLHF to overcome.
• Oversight, data quality, and evaluation complexities pose persistent risks like misalignment, bias, and manipulation.
• RLHF should be embedded in multi-layered safety frameworks rather than relied on as a sole alignment method.
• There are governance and transparency considerations that can improve accountability for RLHF-based systems.

Figure 2: An example of RLHF for finetuning chatbots with binary preference feedback. Humans indicate which example between a pair they prefer. A reward model is trained using each example pair to provide rewards that reflect the human’s decisions. Finally, the LLM policy is finetuned using the rewa