[Paper Review] Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

Rajkumar Ramamurthy, Prithviraj Ammanabrolu+6 more|arXiv (Cornell University)|Oct 3, 2022

Topic ModelingComputer Science54 citations

TL;DR

The paper introduces RL4LMs, an open-source library for training language models with reinforcement learning, a GRUE benchmark to evaluate RL-based NLP alignment to human preferences, and NLPO, an on-policy RL algorithm with token-level masking to improve stability and performance over PPO in LM alignment tasks.

ABSTRACT

We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open-source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community: is RL a practical paradigm for NLP? To help answer this, we first introduce an open-source modular library, RL4LMs (Reinforcement Learning for Language Models), for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al. 2020) with an arbitrary reward function. Next, we present the GRUE (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference. GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language Policy Optimization) that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al. 2017)), based on both automatic and human evaluations.

Motivation & Objective

• Demonstrate that RL can be effective for aligning pre-trained LMs with human preferences.
• Provide an open-source modular toolkit for RL-based LM optimization.
• Introduce GRUE as a benchmark for RL-based NLP tasks driven by human-preference rewards.
• Propose NLPO to mitigate large action spaces in language generation and improve training stability.

Proposed method

• Develop RL4LMs, an on-policy RL toolkit compatible with HuggingFace models and stable-baselines-3.
• Model language generation as a token-level MDP with per-token or per-sequence rewards.
• Introduce NLPO, a masked PPO variant using top-p masking to reduce the action space during training.
• Define a KL-based regularized reward to balance task rewards with staying close to the base LM.
• Create GRUE, a multi-task benchmark with diverse reward-based evaluation and human studies.
• Provide extensive ablations and analyses comparing PPO, NLPO, and supervised+RL setups.

Experimental results

Research questions

• RQ1 Can RL techniques outperform supervised fine-tuning for aligning LMs to human preferences across diverse NLP tasks?
• RQ2 Does NLPO offer stability and performance benefits over PPO in large-action-space language generation?
• RQ3 To what extent do reward quality, baseline KL regularization, and masking influence RL stability and alignment quality?
• RQ4 Is data efficiency or parameter efficiency improved by RL approaches compared to purely supervised methods?
• RQ5 How well do automated metrics correlate with human judgments in RL-based language policy optimization?

Key findings

• RL methods generally outperform supervised approaches at aligning LMs to human preferences across evaluated tasks.
• NLPO shows greater stability and performance than PPO in both automatic and human evaluations.
• A KL penalty and task-specific masking (top-p) help mitigate reward hacking and improve alignment quality.
• Supervised warm-starts and data-efficient reward learning can yield strong performance with smaller models.
• RL can be more data-efficient than supervised learning when improving reward models, and NLPO with supervision can outperform large supervised models in some tasks.