[Paper Review] CHAI: A CHatbot AI for Task-Oriented Dialogue with Offline Reinforcement Learning

Siddharth Verma, Justin Fu+2 more|arXiv (Cornell University)|Apr 18, 2022

Speech and dialogue systems4 citations

TL;DR

CHAI proposes a task-oriented dialogue agent that combines fine-tuned language models with offline reinforcement learning to generate goal-directed, fluent, and effective dialogue without online interaction. By training on static human conversation datasets, CHAI achieves higher negotiation success rates and better pricing outcomes than prior methods, demonstrating that offline RL enables language models to pursue task goals while maintaining natural language quality.

ABSTRACT

Conventionally, generation of natural language for dialogue agents may be viewed as a statistical learning problem: determine the patterns in human-provided data and generate appropriate responses with similar statistical properties. However, dialogue can also be regarded as a goal directed process, where speakers attempt to accomplish a specific task. Reinforcement learning (RL) algorithms are designed specifically for solving such goal-directed problems, but the most direct way to apply RL -- through trial-and-error learning in human conversations, -- is costly. In this paper, we study how offline reinforcement learning can instead be used to train dialogue agents entirely using static datasets collected from human speakers. Our experiments show that recently developed offline RL methods can be combined with language models to yield realistic dialogue agents that better accomplish task goals.

Motivation & Objective

• To develop a dialogue agent that effectively pursues task goals while generating fluent, natural language responses.
• To address the limitations of supervised learning in dialogue systems, which often produce generic or goal-ignorant responses.
• To enable reinforcement learning in dialogue without costly online interaction by leveraging offline datasets.
• To mitigate issues like policy exploitation and distribution shift common in offline RL for dialogue.
• To combine the fluency of pre-trained language models with the goal-directed optimization of offline RL.

Proposed method

• CHAI uses a pre-trained language model (GPT-2) fine-tuned on a dialogue dataset to generate natural language responses.
• It applies offline reinforcement learning to optimize dialogue policies using static human-human conversation data, avoiding online interaction.
• The method employs a value-based RL algorithm to select responses that maximize task-specific rewards, such as successful negotiation or favorable pricing.
• A reward function is designed to reflect task outcomes, such as acceptance of offers and revenue, guiding the policy toward goal-oriented behavior.
• The system uses a price prediction head to ground responses in realistic negotiation dynamics, improving strategic coherence.
• Training is performed entirely from offline data, with no need for a simulated human model or online rollouts.

Experimental results

Research questions

• RQ1 Can offline reinforcement learning be effectively combined with pre-trained language models to train dialogue agents that are both fluent and goal-oriented?
• RQ2 Does training with offline RL improve task performance in negotiation tasks compared to supervised fine-tuning of language models?
• RQ3 Can CHAI generate more coherent and strategically sound responses than retrieval-based or language model-only baselines?
• RQ4 How does CHAI’s performance compare in terms of negotiation success rate and pricing accuracy?
• RQ5 To what extent does offline RL mitigate issues like policy exploitation and distribution shift in dialogue systems?

Key findings

• CHAI achieved a statistically significant improvement in negotiation success rate compared to both the retrieval-based baseline and the language modeling baseline (p < 0.01).
• CHAI outperformed baselines in all metrics except fluency, where it matched the language model baseline, confirming that fluency is preserved through the language model component.
• CHAI demonstrated superior strategic reasoning, such as offering reasonable counter-offers (e.g., 'I can’t go that low') and accepting offers only when beneficial.
• In human evaluation, CHAI was rated as more coherent, task-oriented, and human-like than the language modeling baseline, which frequently made illogical or off-topic responses.
• CHAI successfully negotiated better prices and achieved higher revenue than prior dialogue management systems, indicating effective optimization of task rewards.
• The system avoided non-sequitur responses common in retrieval-based models, such as irrelevant answers to questions about utilities or timing.