[Paper Review] Document-editing Assistants and Model-based Reinforcement Learning as a Path to Conversational AI

Katya Kudashkina, Patrick M. Pilarski+1 more|arXiv (Cornell University)|Aug 27, 2020

Topic Modeling199 references4 citations

TL;DR

This paper proposes voice document editing as a focused domain for developing conversational AI assistants, using model-based reinforcement learning (MBRL) to enable purposive, goal-oriented interaction. By learning from real user interactions and building internal world models, MBRL allows assistants to understand user intent, adapt dynamically, and improve collaboration through continuous online learning, advancing toward truly conversational and intelligent assistants.

ABSTRACT

Intelligent assistants that follow commands or answer simple questions, such as Siri and Google search, are among the most economically important applications of AI. Future conversational AI assistants promise even greater capabilities and a better user experience through a deeper understanding of the domain, the user, or the user's purposes. But what domain and what methods are best suited to researching and realizing this promise? In this article we argue for the domain of voice document editing and for the methods of model-based reinforcement learning. The primary advantages of voice document editing are that the domain is tightly scoped and that it provides something for the conversation to be about (the document) that is delimited and fully accessible to the intelligent assistant. The advantages of reinforcement learning in general are that its methods are designed to learn from interaction without explicit instruction and that it formalizes the purposes of the assistant. Model-based reinforcement learning is needed in order to genuinely understand the domain of discourse and thereby work efficiently with the user to achieve their goals. Together, voice document editing and model-based reinforcement learning comprise a promising research direction for achieving conversational AI.

Motivation & Objective

• To address the challenge of creating conversational AI assistants that understand user purposes and adapt over time.
• To identify a tightly scoped, accessible domain—voice document editing—that provides a concrete, delimited context for dialogue and goal-oriented behavior.
• To argue that model-based reinforcement learning (MBRL) is essential for enabling assistants to learn internal world models and reason about user goals efficiently.
• To promote online, interactive learning from real user interactions as a way to build communication resources and improve assistant performance over time.
• To position voice document-editing assistants as a testbed for developing broader conversational AI systems with genuine understanding and adaptability.

Proposed method

• Use voice document editing as a controlled, real-world domain where the assistant interacts with a user to modify text through natural language commands.
• Apply model-based reinforcement learning (MBRL) to enable the assistant to learn a predictive model of the environment (i.e., document state changes) from interaction.
• Train the assistant using online learning from real human-computer interactions, allowing dynamic adaptation and co-development of communication strategies.
• Pre-train the assistant with offline datasets—such as crowdsourced dictation and editing logs from Mechanical Turk or organizational workflows—to provide initial structural and linguistic knowledge.
• Use simulated users or synthetic data pipelines to generate training data when real interaction data is limited, especially in early development stages.
• Leverage temporally abstract behaviors (options) in MBRL to support higher-level planning and reasoning, enabling the assistant to manage complex editing tasks as sequences of subgoals.

Experimental results

Research questions

• RQ1 Can voice document editing serve as a viable, focused domain for developing conversational AI assistants with deeper understanding of user goals?
• RQ2 How can model-based reinforcement learning (MBRL) enable assistants to learn internal world models and reason about user intentions in a document-editing context?
• RQ3 What are the relative advantages of online versus offline training for voice editing assistants in terms of adaptability and initial performance?
• RQ4 How can communication resources—shared understanding and interaction patterns—be co-developed between users and assistants during real-time interaction?
• RQ5 To what extent can a voice editing assistant trained via MBRL achieve purposive behavior, i.e., understanding and acting on high-level user goals rather than just executing commands?

Key findings

• Voice document editing provides a well-defined, accessible domain where the assistant and user can co-construct a shared context, making it ideal for studying conversational AI.
• Model-based reinforcement learning (MBRL) enables assistants to build internal models of document state transitions, allowing for more efficient planning and goal-directed behavior.
• Online learning from real user interactions allows the assistant to adapt continuously and co-develop communication strategies with users, improving long-term collaboration.
• Pre-training with offline datasets—such as crowdsourced dictation and editing sequences—can provide essential linguistic and structural knowledge, improving initial performance before online adaptation.
• The use of temporally abstract behaviors (options) in MBRL supports higher-level reasoning, enabling the assistant to manage complex editing tasks as sequences of subgoals.
• The proposed approach not only advances voice editing assistants but also provides transferable insights for building other purposive, adaptive AI assistants in broader domains.