[Paper Review] Subgoal Discovery for Hierarchical Dialogue Policy Learning

Da Tang, Xiujun Li+4 more|arXiv (Cornell University)|Apr 20, 2018

Speech and dialogue systems3 citations

TL;DR

This paper proposes the Subgoal Discovery Network (SDN), an unsupervised method that automatically identifies meaningful subgoals from successful dialogue trajectories to enable hierarchical reinforcement learning (HRL) in complex task-oriented dialogues. By detecting frequently visited 'hub' states in successful paths, SDN enables efficient policy learning without human-defined subgoals, achieving performance competitive with expert-curated subgoals in travel planning tasks with both simulated and real users.

ABSTRACT

Developing agents to engage in complex goal-oriented dialogues is challenging partly because the main learning signals are very sparse in long conversations. In this paper, we propose a divide-and-conquer approach that discovers and exploits the hidden structure of the task to enable efficient policy learning. First, given successful example dialogues, we propose the Subgoal Discovery Network (SDN) to divide a complex goal-oriented task into a set of simpler subgoals in an unsupervised fashion. We then use these subgoals to learn a multi-level policy by hierarchical reinforcement learning. We demonstrate our method by building a dialogue agent for the composite task of travel planning. Experiments with simulated and real users show that our approach performs competitively against a state-of-the-art method that requires human-defined subgoals. Moreover, we show that the learned subgoals are often human comprehensible.

Motivation & Objective

• To address the challenge of sparse rewards and inefficient exploration in long, complex goal-oriented dialogues.
• To eliminate the need for human-defined subgoals in hierarchical reinforcement learning (HRL) for dialogue policy learning.
• To automatically discover interpretable, structurally meaningful subgoals from successful dialogue trajectories.
• To demonstrate that unsupervised subgoal discovery can yield policy performance comparable to that of HRL with expert-provided subgoals.
• To enable scalable, data-driven subgoal discovery applicable to real-world dialogue systems.

Proposed method

• SDN uses a probabilistic sequence segmentation model to identify 'hub' states—frequently visited states in successful dialogue trajectories but rare in failed ones.
• Hub states are treated as subgoals, dividing the dialogue into segments where each segment corresponds to a distinct subtask.
• The method applies hierarchical reinforcement learning (HRL) using the discovered subgoals as options, with a top-level policy selecting subgoals and low-level policies executing primitive actions.
• Intrinsic rewards are assigned at subgoal boundaries to guide exploration and credit assignment in the HRL framework.
• The approach leverages only successful dialogue trajectories, requiring no human-annotated subgoal labels.
• Subgoal discovery is unsupervised and based on state visitation frequency and trajectory structure, not reward shaping or reward modeling.

Experimental results

Research questions

• RQ1 Can subgoals be automatically discovered from successful dialogue trajectories without human supervision?
• RQ2 Can automatically discovered subgoals lead to effective hierarchical policy learning in complex dialogue tasks?
• RQ3 How does performance with automatically discovered subgoals compare to that with human-defined subgoals in dialogue policy learning?
• RQ4 Are the discovered subgoals interpretable and meaningful to humans?
• RQ5 Can the method generalize to real-user interactions and complex composite tasks like travel planning?

Key findings

• The dialogue agent trained with subgoals discovered by SDN achieved performance competitive with a state-of-the-art agent that uses human-defined subgoals in both simulated and real-user evaluations.
• The agent using SDN-discovered subgoals significantly outperformed a non-hierarchical baseline agent without any subgoals.
• The subgoals discovered by SDN were found to be human-comprehensible, often corresponding to natural milestones in the travel planning process, such as selecting a destination or booking a flight.
• SDN successfully identified meaningful subgoals using only a small amount of suboptimal but successful dialogue trajectories, demonstrating robustness to noisy data.
• The method effectively mitigated reward sparsity and improved exploration efficiency in long-horizon dialogue tasks.
• The results suggest that unsupervised subgoal discovery can serve as a viable alternative to expert-defined subgoals in HRL-based dialogue systems.