[Paper Review] "Think Before You Speak": Improving Multi-Action Dialog Policy by Planning Single-Action Dialogs

Shuo Zhang, Junzhou Zhao+4 more|arXiv (Cornell University)|Apr 25, 2022

Speech and dialogue systems4 citations

TL;DR

This paper proposes Planning Enhanced Dialog Policy (PEDP), a supervised multi-task learning framework that improves multi-action dialog policy by simulating single-action dialog fragments through model-based planning before multi-action prediction. By learning single-action dynamics and aggregating predictions across multiple planned paths, PEDP achieves a 90.6% task success rate on MultiWOZ, outperforming state-of-the-art methods by 3% without relying on reinforcement learning or external data.

ABSTRACT

Multi-action dialog policy (MADP), which generates multiple atomic dialog actions per turn, has been widely applied in task-oriented dialog systems to provide expressive and efficient system responses. Existing MADP models usually imitate action combinations from the labeled multi-action dialog samples. Due to data limitations, they generalize poorly toward unseen dialog flows. While interactive learning and reinforcement learning algorithms can be applied to incorporate external data sources of real users and user simulators, they take significant manual effort to build and suffer from instability. To address these issues, we propose Planning Enhanced Dialog Policy (PEDP), a novel multi-task learning framework that learns single-action dialog dynamics to enhance multi-action prediction. Our PEDP method employs model-based planning for conceiving what to express before deciding the current response through simulating single-action dialogs. Experimental results on the MultiWOZ dataset demonstrate that our fully supervised learning-based method achieves a solid task success rate of 90.6%, improving 3% compared to the state-of-the-art methods.

Motivation & Objective

• To address the poor generalization of supervised multi-action dialog policy (MADP) models to unseen dialog flows due to limited labeled data.
• To overcome the limitations of imitation learning in capturing diverse, contextually relevant action combinations not present in training data.
• To improve dialog policy performance without relying on costly human annotation, user simulators, or unstable reinforcement learning.
• To enhance decision-making in MADP by simulating plausible single-action dialog fragments before generating multi-action responses.
• To achieve better performance and robustness through multi-task learning of single-action dynamics and multi-action prediction.

Proposed method

• Introduces a Single-action Dialog Planning module that uses model-based planning to simulate multiple single-action dialog fragments (paths) from the current dialog state.
• Employs a 'self-play' mechanism between a discrete dialog policy and a world model to simulate user interactions over several turns in each path.
• Uses an Ensemble Prediction module to aggregate multi-action prediction probabilities across multiple planned paths, improving robustness and reducing variance.
• Trains the model end-to-end in a supervised learning framework using only existing labeled dialog data, avoiding external data collection.
• Leverages contextually relevant single-action dynamics to guide and inform multi-action predictions, enhancing reasoning and explainability.
• Applies multi-task learning to jointly optimize for single-action dialog planning and final multi-action response generation.

Experimental results

Research questions

• RQ1 Can simulating single-action dialog fragments improve the generalization of multi-action dialog policies in unseen dialog scenarios?
• RQ2 How does model-based planning of single-action dialog paths enhance multi-action prediction performance compared to direct imitation of multi-action combinations?
• RQ3 To what extent does multi-task learning of single-action dynamics improve robustness and stability in multi-action dialog policy learning?
• RQ4 Does the proposed method achieve better performance than state-of-the-art methods without relying on reinforcement learning or human-in-the-loop data collection?
• RQ5 How do the components of the planning and ensemble modules individually contribute to the overall performance gain?

Key findings

• PEDP achieves a task success rate of 90.6% on the MultiWOZ dataset, representing a 3% improvement over state-of-the-art methods.
• The Single-action Dialog Planning module contributes more significantly to performance gains than the Ensemble Prediction module, which primarily stabilizes results and reduces standard deviation.
• Action sampling during inference improves match rate and inform recall but reduces inform precision, indicating potential redundancy in responses.
• Human evaluation shows PEDP outperforms baseline models (GDPL, DiaAdv, DiaSeq) in user experience, with a Krippendorff’s alpha of 0.820–0.856, indicating high inter-rater reliability.
• Despite slightly lower inform precision (0.79) than some baselines, human users prioritize goal completion over redundancy, suggesting that the performance gap does not severely impact user experience.
• Ablation studies confirm that the proposed method is robust and effective, with sensitivity analysis showing stable performance across different numbers of planned paths (K).