[Paper Review] Joint System-Wise Optimization for Pipeline Goal-Oriented Dialog System

Zichuan Lin, Jing Huang+3 more|arXiv (Cornell University)|Jun 9, 2021

Topic Modeling44 references4 citations

TL;DR

This paper proposes joint system-wise optimization for pipeline goal-oriented dialog systems, introducing automatic data augmentation for NLU training, a Poisson-distributed stochastic policy for improved exploration, and a reward bonus to guide policy toward successful dialog turns. The method achieves 93.8% success rate in system-wise evaluation and 16% higher human evaluation success rate than prior work on MultiWOZ 2.1.

ABSTRACT

Recent work (Takanobu et al., 2020) proposed the system-wise evaluation on dialog systems and found that improvement on individual components (e.g., NLU, policy) in prior work may not necessarily bring benefit to pipeline systems in system-wise evaluation. To improve the system-wise performance, in this paper, we propose new joint system-wise optimization techniques for the pipeline dialog system. First, we propose a new data augmentation approach which automates the labeling process for NLU training. Second, we propose a novel stochastic policy parameterization with Poisson distribution that enables better exploration and offers a principled way to compute policy gradient. Third, we propose a reward bonus to help policy explore successful dialogs. Our approaches outperform the competitive pipeline systems from Takanobu et al. (2020) by big margins of 12% success rate in automatic system-wise evaluation and of 16% success rate in human evaluation on the standard multi-domain benchmark dataset MultiWOZ 2.1, and also outperform the recent state-of-the-art end-to-end trained model from DSTC9.

Motivation & Objective

• Address the bottleneck in pipeline dialog systems where improvements to individual components do not translate to better system-wide performance.
• Overcome data scarcity in NLU training by automating labeling through inverse generation from dialog acts.
• Enhance policy exploration and success rate through a novel stochastic policy parameterization using Poisson distribution.
• Improve system-wide success by introducing a reward bonus that encourages exploration of successful dialog trajectories.
• Outperform competitive pipeline systems and state-of-the-art end-to-end models in both automatic and human evaluations on MultiWOZ 2.1.

Proposed method

• Propose a data augmentation technique that uses a pre-trained user NLG model to generate diverse utterances from dialog acts, then trains the system NLU on these synthetic utterance-act pairs.
• Introduce a stochastic policy network parameterized with a Poisson distribution to control the number of dialog acts and enable more effective exploration during training.
• Design a reward bonus mechanism that encourages the policy to explore successful dialog turns, reducing errors in downstream NLU and improving policy generalization.
• Train the NLU and policy components jointly in a system-wise optimization framework, while keeping the DST and NLG modules fixed as in the rule-based baseline.
• Use a system-wise evaluation protocol with an agenda-based user simulator to measure success rate based on database slot matching.
• Apply policy gradient methods with the proposed stochastic policy and reward bonus to optimize the policy end-to-end for system-level success.

Experimental results

Research questions

• RQ1 What is the primary bottleneck in pipeline goal-oriented dialog systems that prevents component-level improvements from translating to system-wide performance gains?
• RQ2 Can automatic data augmentation based on inverse NLG generation significantly improve NLU performance in low-resource settings?
• RQ3 Does a Poisson-distributed stochastic policy parameterization lead to better exploration and higher success rates compared to deterministic or standard stochastic policies?
• RQ4 Can a reward bonus that encourages exploration of successful dialog turns improve policy generalization and system-wide success?
• RQ5 How does the joint system-wise optimization of NLU and policy compare to state-of-the-art end-to-end models in multi-domain, real-world evaluation settings?

Key findings

• The proposed method achieves a 93.8% success rate in automatic system-wise evaluation on MultiWOZ 2.1, outperforming the rule-based baseline by 12 percentage points.
• In human evaluation, the method achieves a 16% higher success rate than the rule-based system, demonstrating superior real-world performance.
• The ablation study shows that data augmentation alone contributes a 10 percentage point gain in success rate, indicating that NLU is the primary bottleneck in pipeline systems.
• The Poisson-distributed policy improves success rate by 3% over the vanilla policy, demonstrating the benefit of controlled stochasticity in action selection.
• The reward bonus contributes an additional 3% improvement, showing that guiding exploration toward successful trajectories enhances learning efficiency.
• The S-PPO model scales well to multi-domain tasks, maintaining high performance even as the number of domains increases from 1 to 3, unlike baseline methods that degrade significantly.