[Paper Review] CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers

Shiyang Li, Semih Yavuz+7 more|arXiv (Cornell University)|Oct 24, 2020

Topic Modeling43 references41 citations

TL;DR

CoCo generates controllable counterfactual conversations to stress-test DST models and reveals substantial generalization gaps beyond held-out accuracy, while enabling robust data augmentation.

ABSTRACT

Dialogue state trackers have made significant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the held-out conversations is less understood. We propose controllable counterfactuals (CoCo) to bridge this gap and evaluate dialogue state tracking (DST) models on novel scenarios, i.e., would the system successfully tackle the request if the user responded differently but still consistently with the dialogue flow? CoCo leverages turn-level belief states as counterfactual conditionals to produce novel conversation scenarios in two steps: (i) counterfactual goal generation at turn-level by dropping and adding slots followed by replacing slot values, (ii) counterfactual conversation generation that is conditioned on (i) and consistent with the dialogue flow. Evaluating state-of-the-art DST models on MultiWOZ dataset with CoCo-generated counterfactuals results in a significant performance drop of up to 30.8% (from 49.4% to 18.6%) in absolute joint goal accuracy. In comparison, widely used techniques like paraphrasing only affect the accuracy by at most 2%. Human evaluations show that COCO-generated conversations perfectly reflect the underlying user goal with more than 95% accuracy and are as human-like as the original conversations, further strengthening its reliability and promise to be adopted as part of the robustness evaluation of DST models.

Motivation & Objective

• Assess how well state-of-the-art DST models generalize to novel, realistic dialogue scenarios beyond held-out MultiWOZ test sets.
• Introduce a model-agnostic framework to generate counterfactual turn-level belief states and corresponding conversations.
• Evaluate the impact of counterfactuals on DST performance across multiple models.
• Explore whether counterfactual data augmentation improves robustness and original-task performance.

Proposed method

• Represent DST as a turn-level and dialog-level belief state tracking problem.
• Generate counterfactual goals by dropping, changing, or adding slot-values to the turn-level belief state.
• Use a pretrained encoder-decoder model to generate counterfactual user utterances conditioned on the modified belief states.
• Apply two filtering mechanisms to ensure generated utterances reflect the counterfactual goals: a slot-value match filter and a classifier-based filter.
• Introduce Value Substitution (VS) as a special CoCo case and a CoCo+ variant that uses VS as fallback when needed.
• Train a goal-conditioned utterance generator p_theta(U_usr_t | U_sys_t, L_t) with maximum likelihood and beam search for inference.

Experimental results

Research questions

• RQ1 How do state-of-the-art DST models perform when evaluated on CoCo-generated counterfactual conversations?
• RQ2 To what extent do unseen slot values and rare slot combinations affect generalization of DST models?
• RQ3 Can CoCo-based data augmentation improve robustness and/or original-task performance of DST models?

Key findings

• DST models experience a significant drop in joint goal accuracy (up to 30.8 percentage points) on CoCo- and CoCo+-generated counterfactual conversations compared to the original MultiWOZ test set.
• Paraphrasing via back-translation only modestly affects performance, with drops up to 2%.
• Unseen in-domain and out-of-domain slot values cause substantial accuracy degradation (up to 11.8% and ~10% additional drops respectively).
• CoCo+ rare-and-rare slot combinations yield the largest generalization gaps, with accuracy for a model like Trade dropping from 49.4% to 18.6%.
• Human evaluation shows CoCo-generated conversations are highly faithful to user goals (>95% accuracy) and human-like relative to originals; CoCo-ori often matches or slightly exceeds original turn-level belief-state correctness.
• Data augmentation with CoCo+ rare configurations improves robustness across models and can boost original test-set performance for some models (e.g., TripPy).