[Paper Review] Disentangling Language and Knowledge in Task-Oriented Dialogs

Dinesh Raghu, Nikhil Gupta+1 more|arXiv (Cornell University)|May 3, 2018

Topic Modeling21 references4 citations

TL;DR

This paper proposes BoSsNet, a novel encoder-decoder architecture with a Bag-of-Sequences (BoSs) memory that disentangles language modeling from knowledge grounding in task-oriented dialog systems. By encoding dialog history and KB tuples as sequences within a bagged memory structure and introducing a copy-augmented loss, BoSsNet achieves state-of-the-art performance, especially under out-of-vocabulary (OOV) conditions, with over 10% absolute improvement on bAbI OOV test sets and high robustness to KB modifications.

ABSTRACT

The Knowledge Base (KB) used for real-world applications, such as booking a movie or restaurant reservation, keeps changing over time. End-to-end neural networks trained for these task-oriented dialogs are expected to be immune to any changes in the KB. However, existing approaches breakdown when asked to handle such changes. We propose an encoder-decoder architecture (BoSsNet) with a novel Bag-of-Sequences (BoSs) memory, which facilitates the disentangled learning of the response's language model and its knowledge incorporation. Consequently, the KB can be modified with new knowledge without a drop in interpretability. We find that BoSsNet outperforms state-of-the-art models, with considerable improvements (> 10\%) on bAbI OOV test sets and other human-human datasets. We also systematically modify existing datasets to measure disentanglement and show BoSsNet to be robust to KB modifications.

Motivation & Objective

• To address the challenge of maintaining performance in task-oriented dialog systems when the knowledge base (KB) changes over time, such as with new entities or updated information.
• To disentangle the language model from the knowledge incorporation mechanism, enabling generalization to unseen KB entities without retraining.
• To improve robustness and performance on out-of-vocabulary (OOV) test sets and real-world datasets with dynamic KBs.
• To develop a systematic evaluation framework—knowledge adaptability (KA)—to measure model resilience to KB modifications.
• To demonstrate that disentangled learning leads to more interpretable, grammatically correct, and informative dialog responses.

Proposed method

• BoSsNet employs a dual-level memory representation: a flat bag of sequences (BoSs) that stores both dialog history and KB tuples, preserving sequential context within each utterance or tuple.
• Each utterance and KB tuple is encoded via a bidirectional GRU (Bi-GRU), enabling contextual understanding of individual tokens while maintaining sequence-level structure.
• The encoder uses only memory cell representations to attend to relevant parts of the dialog and KB, enabling effective inference over the knowledge base.
• The decoder generates responses word-by-word using a copy-augmented sequence-to-sequence architecture, with a novel loss function that encourages copying from KB tokens rather than generating them from the language model.
• An additional loss term is introduced to promote disentanglement by penalizing non-copy generation of KB tokens, reinforcing the separation between language and knowledge components.
• Label dropout is applied during training to further disentangle the language model from the knowledge interface, improving generalization to unseen entities.

Experimental results

Research questions

• RQ1 Can a task-oriented dialog system maintain high performance when the knowledge base contains entirely new entities not seen during training?
• RQ2 Does disentangling the language model from the knowledge grounding mechanism lead to improved robustness against KB modifications?
• RQ3 To what extent does the proposed BoSs memory structure preserve contextual information while enabling effective copying of unseen KB entities?
• RQ4 How does the copy-augmented loss with disentanglement regularization improve generalization on OOV benchmarks compared to standard sequence-to-sequence models?
• RQ5 Are the dialog responses generated by BoSsNet more grammatically correct and informative in human evaluations compared to existing baselines?

Key findings

• BoSsNet achieves a 10.2% absolute improvement on the bAbI OOV test set compared to state-of-the-art models, demonstrating strong generalization to unseen entities.
• On the CamRest dataset, BoSsNet maintains high performance even when 100% of KB entities are unseen at test time, outperforming baselines in knowledge adaptability (KA) evaluations.
• Human evaluations show that BoSsNet responses are rated as more informative and slightly more grammatically correct than those from Seq2Seq, Mem2Seq, and Seq2Seq+Copy.
• The model exhibits robust performance across all levels of KB modification, with minimal performance drop as the percentage of unseen entities increases from 0% to 100%.
• The introduction of the knowledge adaptability (KA) evaluation protocol reveals that BoSsNet is significantly more resilient to KB changes than existing models.
• The model's performance is stable and consistent across multiple real-world datasets, including CamRest and Stanford Multi-Domain, confirming its generalization capability.