[Paper Review] Generalized Intent Discovery: Learning from Open World Dialogue System

Yutao Mou, Keqing He+7 more|arXiv (Cornell University)|Sep 13, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces Generalized Intent Discovery (GID), a novel task that enables an in-domain (IND) intent classifier to simultaneously classify labeled IND intents and incrementally discover unlabeled out-of-domain (OOD) intent types. The authors propose both pipeline-based and end-to-end frameworks, using self-supervised contrastive learning with pseudo-labeling via augmented views and the Sinkhorn-Knopp algorithm, achieving improved generalization and robustness on three public GID benchmarks with real-world dialogue data.

ABSTRACT

Traditional intent classification models are based on a pre-defined intent set and only recognize limited in-domain (IND) intent classes. But users may input out-of-domain (OOD) queries in a practical dialogue system. Such OOD queries can provide directions for future improvement. In this paper, we define a new task, Generalized Intent Discovery (GID), which aims to extend an IND intent classifier to an open-world intent set including IND and OOD intents. We hope to simultaneously classify a set of labeled IND intent classes while discovering and recognizing new unlabeled OOD types incrementally. We construct three public datasets for different application scenarios and propose two kinds of frameworks, pipeline-based and end-to-end for future work. Further, we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide new guidance for future GID research.

Motivation & Objective

• To address the limitation of traditional intent classification models that cannot recognize new, unseen out-of-domain (OOD) intent types in real-world deployment.
• To propose a new task, Generalized Intent Discovery (GID), that enables incremental discovery of unlabeled OOD intent types while maintaining classification performance on labeled in-domain (IND) intents.
• To construct three public benchmarks—single-domain, multi-domain, and cross-domain—for diverse application scenarios in open-world dialogue systems.
• To develop and evaluate two framework paradigms: pipeline-based and end-to-end, for joint IND and OOD intent learning.
• To identify and analyze key challenges in GID, such as fine-grained OOD types, domain gaps, data imbalance, and OOD noise, to guide future research.

Proposed method

• Proposes a pipeline-based framework that first performs clustering on unlabeled OOD data to generate pseudo-labels, then jointly trains on labeled IND data and pseudo-labeled OOD data.
• Introduces an end-to-end framework that jointly optimizes for IND classification and OOD pseudo-labeling using a unified objective function.
• Employs a contrastive learning strategy with view augmentation to generate consistent representations for the same OOD sample, enabling reliable pseudo-label prediction.
• Uses the Sinkhorn-Knopp (SK) algorithm to solve the differentiable optimal transport problem for assigning soft pseudo-labels to OOD samples.
• Leverages swapped prediction between augmented views to generate pseudo-labels without requiring ground-truth OOD labels.
• Trains the model end-to-end using a joint loss combining supervised cross-entropy for IND classes and contrastive loss for OOD clustering.

Experimental results

Research questions

• RQ1 How can an intent classifier be extended to recognize both labeled in-domain (IND) intents and unlabeled out-of-domain (OOD) intent types in a single unified framework?
• RQ2 What are the key challenges in Generalized Intent Discovery (GID), such as domain shift, data imbalance, and noisy OOD samples, and how do they affect model performance?
• RQ3 How do pipeline-based and end-to-end frameworks compare in terms of performance and robustness in discovering new OOD intent types?
• RQ4 Can self-supervised contrastive learning with view augmentation and Sinkhorn-Knopp optimization effectively generate reliable pseudo-labels for OOD intent discovery?
• RQ5 What are the practical implications of GID for continual system improvement in real-world dialogue systems with evolving user queries?

Key findings

• The end-to-end framework outperforms the pipeline-based approach by better integrating OOD discovery with IND classification, reducing error propagation from noisy pseudo-labels.
• Fine-grained OOD intent types pose a significant challenge, as they are easily misclassified or clustered incorrectly due to semantic similarity with existing IND intents.
• Domain gaps between training and inference data lead to performance degradation, especially when OOD data comes from a different domain than the IND training set.
• Data imbalance between IND and OOD classes negatively impacts model generalization, with underrepresented OOD types being harder to discover.
• Real-world OOD noise—such as ambiguous or misclassified queries—significantly affects clustering quality and model robustness.
• Estimating the correct number of OOD intent types remains a critical challenge, as over- or under-estimation leads to poor clustering and classification performance.