[Paper Review] An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction

Stefan Larson, Anish Mahendran+9 more|arXiv (Cornell University)|Sep 4, 2019

Topic Modeling4 citations

TL;DR

This paper introduces a new evaluation dataset with 23,700 queries across 150 in-scope intents and 1,200 out-of-scope queries, designed to assess intent classification and out-of-scope detection in task-oriented dialog systems. Despite strong in-scope performance (BERT achieves 96% accuracy), all models struggle with out-of-scope detection, with the best system reaching only 66% recall, highlighting a critical gap in current NLP benchmarks.

ABSTRACT

Task-oriented dialog systems need to know when a query falls outside their range of supported intents, but current text classification corpora only define label sets that cover every example. We introduce a new dataset that includes queries that are out-of-scope---i.e., queries that do not fall into any of the system's supported intents. This poses a new challenge because models cannot assume that every query at inference time belongs to a system-supported intent class. Our dataset also covers 150 intent classes over 10 domains, capturing the breadth that a production task-oriented agent must handle. We evaluate a range of benchmark classifiers on our dataset along with several different out-of-scope identification schemes. We find that while the classifiers perform well on in-scope intent classification, they struggle to identify out-of-scope queries. Our dataset and evaluation fill an important gap in the field, offering a way of more rigorously and realistically benchmarking text classification in task-driven dialog systems.

Motivation & Objective

• To address the lack of realistic evaluation for out-of-scope query detection in task-oriented dialog systems.
• To create a large-scale, diverse dataset with both in-scope and out-of-scope queries that mirror real user behavior.
• To evaluate existing classifiers and out-of-scope detection methods on a realistic benchmark that includes ambiguous, off-topic queries.
• To demonstrate that current models, even state-of-the-art ones like BERT, perform poorly on out-of-scope detection despite strong in-scope accuracy.
• To provide a standardized, publicly available dataset to enable more rigorous and realistic benchmarking of dialog system robustness.

Proposed method

• The dataset was collected via crowdsourcing using scoping, rephrasing, and scenario-based tasks to generate 22,500 in-scope queries across 150 intent classes in 10 domains.
• An additional 1,200 out-of-scope queries were collected to represent realistic user inputs that fall outside the system’s supported capabilities.
• The dataset includes short, unstructured, natural language queries similar in style to real user interactions with virtual assistants.
• A range of benchmark classifiers (e.g., FastText, SVM, CNN, MLP, BERT) were evaluated on both in-scope intent classification and out-of-scope detection.
• Out-of-scope detection was evaluated using binary classification (in-scope vs. out-of-scope), with ablation studies on data augmentation and class balancing.
• The dataset was released publicly at https://github.com/clinc/oos-eval to support reproducible research and future benchmarking.

Experimental results

Research questions

• RQ1 How do state-of-the-art classifiers perform on in-scope intent classification when trained on our new dataset?
• RQ2 To what extent can existing models detect out-of-scope queries that are stylistically and topically similar to in-scope queries?
• RQ3 How does the availability of out-of-scope training data affect model performance on out-of-scope detection?
• RQ4 How does the performance of out-of-scope detection vary across different model architectures and data augmentation strategies?
• RQ5 Can existing benchmark datasets adequately support the evaluation of out-of-scope handling in real-world dialog systems?

Key findings

• BERT achieved the highest in-scope intent classification accuracy, exceeding 96% even under low-data and class-imbalanced settings.
• All models showed significant performance degradation on out-of-scope detection, with the best system achieving only 66% recall on the out-of-scope class.
• Increasing the number of out-of-scope training examples improved detection performance, but the gain was limited, indicating inherent model difficulty with out-of-distribution generalization.
• Data augmentation using Wikipedia sentences improved out-of-scope detection for some models, but was infeasible for others due to computational constraints.
• The dataset revealed that current benchmarks fail to capture the real-world challenge of out-of-scope queries, which are common and critical for system robustness.
• The study demonstrates a critical gap in current NLP evaluation: models excel on known classes but fail on unseen, out-of-scope inputs, even when they are semantically and stylistically similar to in-scope queries.