[Paper Review] A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding

Young‐Bum Kim, Dongchan Kim+2 more|arXiv (Cornell University)|Apr 22, 2018

Topic Modeling33 references3 citations

TL;DR

This paper proposes a scalable two-stage neural approach—shortlisting followed by reranking—for large-scale domain classification in natural language understanding. It uses lightweight BiLSTM models to efficiently identify the top-k candidate domains and then applies list-wise reranking with contextual BiLSTM models to improve accuracy, achieving 91.13% top-1 accuracy on 1,500 domains.

ABSTRACT

Intelligent personal digital assistants (IPDAs), a popular real-life application with spoken language understanding capabilities, can cover potentially thousands of overlapping domains for natural language understanding, and the task of finding the best domain to handle an utterance becomes a challenging problem on a large scale. In this paper, we propose a set of efficient and scalable neural shortlisting-reranking models for large-scale domain classification in IPDAs. The shortlisting stage focuses on efficiently trimming all domains down to a list of k-best candidate domains, and the reranking stage performs a list-wise reranking of the initial k-best domains with additional contextual information. We show the effectiveness of our approach with extensive experiments on 1,500 IPDA domains.

Motivation & Objective

• Address the challenge of scalable domain classification in intelligent personal digital assistants (IPDAs) with thousands of overlapping domains.
• Overcome the limitations of traditional schema-based systems that become bottlenecks as new domains are added.
• Reduce user interaction friction caused by requiring explicit domain mentions in utterances.
• Develop a system that efficiently identifies the most relevant domain from a large set of overlapping domains using neural models.
• Improve classification accuracy by combining lightweight shortlisting with context-rich reranking using BiLSTM-based models.

Proposed method

• Use a lightweight BiLSTM model in the shortlisting stage that processes only character and word-level features to generate a k-best list of candidate domains.
• Apply a list-wise reranking model using a second BiLSTM that incorporates rich contextual information and cross-hypothesis features from the k-best candidates.
• Employ three types of reranking strategies: point-wise, pair-wise, and list-wise, with the list-wise approach showing superior performance.
• Use softmax over all domains ($smx_a$) and a top-k focused softmax ($smx_b$) to compare scoring strategies, with $smx_b$ showing better robustness in large-scale settings.
• Train models using Dynet with Adam optimization and variational dropout for regularization, ensuring generalization and stability.
• Evaluate the system under both traditional (tens of domains) and large-scale (1,500 domains) settings to demonstrate scalability and performance gains.

Experimental results

Research questions

• RQ1 Can a two-stage neural shortlisting-reranking framework effectively scale to 1,500 overlapping domains in NLU systems?
• RQ2 How does the inclusion of contextual and cross-hypothesis features in reranking improve domain classification accuracy compared to relying solely on shortlisting?
• RQ3 What is the relative performance of point-wise, pair-wise, and list-wise reranking strategies in large-scale domain classification?
• RQ4 Does using a top-k focused softmax ($smx_b$) outperform a full-domain softmax ($smx_a$) in scenarios with high domain overlap?
• RQ5 To what extent can lightweight shortlisting models achieve high accuracy, and how much does reranking improve performance on large-scale datasets?

Key findings

• The shortlisting model achieves over 95% accuracy on the top-1 prediction in the traditional IPDA setting (tens of domains), with near 99% accuracy when the ground-truth domain is within the top 5.
• In the large-scale setting with 1,500 domains, the top-1 accuracy of the shortlisting model alone drops to 81.38% ($smx_a$) and 81.49% ($smx_b$), indicating the need for reranking.
• The list-wise reranking model $LSTM^C$ achieves the highest accuracy of 91.13% under the $smx_b$ setting, significantly outperforming point-wise and pair-wise approaches.
• The performance gap between $smx_a$ and $smx_b$ widens in large-scale settings, suggesting that $smx_b$ better handles overlapping domains by preserving relative confidence across candidates.
• Adding manually engineered cross-hypothesis features to the automatically learned LSTM features does not improve performance, indicating that the model’s learned representations are already effective.
• The upper bound of HypRank performance is set by the shortlisting stage, with the $k$-best list (especially k=5) capturing most of the relevant candidates, and performance levels off beyond k=5.