[Paper Review] Pre-Training for Query Rewriting in A Spoken Language Understanding System

Zheng Chen, Xing Fan+3 more|arXiv (Cornell University)|Feb 13, 2020

Topic Modeling22 references4 citations

TL;DR

This paper proposes a neural retrieval-based query rewriting (QR) system for spoken language understanding (SLU) that leverages pre-trained contextual embeddings from historical user conversations and NLU hypotheses to significantly reduce training data requirements. Pre-training on large-scale, unlabeled dialogue data enables the model to achieve strong performance with minimal fine-tuning, outperforming a fully fine-tuned baseline even when trained on just 20% of the QR data.

ABSTRACT

Query rewriting (QR) is an increasingly important technique to reduce customer friction caused by errors in a spoken language understanding pipeline, where the errors originate from various sources such as speech recognition errors, language understanding errors or entity resolution errors. In this work, we first propose a neural-retrieval based approach for query rewriting. Then, inspired by the wide success of pre-trained contextual language embeddings, and also as a way to compensate for insufficient QR training data, we propose a language-modeling (LM) based approach to pre-train query embeddings on historical user conversation data with a voice assistant. In addition, we propose to use the NLU hypotheses generated by the language understanding system to augment the pre-training. Our experiments show pre-training provides rich prior information and help the QR task achieve strong performance. We also show joint pre-training with NLU hypotheses has further benefit. Finally, after pre-training, we find a small set of rewrite pairs is enough to fine-tune the QR model to outperform a strong baseline by full training on all QR training data.

Motivation & Objective

• To reduce user friction in spoken language understanding (SLU) systems caused by speech recognition and NLU errors.
• To address the scarcity of high-quality labeled query rewriting (QR) training pairs by leveraging pre-training on large-scale historical dialogue data.
• To improve QR performance by incorporating NLU hypotheses as weak supervision during pre-training.
• To demonstrate that pre-trained models require far less labeled QR data to achieve superior performance compared to standard fine-tuning.

Proposed method

• Frame QR as an information retrieval task using a neural retriever with a query embedder based on BERT.
• Pre-train the query embedder on historical user sessions using masked language modeling (MLM) on raw utterances.
• Enhance pre-training by jointly optimizing with NLU hypotheses (domain, intent, slots) as weak supervision.
• Use multi-head self-attention pooling to generate fixed-length utterance embeddings from contextualized token representations.
• Fine-tune the pre-trained model on a small subset (20%) of the QR training data to adapt to the rewriting task.
• Evaluate performance using p@1, p@5, p@10, p@20, and MRR metrics on held-out test data.

Experimental results

Research questions

• RQ1 Can pre-training on large-scale, unlabeled historical user conversations improve query rewriting performance?
• RQ2 Does incorporating NLU hypotheses during pre-training further enhance QR model performance?
• RQ3 To what extent can pre-training reduce the need for labeled QR training data?
• RQ4 How does the performance of a pre-trained model compare to a fully fine-tuned baseline when trained on limited QR data?

Key findings

• Pre-training on historical user conversations alone (pretrain text) achieves p@1 of 0.284, outperforming the baseline (0.293) without fine-tuning.
• Joint pre-training with NLU hypotheses (pretrain textNLU) improves p@1 to 0.290 and p@5 to 0.481, showing added benefit from weak supervision.
• Fine-tuning the pre-trained model on just 20% of the QR data yields p@1 of 0.315, a 7.5% relative gain over the baseline.
• The pre-trained model achieves near-optimal performance with only 20% of the QR data, while the baseline continues to improve with more data, indicating data efficiency.
• Performance analysis shows the pre-trained model maintains a consistent advantage, especially in low-data regimes, as shown in Figure 3.