[Paper Review] Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents

Aditya Siddhant, Anuj Goyal+1 more|arXiv (Cornell University)|Nov 13, 2018

Topic Modeling21 references4 citations

TL;DR

This paper proposes ELMo-Light (ELMoL), a lightweight unsupervised pre-training method for Spoken Language Understanding (SLU) that leverages large-scale unlabeled ASR transcripts to improve intent classification and entity tagging. It demonstrates that unsupervised transfer using ELMoL outperforms training from scratch and even surpasses conventional supervised transfer, especially in low-resource settings—matching the performance of 10–15× more labeled data with only 1,000 in-domain samples.

ABSTRACT

User interaction with voice-powered agents generates large amounts of unlabeled utterances. In this paper, we explore techniques to efficiently transfer the knowledge from these unlabeled utterances to improve model performance on Spoken Language Understanding (SLU) tasks. We use Embeddings from Language Model (ELMo) to take advantage of unlabeled data by learning contextualized word representations. Additionally, we propose ELMo-Light (ELMoL), a faster and simpler unsupervised pre-training method for SLU. Our findings suggest unsupervised pre-training on a large corpora of unlabeled utterances leads to significantly better SLU performance compared to training from scratch and it can even outperform conventional supervised transfer. Additionally, we show that the gains from unsupervised transfer techniques can be further improved by supervised transfer. The improvements are more pronounced in low resource settings and when using only 1000 labeled in-domain samples, our techniques match the performance of training from scratch on 10-15x more labeled in-domain data.

Motivation & Objective

• To address the high cost and scalability limitations of supervised SLU training by leveraging vast amounts of unlabeled ASR-generated utterances.
• To develop a faster, lighter alternative to ELMo for deployment in real-time, resource-constrained SLU systems.
• To evaluate the effectiveness of unsupervised transfer learning on both benchmark and real-world commercial SLU datasets.
• To investigate the additive benefits of combining unsupervised and supervised transfer learning for low-resource SLU tasks.
• To demonstrate that unsupervised pre-training can outperform supervised transfer when labeled data is scarce.

Proposed method

• The method uses ELMo-style contextualized word embeddings trained on large-scale unlabeled ASR transcripts to capture rich linguistic representations for SLU.
• ELMo-Light (ELMoL) is introduced as a simplified, faster, and smaller alternative to ELMo, with reduced parameter count (37M vs. 140M) and lower memory usage.
• The approach applies unsupervised pre-training on domain-incoherent unlabeled data, followed by fine-tuning on small amounts of labeled in-domain data.
• Training strategies such as gradual unfreezing (guf), discriminative fine-tuning (discr), and triangular learning rate (tlr) are used to prevent catastrophic forgetting during fine-tuning.
• Supervised transfer is applied by fine-tuning the pre-trained model on labeled in-domain data, combining both unsupervised and supervised knowledge transfer.
• The framework is evaluated on both public benchmarks (ATIS, SNIPS) and internal commercial datasets across varying data resource levels.

Experimental results

Research questions

• RQ1 Can unsupervised pre-training on large-scale unlabeled ASR transcripts significantly improve SLU performance compared to training from scratch?
• RQ2 Does ELMo-Light (ELMoL) achieve comparable performance to ELMo while being significantly faster and more memory-efficient?
• RQ3 How do unsupervised and supervised transfer techniques interact, and is their combination additive in performance gain?
• RQ4 To what extent can unsupervised transfer reduce the dependency on large labeled datasets in low-resource SLU settings?
• RQ5 Can unsupervised transfer outperform conventional supervised transfer when labeled data is limited?

Key findings

• ELMo-Light (ELMoL) achieves SER scores of 36.86 and 28.42 on internal domains A and B, respectively, outperforming FastText and approaching ELMo’s performance of 36.58 and 27.95.
• On the ATIS dataset, ELMoL achieves 13.38% SER, outperforming FastText (15.01%) and approaching ELMo (12.65%), while on SNIPS it achieves 15.62% SER, outperforming the SOTA of 16.00%.
• With only 1,000 in-domain labeled samples, ELMoL-based models match the performance of training from scratch on 10–15× more labeled data (10,000–15,000 samples).
• Combining unsupervised and supervised transfer (UT+ST) yields additive performance gains: ELMo+ST achieves 35.86% SER on Domain A, down from 36.58% with UT alone.
• ELMoL is 1.6x faster at inference and 1.8x faster at training than ELMo, with 4x less memory usage (37M vs. 140M parameters).
• The ablation study confirms that training techniques like gradual unfreezing and discriminative fine-tuning consistently improve ELMoL performance across datasets.