[Paper Review] Adapting Pretrained Transformer to Lattices for Spoken Language Understanding

Chao-Wei Huang, Yun-Nung Chen|arXiv (Cornell University)|Nov 2, 2020

Speech Recognition and Synthesis16 references4 citations

TL;DR

This paper proposes adapting pre-trained transformers to process lattice inputs from automatic speech recognition (ASR) for improved spoken language understanding (SLU). By introducing lattice reachability masks and positional encoding, the model effectively encodes multiple ASR hypotheses, achieving 20.6% relative error reduction over 1-best transcripts on the ATIS benchmark under varying acoustic conditions.

ABSTRACT

Lattices are compact representations that encode multiple hypotheses, such as speech recognition results or different word segmentations. It is shown that encoding lattices as opposed to 1-best results generated by automatic speech recognizer (ASR) boosts the performance of spoken language understanding (SLU). Recently, pretrained language models with the transformer architecture have achieved the state-of-the-art results on natural language understanding, but their ability of encoding lattices has not been explored. Therefore, this paper aims at adapting pretrained transformers to lattice inputs in order to perform understanding tasks specifically for spoken language. Our experiments on the benchmark ATIS dataset show that fine-tuning pretrained transformers with lattice inputs yields clear improvement over fine-tuning with 1-best results. Further evaluation demonstrates the effectiveness of our methods under different acoustic conditions. Our code is available at https://github.com/MiuLab/Lattice-SLU

Motivation & Objective

• Address the limitation of 1-best ASR transcripts in spoken language understanding (SLU), which are prone to transcription errors that degrade downstream NLU performance.
• Explore whether pre-trained transformer models—known for strong performance on text—can effectively encode uncertain, lattice-structured inputs from ASR systems.
• Develop a method to adapt transformer architectures to handle lattices while preserving their strong inductive bias and transfer learning capabilities.
• Demonstrate consistent performance gains across diverse acoustic conditions, especially when ASR quality is poor.
• Investigate sample efficiency and robustness to ASR error rates, showing that lattice inputs reduce reliance on high-quality transcriptions.

Proposed method

• Introduce lattice reachability masks to represent the structure of word lattices, allowing the transformer to attend only over valid paths in the lattice.
• Design a lattice positional encoding scheme that assigns positional embeddings based on the topological order of nodes in the lattice, preserving structural context.
• Fine-tune a pre-trained GPT-style transformer on SLU tasks using lattice inputs instead of 1-best ASR outputs.
• Use a linear classifier on the [CLS] token representation for intent detection and slot prediction, with end-to-end fine-tuning.
• Apply a linear warm-up learning rate schedule with Adam optimizer, training for 5 epochs with a batch size of 8.
• Evaluate two variants: one with standard attention and one with lattice-aware attention, both using the same masked input representation.

Experimental results

Research questions

• RQ1 Can pre-trained transformers be effectively adapted to encode lattice-structured inputs from ASR systems for SLU tasks?
• RQ2 Does using lattice inputs instead of 1-best transcripts lead to measurable improvements in intent detection and slot prediction performance?
• RQ3 How does the model’s performance vary under different acoustic conditions, particularly when ASR word error rates are high?
• RQ4 Is the model sample-efficient when training on limited data, and does it require more examples to learn from noisy lattice inputs?
• RQ5 To what extent do lattice inputs improve robustness to ASR errors compared to 1-best transcripts?

Key findings

• Fine-tuning pre-trained transformers with lattice inputs yields 32.4%, 15.4%, and 20.6% relative error reduction in intent detection across three acoustic conditions compared to 1-best baselines.
• The model achieves consistent performance gains across all acoustic conditions, with the largest improvement observed under high word error rate (WER) conditions.
• Lattice inputs improve performance even when linearized into topological order, suggesting that the presence of alternative word forms enhances understanding.
• The model maintains a 2% F1 advantage over 1-best baselines across all training data sizes from 300 to 5,000 examples, indicating strong sample efficiency.
• When WER exceeds 15%, lattice-based models outperform 1-best models, confirming their robustness to poor ASR quality.
• Qualitative analysis confirms that the model correctly identifies intent even when the correct word (e.g., 'far') appears in the lattice with low probability, demonstrating effective lattice utilization.