[Paper Review] Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech

Pengwei Wang, Xin Ye+3 more|arXiv (Cornell University)|May 10, 2021

Natural Language Processing Techniques32 references4 citations

TL;DR

This paper proposes Speech2Slot, an end-to-end knowledge-based slot filling model that formulates slot filling as a matching task between speech phoneme posteriors and an entity database, rather than sequence generation. By leveraging a trie-tree to detect slot boundaries and a bridge layer to decode slot phonemes, it achieves 83.86% accuracy on a large-scale Chinese voice navigation dataset, outperforming prior pipeline and end-to-end methods by 12.51% in accuracy, especially excelling on OOV and anti-linguistic slots.

ABSTRACT

In contrast to conventional pipeline Spoken Language Understanding (SLU) which consists of automatic speech recognition (ASR) and natural language understanding (NLU), end-to-end SLU infers the semantic meaning directly from speech and overcomes the error propagation caused by ASR. End-to-end slot filling (SF) from speech is an essential component of end-to-end SLU, and is usually regarded as a sequence-to-sequence generation problem, heavily relied on the performance of language model of ASR. However, it is hard to generate a correct slot when the slot is out-of-vovabulary (OOV) in training data, especially when a slot is an anti-linguistic entity without grammatical rule. Inspired by object detection in computer vision that is to detect the object from an image, we consider SF as the task of slot detection from speech. In this paper, we formulate the SF task as a matching task and propose an end-to-end knowledge-based SF model, named Speech-to-Slot (Speech2Slot), to leverage knowledge to detect the boundary of a slot from the speech. We also release a large-scale dataset of Chinese speech for slot filling, containing more than 830,000 samples. The experiments show that our approach is markedly superior to the conventional pipeline SLU approach, and outperforms the state-of-the-art end-to-end SF approach with 12.51% accuracy improvement.

Motivation & Objective

• To address error propagation in pipeline Spoken Language Understanding (SLU) caused by automatic speech recognition (ASR) errors, especially for out-of-vocabulary (OOV) and anti-linguistic slots.
• To overcome the limitations of sequence-to-sequence generation in end-to-end slot filling, which relies heavily on language models and struggles with OOV and non-grammatical entities.
• To propose a novel formulation of slot filling as a matching task between speech phoneme posteriors and a knowledge base of entities, inspired by object detection in computer vision.
• To release a large-scale, multilingual (Chinese) speech-to-slot dataset with over 830,000 samples to support future research in end-to-end SLU.

Proposed method

• Formulate slot filling as a matching task: detect the start and end timestamps of a slot in the phoneme posterior by matching against an entity database using a trie-tree structure.
• Use a speech encoder to process raw audio and generate phoneme posteriors, which serve as input for slot boundary detection.
• Construct a trie-tree from the entity database to efficiently search for matching phoneme fragments in the speech input.
• Apply a bridge layer to decode the final phone sequence of the detected slot from the entity database based on the matched phoneme fragment.
• Train the end-to-end model using a joint optimization of acoustic modeling and slot matching, with attention mechanisms to align speech and knowledge.
• Leverage a knowledge base (entity database) as a closed set of possible slots, enabling detection rather than generation.

Experimental results

Research questions

• RQ1 Can formulating slot filling as a matching task between speech and a knowledge base outperform sequence-to-sequence generation in handling OOV and anti-linguistic slots?
• RQ2 How does the performance of an end-to-end slot filling model based on matching compare to conventional pipeline SLU and state-of-the-art end-to-end generation-based models?
• RQ3 To what extent can a knowledge-based matching approach mitigate error propagation from ASR and improve robustness on real human speech?
• RQ4 How effective is the proposed method on out-of-vocabulary (OOV) slots, especially those with no grammatical structure?

Key findings

• Speech2Slot achieves 83.86% slot filling accuracy on the TTS-generated test set using a domain-specific acoustic model, significantly outperforming the next best baseline (AM+SF) by 12.51 percentage points.
• On the OOV subset, Speech2Slot achieves 81.90% accuracy, demonstrating robustness to unseen slots, while the pipeline AM+LM method only reaches 23.18%.
• Even with a general-purpose acoustic model, Speech2Slot achieves 72.99% overall accuracy on the human-read test set, outperforming the next best method (AM+SF) by 28.98 percentage points.
• On the OOV subset of human speech, Speech2Slot achieves 16.50% accuracy, which is substantially higher than the 0.95% of the next best method, confirming its effectiveness in real-world scenarios.
• The model maintains consistent performance across OOV and non-OOV slots, indicating that the matching-based approach is less sensitive to vocabulary coverage than generation-based methods.
• The ablation study confirms that the matching mechanism is more effective than generation for handling anti-linguistic entities like music or movie titles, which are often misrecognized by language models.