[Paper Review] Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability

Antoine Caubrière, Natalia Tomashenko+4 more|arXiv (Cornell University)|Jun 18, 2019

Speech Recognition and Synthesis31 references46 citations

TL;DR

The paper proposes an end-to-end SLU model trained with curriculum-based transfer learning to extract semantic concepts directly from speech, achieving state-of-the-art results on French MEDIA/PORTMEDIA and addressing domain portability.

ABSTRACT

We present an end-to-end approach to extract semantic concepts directly from the speech audio signal. To overcome the lack of data available for this spoken language understanding approach, we investigate the use of a transfer learning strategy based on the principles of curriculum learning. This approach allows us to exploit out-of-domain data that can help to prepare a fully neural architecture. Experiments are carried out on the French MEDIA and PORTMEDIA corpora and show that this end-to-end SLU approach reaches the best results ever published on this task. We compare our approach to a classical pipeline approach that uses ASR, POS tagging, lemmatizer, chunker... and other NLP tools that aim to enrich ASR outputs that feed an SLU text to concepts system. Last, we explore the promising capacity of our end-to-end SLU approach to address the problem of domain portability.

Motivation & Objective

• Address lack of labeled end-to-end SLU data by leveraging curriculum-based transfer learning to pretrain on easier, related tasks.
• Develop an end-to-end neural SLU architecture that directly maps speech to semantic concepts and values.
• Evaluate crossing from MEDIA to PORTMEDIA domains to study domain portability.
• Compare end-to-end approach with classical pipeline SLU systems and assess benefits of the starred mode for semantic extraction.

Proposed method

• Use an end-to-end neural SLU architecture inspired by Deep Speech 2 with spectrogram inputs and CTC loss to predict words and semantic tags.
• Represent semantic slots with dedicated start tags for each concept and a single end tag, enabling concept/value extraction.
• Introduce a starred mode that replaces text between concept tags with a single star to emphasize semantic regions during training.
• Apply curriculum-based transfer learning by sequencing training data from generic ASR/transcriptions to named entity annotations to combined MEDIA/PORTMEDIA concepts, reinitializing only the output layer at each step.
• Fine-tune with beam-search decoding and 5-gram LM rescoring to improve CER/CVER.

Experimental results

Research questions

• RQ1 Can an end-to-end SLU model trained with curriculum-based transfer learning surpass traditional pipeline approaches on French SLU tasks?
• RQ2 Does integrating MEDIA and PORTMEDIA data in a curriculum improve slot filling performance and enable domain portability?
• RQ3 What is the impact of the starred mode and LM rescoring on end-to-end SLU performance?
• RQ4 How effective is the proposed approach for transferring SLU models across related domains (MEDIA to PORTMEDIA)?

Key findings

• End-to-end SLU with curriculum-based transfer learning achieves substantial CER/CVER reductions over training only on MEDIA data (e.g., 39.8% CER and 52.1% CVER for SF_M alone vs. 21.6% CER and 27.7% CVER with the full curriculum on MEDIA.
• Pretraining on ASR and then transferring to MEDIA (ASR•SF_M) reduces CER to 23.7% and CVER to 30.3%.
• Curriculum chaining with merged MEDIA+PORTMEDIA (ASR•SF_P+M•SF_M) further reduces CER to 22.2% and CVER to 28.8%.
• Full curriculum including NER between steps yields 21.6% CER and 27.7% CVER on MEDIA in greedy mode; beam-rescoring with LM further improves to 18.1% CER and 22.1% CVER (with STARred mode in later steps achieving 16.4% CER and 20.9% CVER in some configurations).
• Compared to a pipeline CRF system, the end-to-end approach with strong ASR can achieve competitive CER/CVER (e.g., 16.1% CER and 20.4% CVER for pipeline with full features), with differences not statistically significant at 95% CI.