[Paper Review] Bridging the Modality Gap for Speech-to-Text Translation

Yuchen Liu, Junnan Zhu+2 more|arXiv (Cornell University)|Oct 28, 2020

Speech Recognition and Synthesis38 references39 citations

TL;DR

The STAST model decouples the speech translation encoder, introduces a shrink mechanism, integrates a text-based MT model in a shared latent space, and applies cross-modal adaptation to bridge speech-text modality gaps, achieving state-of-the-art results on English-French and English-German ST tasks.

ABSTRACT

End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously, which ignores the speech-and-text modality differences and makes the encoder overloaded, leading to great difficulty in learning such a model. To address these issues, we propose a Speech-to-Text Adaptation for Speech Translation (STAST) model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text. Specifically, we decouple the speech translation encoder into three parts and introduce a shrink mechanism to match the length of speech representation with that of the corresponding text transcription. To obtain better semantic representation, we completely integrate a text-based translation model into the STAST so that two tasks can be trained in the same latent space. Furthermore, we introduce a cross-modal adaptation method to close the distance between speech and text representation. Experimental results on English-French and English-German speech translation corpora have shown that our model significantly outperforms strong baselines, and achieves the new state-of-the-art performance.

Motivation & Objective

• Motivate end-to-end speech-to-text translation and address modality gaps between speech and text.
• Decouple the speech translation encoder into specialized components to reduce learning burden.
• Leverage a text-based MT model within the ST model to align semantic representations.
• Close the speech-text representation gap with cross-modal adaptation.
• Demonstrate state-of-the-art performance on English-French and English-German ST benchmarks.

Proposed method

• Decouple the ST encoder into three parts: an acoustic encoder, a shrink mechanism, and a semantic encoder.
• Use a CTC module on the acoustic encoder to predict source transcription and apply a shrink mechanism to align speech representation length with transcription length.
• Integrate a complete MT model by sharing semantic encoder/decoder and aligning CTC, embedding, and MT output spaces.
• Apply cross-modal adaptation to align speech and text representations, with sequence-level or word-level MSE-based losses.
• Train with a combination of CTC loss, ST loss, MT loss, and cross-modal adaptation loss in a multi-task setup.

Experimental results

Research questions

• RQ1 Can decoupling the ST encoder ease learning by separating acoustic and semantic representation learning?
• RQ2 Does a shrink mechanism effectively solve length mismatch between speech frames and text tokens?
• RQ3 Can integrating MT in a shared latent space improve semantic representation for ST?
• RQ4 Does cross-modal adaptation further reduce modality distance and improve translation accuracy?

Key findings

• STAST achieves state-of-the-art or competitive BLEU on Augmented LibriSpeech En-Fr, outperforming several end-to-end baselines.
• On MuST_C En-De, STAST surpasses all previous end-to-end models in the base setting and remains strong with SpecAugment.
• Ablations show all proposed components (shrink, semantic encoder, cross-modal adaptation, multi-task learning, CTC loss) contribute positively to performance.
• Sequence-level cross-modal adaptation generally performs slightly better than word-level adaptation.
• STAST demonstrates robustness to data scales and benefits from extra ASR data and, to a degree, extra MT data.