[Paper Review] Learning Shared Semantic Space for Speech-to-Text Translation

Chi Han, Mingxuan Wang+2 more|arXiv (Cornell University)|May 7, 2021

Natural Language Processing Techniques57 references7 citations

TL;DR

Chimera proposes a shared semantic memory network that bridges speech and text representations via a modality-agnostic embedding space, enabling end-to-end speech-to-text translation to leverage massive text machine translation (MT) data. By projecting audio and text features into a unified semantic space and using a bi-modal contrastive loss, Chimera achieves a new state-of-the-art 27.1 BLEU on MuST-C EN-DE, improving SOTA by +1.9 BLEU while demonstrating effective knowledge transfer across modalities.

ABSTRACT

Having numerous potential applications and great impact, end-to-end speech translation (ST) has long been treated as an independent task, failing to fully draw strength from the rapid advances of its sibling - text machine translation (MT). With text and audio inputs represented differently, the modality gap has rendered MT data and its end-to-end models incompatible with their ST counterparts. In observation of this obstacle, we propose to bridge this representation gap with Chimera. By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art. Specifically, Chimera obtains 27.1 BLEU on MuST-C EN-DE, improving the SOTA by a +1.9 BLEU margin. Further experimental analyses demonstrate that the shared semantic space indeed conveys common knowledge between these two tasks and thus paves a new way for augmenting training resources across modalities. Code, data, and resources are available at https://github.com/Glaciohound/Chimera-ST.

Motivation & Objective

• To address the modality gap between speech and text in end-to-end speech translation (ST), which limits performance and prevents leveraging large-scale text machine translation (MT) data.
• To unify ST and MT workflows by learning a shared semantic representation that captures modality-agnostic linguistic meaning.
• To enable knowledge transfer from high-resource MT data to low-resource ST tasks through a unified training framework.
• To validate that the shared semantic space captures meaningful, structured semantic and syntactic patterns across modalities.

Proposed method

• Chimera employs a shared semantic memory module that projects both speech and text features into a common, modality-agnostic embedding space.
• It introduces a bi-modal contrastive learning objective to align semantic memories from speech and text inputs, encouraging similar representations for semantically equivalent content.
• The model uses a Transformer-based architecture with multiple heads to extract key semantic categories from input sequences.
• Pre-training on large-scale MT corpora (e.g., OpenSubtitles) is used to initialize the shared semantic space before fine-tuning on speech-text parallel data.
• A contrastive loss is applied between paired speech and text samples to ensure that semantically equivalent inputs are mapped close in the shared space.
• The architecture supports joint training of ST and MT objectives, enabling knowledge sharing across tasks.

Experimental results

Research questions

• RQ1 Can a shared semantic space effectively bridge the representation gap between speech and text in end-to-end speech translation?
• RQ2 Does pre-training on large-scale text machine translation data significantly improve ST performance when combined with a unified semantic representation?
• RQ3 Is the learned shared semantic space structured in a way that reflects semantic and syntactic patterns, such as question formation or verb types?
• RQ4 To what extent does the bi-modal contrastive training task improve alignment between speech and text representations?
• RQ5 Can the unified model achieve state-of-the-art performance across multiple language directions in speech translation benchmarks?

Key findings

• Chimera achieves a new state-of-the-art 27.1 BLEU score on the MuST-C EN-DE translation direction, surpassing the previous SOTA by +1.9 BLEU.
• Performance improves significantly with larger MT corpora during pretraining, confirming the value of high-resource MT data in boosting ST performance.
• Visualization via PCA shows that speech and text representations are projected close to each other in the same semantic memory regions, validating modality gap bridging.
• The shared semantic space exhibits structured clustering by semantic and syntactic patterns, such as questions, future tense, and copular verbs, indicating meaningful representation learning.
• Inter-modal attention visualization reveals alignment between speech pauses and text punctuation (e.g., periods), suggesting semantic rather than positional attention.
• Ablation studies confirm the effectiveness of the bi-modal contrastive loss and the shared semantic memory module in improving ST performance.