[Paper Review] Translatotron 2: Robust direct speech-to-speech translation.

Jia Ye, Michelle Tadmor Ramanovich+2 more|arXiv (Cornell University)|Jul 19, 2021

Speech Recognition and Synthesis47 references14 citations

TL;DR

Translatotron 2 is an end-to-end neural speech-to-speech translation model that improves translation quality and speech naturalness while reducing over-generation artifacts like babbling and pauses. It integrates a speech encoder, phoneme decoder, mel-spectrogram synthesizer, and attention mechanism, and introduces speaker voice retention to enhance robustness and prevent misuse in production settings.

ABSTRACT

We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a phoneme decoder, a mel-spectrogram synthesizer, and an attention module that connects all the previous three components. Experimental results suggest that Translatotron 2 outperforms the original Translatotron by a large margin in terms of translation quality and predicted speech naturalness, and drastically improves the robustness of the predicted speech by mitigating over-generation, such as babbling or long pause. We also propose a new method for retaining the source speaker's voice in the translated speech. The trained model is restricted to retain the source speaker's voice, and unlike the original Translatotron, it is not able to generate speech in a different speaker's voice, making the model more robust for production deployment, by mitigating potential misuse for creating spoofing audio artifacts. When the new method is used together with a simple concatenation-based data augmentation, the trained Translatotron 2 model is able to retain each speaker's voice for input with speaker turns.

Motivation & Objective

• To develop a more robust and natural end-to-end speech-to-speech translation system that outperforms prior models.
• To mitigate over-generation issues such as babbling and long pauses in synthesized speech.
• To retain the source speaker's voice in translated output, enhancing authenticity and reducing spoofing risks.
• To enable production deployment by restricting voice synthesis to the source speaker's voice.
• To improve model robustness through a novel voice retention method combined with data augmentation.

Proposed method

• The model architecture integrates a speech encoder, phoneme decoder, mel-spectrogram synthesizer, and an attention module connecting all components.
• A new voice retention mechanism ensures the translated speech preserves the source speaker's voice characteristics.
• The model is trained end-to-end with a constraint that prevents generation in a different speaker's voice.
• A concatenation-based data augmentation technique is applied to improve speaker turn handling in multi-speaker inputs.
• The attention module dynamically aligns the speech encoder output with the phoneme decoder and synthesizer for better context integration.
• The system uses a sequence-to-sequence framework with joint optimization of translation and speech synthesis components.

Experimental results

Research questions

• RQ1 Can an end-to-end speech-to-speech translation model achieve superior translation quality and speech naturalness compared to prior models?
• RQ2 How effectively can the model mitigate over-generation artifacts like babbling and long pauses?
• RQ3 Can the model reliably retain the source speaker's voice in translated speech to prevent misuse?
• RQ4 To what extent does speaker voice retention improve robustness for real-world deployment?
• RQ5 How does data augmentation enhance the model’s ability to handle speaker turns in multi-speaker scenarios?

Key findings

• Translatotron 2 significantly outperforms the original Translatotron in translation quality and speech naturalness.
• The model drastically reduces over-generation artifacts such as babbling and long pauses.
• Speaker voice retention is successfully enforced, preventing the model from generating speech in a different speaker's voice.
• The combination of voice retention and data augmentation enables effective handling of speaker turns in input speech.
• The model achieves improved robustness for production deployment by minimizing spoofing risks through constrained voice synthesis.
• The system maintains high performance even when translating speech with multiple speakers, thanks to the proposed data augmentation and voice retention method.