[Paper Review] Decision Attentive Regularization to Improve Simultaneous Speech Translation Systems

Mohd Abbas Zaidi, Beomseok Lee+2 more|arXiv (Cornell University)|Oct 13, 2021

Natural Language Processing Techniques20 references4 citations

TL;DR

This paper proposes Decision Attentive Regularization (DAR), a method that improves simultaneous speech translation (SimulST) by leveraging monotonic attention energies from a simultaneous text-to-text translation (SimulMT) model to refine read/write decision policies. DAR reduces latency while boosting translation quality, achieving a 34.66% relative improvement (4.5 BLEU points) over the baseline across multiple latency regimes on the MuST-C En-De dataset.

ABSTRACT

Simultaneous translation systems start producing the output while processing the partial source sentence in the incoming input stream. These systems need to decide when to read more input and when to write the output. These decisions depend on the structure of source/target language and the information contained in the partial input sequence. Hence, read/write decision policy remains the same across different input modalities, i.e., speech and text. This motivates us to leverage the text transcripts corresponding to the speech input for improving simultaneous speech-to-text translation (SimulST). We propose Decision Attentive Regularization (DAR) to improve the decision policy of SimulST systems by using the simultaneous text-to-text translation (SimulMT) task. We also extend several techniques from the offline speech translation domain to explore the role of SimulMT task in improving SimulST performance. Overall, we achieve 34.66% / 4.5 BLEU improvement over the baseline model across different latency regimes for the MuST-C English-German (EnDe) SimulST task.

Motivation & Objective

• To improve the read/write decision policy in simultaneous speech translation (SimulST) systems, which are inherently challenging due to speech input complexity.
• To investigate whether decision-making from a simultaneous text-to-text translation (SimulMT) model can serve as a supervisory signal for SimulST.
• To extend offline speech translation techniques—such as multitask learning, online knowledge distillation (KD), and cross-attentive regularization (CAR)—to the simultaneous translation setting.
• To develop a cross-modal regularization mechanism that aligns monotonic attention energies between speech and text inputs for improved decision policy learning.
• To achieve significant performance gains in SimulST without increasing inference latency.

Proposed method

• DAR computes cross-modal similarity between monotonic attention energy matrices of speech (SimulST) and text (SimulMT) inputs by projecting them into a shared space using self- and cross-attention operations.
• The method uses L2 distance between reconstructed attention representations (A^{s→t} and A^{t→t}) to measure similarity and minimize discrepancy during joint training.
• The proposed loss is integrated into the training of an MMA-based SimulST model, which uses monotonic multihead attention (MMA) to learn discrete read/write decisions via sigmoid-transformed attention energies.
• The approach combines DAR with existing techniques: multitask learning, online KD, CAR, and data augmentation to jointly optimize SimulST and SimulMT objectives.
• The model is trained end-to-end on the MuST-C En-De dataset, with attention energy alignment enforced during training to guide decision policy learning.
• The final model, MMA-DAR, integrates all components: data augmentation, multitask learning, online KD, CAR, and DAR, for optimal performance.

Experimental results

Research questions

• RQ1 Can the decision policy learned from SimulMT improve the performance of SimulST models?
• RQ2 Does aligning monotonic attention energies between speech and text inputs lead to better read/write decisions in SimulST?
• RQ3 To what extent do offline ST techniques like online KD and CAR generalize to the simultaneous translation setting?
• RQ4 Can cross-modal attention regularization reduce the latency-quality trade-off in SimulST without increasing inference delay?
• RQ5 Is the performance gain from DAR statistically significant across different latency regimes?

Key findings

• DAR achieves a 4.5 BLEU point improvement over the baseline MMA model across multiple latency regimes on the MuST-C En-De dataset.
• The relative improvement of 34.66% is consistent across all latency levels, with gains ranging from 0.7 to 1.2 BLEU points.
• The final MMA-DAR model, combining DAR with data augmentation, multitask learning, online KD, and CAR, achieves 22.35 BLEU on the MuST-C En-De test set.
• The improvement from DAR alone is statistically significant at the 99% confidence level (p = 0.002) via paired t-test.
• The model maintains lower latency than the baseline across all evaluation points, with MMA-DAR achieving higher BLEU scores at comparable or reduced latency.
• The gains from DAR are substantially higher than those from input-level regularization (e.g., CAR), indicating that decision policy regularization is more effective than feature-level distillation.