[Paper Review] Bayes risk CTC: Controllable CTC alignment in Sequence-to-Sequence tasks

Jinchuan Tian, Brian Yan+4 more|arXiv (Cornell University)|Oct 14, 2022

Algorithms and Data Compression4 citations

TL;DR

This paper proposes Bayes Risk CTC (BRCTC), a novel CTC criterion that makes alignment prediction controllable by incorporating a customizable Bayes risk function during training. By reweighting paths based on desired properties—such as earlier token emissions or reduced sequence length—BRCTC enables significant reductions in inference cost (up to 47%) and latency (30% reduction), while preserving transcription accuracy across offline and online sequence-to-sequence tasks like ASR and machine translation.

ABSTRACT

Sequence-to-Sequence (seq2seq) tasks transcribe the input sequence to a target sequence. The Connectionist Temporal Classification (CTC) criterion is widely used in multiple seq2seq tasks. Besides predicting the target sequence, a side product of CTC is to predict the alignment, which is the most probable input-long sequence that specifies a hard aligning relationship between the input and target units. As there are multiple potential aligning sequences (called paths) that are equally considered in CTC formulation, the choice of which path will be most probable and become the predicted alignment is always uncertain. In addition, it is usually observed that the alignment predicted by vanilla CTC will drift compared with its reference and rarely provides practical functionalities. Thus, the motivation of this work is to make the CTC alignment prediction controllable and thus equip CTC with extra functionalities. The Bayes risk CTC (BRCTC) criterion is then proposed in this work, in which a customizable Bayes risk function is adopted to enforce the desired characteristics of the predicted alignment. With the risk function, the BRCTC is a general framework to adopt some customizable preference over the paths in order to concentrate the posterior into a particular subset of the paths. In applications, we explore one particular preference which yields models with the down-sampling ability and reduced inference costs. By using BRCTC with another preference for early emissions, we obtain an improved performance-latency trade-off for online models. Experimentally, the proposed BRCTC reduces the inference cost of offline models by up to 47% without performance degradation and cuts down the overall latency of online systems to an unseen level.

Motivation & Objective

• To address the lack of control in vanilla CTC's alignment prediction, which often drifts from reference alignments and lacks practical utility.
• To enable customization of CTC alignment characteristics—such as earlier emissions or reduced sequence length—through a principled training criterion.
• To provide a general framework that maintains transcription accuracy while enhancing alignment controllability for both offline and online sequence-to-sequence models.
• To reduce inference cost and latency in real-world applications by leveraging path-specific risk functions during training.
• To demonstrate that BRCTC can achieve state-of-the-art trade-offs between performance, latency, and computational efficiency in ASR and machine translation tasks.

Proposed method

• Reformulates the CTC forward-backward algorithm using a divide-and-conquer approach, grouping paths based on customizable properties (e.g., emission timing, length).
• Introduces a Bayes risk function that assigns higher risk values to preferred path groups, guiding the model to concentrate posterior probability on paths with desired characteristics.
• Preserves the original CTC objective by summing over all paths during training, ensuring transcription accuracy is maintained.
• Applies a trimming technique post-training to further reduce sequence length and inference cost, especially effective in offline models.
• Employs a risk factor λ to balance alignment preference (e.g., early emission) with transcription performance, enabling tunable trade-offs in online systems.
• Uses gradient analysis and attention visualization to validate that BRCTC shifts emission peaks earlier and enables efficient down-sampling in later encoder layers.

Experimental results

Research questions

• RQ1 Can CTC alignment prediction be made controllable to reduce inference cost and latency without degrading transcription accuracy?
• RQ2 How can a Bayes risk function be integrated into CTC training to steer the model toward preferred path characteristics such as earlier emissions or shorter sequences?
• RQ3 To what extent can BRCTC reduce inference cost in offline models while maintaining competitive ASR or MT performance?
• RQ4 Can BRCTC achieve unprecedented latency reductions in online sequence-to-sequence models compared to vanilla CTC?
• RQ5 How do the attention patterns and gradient dynamics in BRCTC differ from vanilla CTC, particularly in terms of emission timing and down-sampling efficiency?

Key findings

• BRCTC reduces the inference cost of offline models by up to 47% without performance degradation, as demonstrated on LibriSpeech and Aishell-2 datasets.
• In online ASR, BRCTC achieves a minimum overall latency of 302ms on Aishell-2, representing a 30% reduction compared to the 431ms baseline of vanilla CTC.
• The model with BRCTC maintains competitive transcription quality, achieving BLEU scores of 31.7 (De-En) and 38.0 (Es-En) on IWSLT14, comparable to the baseline with vanilla CTC.
• The down-sampling process in BRCTC is localized primarily in the last two encoder layers before the CTC layer, indicating efficient global context aggregation.
• Gradient analysis shows that BRCTC shifts emission peaks earlier during training, with gradients localized to earlier time steps, confirming the model's ability to learn early emissions.
• Visualization confirms that BRCTC enables more compact representations: the average length of the hidden sequence h is reduced to 63% of the original, saving 27% in inference cost.