[Paper Review] Why does CTC result in peaky behavior?

Albert Zeyer, Ralf Schlüter+1 more|arXiv (Cornell University)|May 31, 2021

Epilepsy research and treatmentMedicine29 references18 citations

TL;DR

This paper provides a formal analysis of why Connectionist Temporal Classification (CTC) loss leads to peaky behavior—where models concentrate predictions on a single label (e.g., blank) over time—due to gradient descent dynamics from uniform initialization. It demonstrates that CTC training can converge to suboptimal, peaky solutions with 100% error rate on simple tasks, and shows that incorporating a label prior model eliminates peaky behavior and enables convergence to optimal, non-peaky solutions with zero error.

ABSTRACT

The peaky behavior of CTC models is well known experimentally. However, an understanding about why peaky behavior occurs is missing, and whether this is a good property. We provide a formal analysis of the peaky behavior and gradient descent convergence properties of the CTC loss and related training criteria. Our analysis provides a deep understanding why peaky behavior occurs and when it is suboptimal. On a simple example which should be trivial to learn for any model, we prove that a feed-forward neural network trained with CTC from uniform initialization converges towards peaky behavior with a 100% error rate. Our analysis further explains why CTC only works well together with the blank label. We further demonstrate that peaky behavior does not occur on other related losses including a label prior model, and that this improves convergence.

Motivation & Objective

• To formally explain the root cause of peaky behavior in CTC-trained models, which has been observed experimentally but not theoretically understood.
• To investigate why gradient descent from uniform initialization leads to suboptimal local optima with peaky output distributions in CTC.
• To clarify the essential role of the blank label in preventing divergence and enabling convergence in CTC.
• To demonstrate that peaky behavior is avoidable by modifying the training criterion, particularly through inclusion of a label prior model.
• To compare CTC with alternative training criteria and show that only CTC exhibits peaky behavior due to its specific loss structure and label topology.

Proposed method

• Formal mathematical analysis of the CTC loss function and its gradient dynamics under uniform initialization, using symbolic computation with SymPy.
• Construction of a minimal synthetic example (e.g., B*a+ B*) to isolate and demonstrate peaky convergence in a feedforward neural network.
• Derivation of the gradient of the CTC loss with respect to model parameters, showing how it favors concentration on the blank label.
• Introduction and analysis of a modified training criterion that incorporates a label prior, formalized as a generative model with marginalized alignments.
• Empirical validation using TensorFlow and RETURNN on synthetic datasets to compare convergence behavior between standard CTC and prior-augmented CTC.
• Reparameterization of the model using a softplus or sigmoid-based parameterization to analyze gradient flow and convergence paths.

Experimental results

Research questions

• RQ1 Why does CTC training with gradient descent from uniform initialization lead to peaky behavior, even on trivial sequence learning tasks?
• RQ2 What role does the blank label play in preventing or enabling convergence to peaky solutions in CTC?
• RQ3 Why does the inclusion of a label prior model eliminate peaky behavior and improve convergence in sequence modeling?
• RQ4 How do the local convergence properties of CTC differ from those of other sequence training criteria, such as label prior models or full-sum marginalization?
• RQ5 Under what conditions does peaky behavior become suboptimal, and can it be formally avoided through loss modification?

Key findings

• A feedforward neural network trained with CTC from uniform initialization converges to a peaky solution with 100% error rate on a simple, trivial sequence task (e.g., B*a+ B*), proving that peaky behavior is a fundamental flaw of the CTC criterion.
• The peaky behavior arises from the label topology and the dominance of the blank label in the alignment space, which biases gradient descent toward concentrating probability mass on blank at the expense of other labels.
• The blank label is crucial for convergence in CTC; without it, peaky behavior becomes even more severe and convergence to meaningful alignments fails.
• A label prior model, when incorporated into the training criterion, eliminates peaky behavior and enables convergence to the global optimum with zero error rate, as demonstrated in synthetic experiments.
• The generative model with full-sum marginalization over alignments (without CTC’s specific structure) converges to a non-peaky, optimal solution from uniform initialization, proving that peaky behavior is specific to CTC’s loss formulation.
• The analysis shows that local-context models like FFNNs are suboptimal for CTC due to their inability to escape peaky local optima, highlighting architectural limitations under CTC training.