[Paper Review] On the Inductive Bias of Word-Character-Level Multi-Task Learning for Speech Recognition

Jan Kremer, Lasse Borgholt+1 more|arXiv (Cornell University)|Nov 28, 2018

Speech Recognition and Synthesis30 references4 citations

TL;DR

This paper proposes a multi-task learning (MTL) framework that jointly trains on word-level and character-level automatic speech recognition (ASR) using connectionist temporal classification (CTC). By combining word-level supervision with character-level regularization, the model achieves a 23% word error rate (WER), significantly outperforming both single-task baselines and eliminating unknown-token predictions. The key contribution is empirically characterizing the inductive bias of MTL as an interpolation between word frequency and word length preferences.

ABSTRACT

End-to-end automatic speech recognition (ASR) commonly transcribes audio signals into sequences of characters while its performance is evaluated by measuring the word-error rate (WER). This suggests that predicting sequences of words directly may be helpful instead. However, training with word-level supervision can be more difficult due to the sparsity of examples per label class. In this paper we analyze an end-to-end ASR model that combines a word-and-character representation in a multi-task learning (MTL) framework. We show that it improves on the WER and study how the word-level model can benefit from character-level supervision by analyzing the learned inductive preference bias of each model component empirically. We find that by adding character-level supervision, the MTL model interpolates between recognizing more frequent words (preferred by the word-level model) and shorter words (preferred by the character-level model).

Motivation & Objective

• To investigate whether training a word-level ASR model from scratch is feasible on small datasets, contrary to prior negative findings.
• To analyze how multi-task learning with word- and character-level supervision shapes the inductive bias of the model.
• To understand the learning dynamics of word recognition in terms of frequency and length distribution during training.
• To determine whether combining word- and character-level supervision improves generalization beyond single-supervision baselines.

Proposed method

• The model uses connectionist temporal classification (CTC) with a shared encoder and separate word- and character-level output heads.
• Word-level outputs are defined over a vocabulary of common words, while character-level outputs use the standard alphabet including blank tokens.
• The total loss is a weighted sum of word-level and character-level CTC losses, enabling joint optimization.
• Model training is monitored by tracking the cumulative distribution of recognized words in the validation set over time.
• The inductive bias is analyzed by comparing the frequency and length rank distributions of words recognized early in training.
• A beam search decoder with lexicon constraints is used to evaluate character-level baselines, and combined decoding is tested for word-level performance.

Experimental results

Research questions

• RQ1 Can a word-level ASR model be successfully trained from scratch on a relatively small dataset, despite label sparsity?
• RQ2 How does multi-task learning with word- and character-level supervision influence the model's preference for recognizing certain types of words?
• RQ3 To what extent does the MTL model interpolate between the inductive biases of word-level and character-level models?
• RQ4 Does the combination of word- and character-level supervision lead to better generalization than either single-task approach?

Key findings

• The MTL model achieves a word error rate (WER) of 23%, which is 5 percentage points lower than the character-level baseline and outperforms the word-level-only model.
• Contrary to prior work, the study demonstrates that training a word-level ASR model from scratch is feasible even on a small dataset, achieving performance comparable to the character-level baseline.
• The word-level model alone learns the most frequent words first, showing a strong bias toward high-frequency words.
• The character-level model learns shorter words more easily and covers the full range of word lengths more uniformly from the start.
• The MTL model combines both biases, resulting in a more uniform distribution across word frequency and length ranks, especially benefiting rare and long words.
• The MTL model eliminates unknown-token predictions by ensuring all recognized words are in-vocabulary, improving transcript readability.