[Paper Review] Learning not to Discriminate: Task Agnostic Learning for Improving Monolingual and Code-switched Speech Recognition

Gurunath Reddy Madhumani, Sanket Shah+3 more|arXiv (Cornell University)|Jun 9, 2020

Speech Recognition and Synthesis13 references5 citations

TL;DR

This paper proposes a task-agnostic learning approach using domain-adversarial training to improve both monolingual and code-switched speech recognition simultaneously. By training shared layers to be invariant to task identity (monolingual vs. code-switched) through an adversarial discriminator, the model learns shared features that generalize across both tasks, reducing WER by up to 3.1% on monolingual and 2.2% on code-switched test sets across three language pairs.

ABSTRACT

Recognizing code-switched speech is challenging for Automatic Speech Recognition (ASR) for a variety of reasons, including the lack of code-switched training data. Recently, we showed that monolingual ASR systems fine-tuned on code-switched data deteriorate in performance on monolingual speech recognition, which is not desirable as ASR systems deployed in multilingual scenarios should recognize both monolingual and code-switched speech with high accuracy. Our experiments indicated that this loss in performance could be mitigated by using certain strategies for fine-tuning and regularization, leading to improvements in both monolingual and code-switched ASR. In this work, we present further improvements over our previous work by using domain adversarial learning to train task agnostic models. We evaluate the classification accuracy of an adversarial discriminator and show that it can learn shared layer parameters that are task agnostic. We train end-to-end ASR systems starting with a pooled model that uses monolingual and code-switched data along with the adversarial discriminator. Our proposed technique leads to reductions in Word Error Rates (WER) in monolingual and code-switched test sets across three language pairs.

Motivation & Objective

• To address the degradation in monolingual ASR performance when fine-tuning on code-switched data.
• To learn shared acoustic representations that are invariant to task identity (monolingual vs. code-switched) without sacrificing task-specific discriminability.
• To improve joint performance on both monolingual and code-switched speech recognition using adversarial training.
• To validate that task-agnostic shared layers lead to better generalization than standard fine-tuning or regularization.
• To explore the integration of task-specific layers with adversarial shared layers for enhanced performance.

Proposed method

• Train an end-to-end ASR model using pooled monolingual and code-switched data with shared layers and task-specific output heads.
• Introduce a gradient reversal layer (GRL) in an adversarial discriminator to minimize classification accuracy of the shared layer features for monolingual vs. code-switched identity.
• Optimize the model using a joint loss: monolingual ASR loss, code-switched ASR loss, and adversarial loss to encourage task-agnostic shared features.
• Use a multi-task adversarial discriminator that classifies utterances as monolingual or code-switched based on shared layer features.
• Update model parameters via stochastic gradient descent with separate loss components for monolingual, code-switched, and adversarial objectives.
• Evaluate performance using Word Error Rate (WER) on monolingual and code-switched test sets across three language pairs.

Experimental results

Research questions

• RQ1 Can adversarial training be used to learn task-agnostic shared representations that improve performance on both monolingual and code-switched speech recognition?
• RQ2 Does the inclusion of a task-specific layer after the shared layers further improve performance beyond adversarial training alone?
• RQ3 Can the shared layers learn invariant features when trained with an adversarial discriminator, as opposed to a standard classifier?
• RQ4 How does the proposed method compare to prior fine-tuning and regularization strategies in terms of WER on both monolingual and code-switched test sets?
• RQ5 What is the impact of task-agnostic shared layers on model generalization in multilingual, code-switching scenarios?

Key findings

• The multi-task adversarial model (Exp6) achieved the lowest WER across all test sets, outperforming the pooled model (Exp3) by 3.1% on Tamil-English monolingual and 2.2% on Tamil-English code-switched.
• The adversarial task-agnostic pooled model (Exp5) reduced WER by 1.21% on Tamil-English monolingual and 1.28% on Tamil-English code-switched compared to the best fine-tuned model (Exp4).
• Classification experiments showed that the validation accuracy of the adversarial discriminator remained constant (~50%), indicating that shared layers learned task-invariant features, whereas the vanilla classifier's accuracy increased, indicating task-specific feature learning.
• The model with task-specific layers and adversarial training achieved the best performance on all three language pairs: Tamil-English, Telugu-English, and Gujarati-English.
• The WER reduction was consistent across all language pairs, with the largest absolute improvement seen in Tamil-English (3.1% on monolingual, 2.2% on code-switched).
• The results confirm that task-agnostic shared layers are essential for maintaining performance on monolingual speech while improving code-switched recognition.