[Paper Review] Improving Code-switching Language Modeling with Artificially Generated Texts using Cycle-consistent Adversarial Networks

Chia-Yu Li, Ngoc Thang Vu|arXiv (Cornell University)|Dec 12, 2021

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a CycleGAN-based framework that treats code-switching as a speaking style to generate artificial code-switched text from monolingual sentences, significantly improving language modeling and automatic speech recognition (ASR) performance on the SEAME corpus. By leveraging cycle-consistent adversarial training without parallel data, the model generates more natural, linguistically consistent code-switched text that mimics real speech patterns, including stuttering and mixed-language distributions.

ABSTRACT

This paper presents our latest effort on improving Code-switching language models that suffer from data scarcity. We investigate methods to augment Code-switching training text data by artificially generating them. Concretely, we propose a cycle-consistent adversarial networks based framework to transfer monolingual text into Code-switching text, considering Code-switching as a speaking style. Our experimental results on the SEAME corpus show that utilising artificially generated Code-switching text data improves consistently the language model as well as the automatic speech recognition performance.

Motivation & Objective

• To address data scarcity in code-switching language modeling by generating synthetic training data.
• To improve automatic speech recognition (ASR) performance for code-switched speech using data augmentation.
• To model code-switching as a speaking style rather than a translation task, enabling end-to-end learning.
• To generate more natural and linguistically consistent code-switched text that mirrors real speech patterns, including fluency and prosody.
• To evaluate the impact of cycle-consistent adversarial training on both language modeling and ASR performance.

Proposed method

• The framework uses a sequence-to-sequence (S2S) model to pre-train monolingual-to-code-switched translation mappings.
• CycleGAN is applied to transfer monolingual text into code-switched text using cycle-consistent adversarial training, without requiring parallel monolingual-code-switched pairs.
• The model employs two generators (G: monolingual → code-switched, F: code-switched → monolingual) and two discriminators to enforce distributional similarity and cycle consistency.
• The training objective combines adversarial loss, cycle-consistency loss, and identity loss to preserve semantic meaning and improve generation quality.
• Monolingual data is leveraged to pre-train the S2S model, which is then fine-tuned within the CycleGAN framework to improve style transfer.
• The framework is evaluated on the SEAME corpus, comparing generated text distributions and ASR performance against baselines.

Figure 1: Network architecture of CycleGAN for CS text generation. X refers to monolingual text, e.g. Chinese and Y refers to CS text, e.g. Chinese English CS sentences.

Experimental results

Research questions

• RQ1 Can code-switching be effectively modeled as a speaking style to generate synthetic training data?
• RQ2 Does CycleGAN-based text generation improve language modeling perplexity compared to baseline and S2S methods?
• RQ3 Can artificially generated code-switched text enhance automatic speech recognition (ASR) performance on real-world data?
• RQ4 How well does the generated text match the linguistic distribution (e.g., code-mixing level) of real code-switched speech?
• RQ5 To what extent does the model simulate natural speech behaviors such as stuttering or repetition?

Key findings

• The CycleGAN-based model achieved a 21.9% word error rate (WER) on the ASR eval set, a 0.5% improvement over the baseline (22.4%) and S2S (22.1%), demonstrating consistent ASR gains.
• Language modeling perplexity was reduced to 10.39 with CycleGAN-generated data, outperforming the baseline and S2S models across all cycle loss weights.
• The distribution of code-mixing intensity (CMI) in CycleGAN-generated text closely matched the real SEAME corpus, especially in the (30,45]% and (45,50]% CMI ranges.
• CycleGAN-generated text exhibited natural speech behaviors such as repetition (e.g., 'take take') and semantic shifts (e.g., replacing 'logistics' with '2016'), suggesting mimicry of real speaking styles.
• The model successfully generated meaningful code-switched sentences that preserved semantic content while introducing natural code-switching patterns, unlike simple dictionary-based replacement.
• The ablation study showed that cycle-consistency loss was critical: optimal performance was achieved at λ₁ = 0.9, with a 10.77% MER, indicating that cycle loss stabilizes the generation process.