[Paper Review] Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals

Jing Shi, Xuankai Chang+6 more|arXiv (Cornell University)|Jun 24, 2020

Fault Detection and Control Systems21 citations

TL;DR

This paper proposes a conditional chain (CondChain) model that extends standard sequence-to-sequence learning to one-to-many sequence transduction by modeling dependencies between multiple output sequences using the probabilistic chain rule. By conditioning each output sequence on the input and previously generated sequences, the model enables flexible, variable-length output generation and achieves consistent improvements in speech separation and multi-speaker ASR over non-conditional baselines with minimal computational overhead.

ABSTRACT

Neural sequence-to-sequence models are well established for applications which can be cast as mapping a single input sequence into a single output sequence. In this work, we focus on one-to-many sequence transduction problems, such as extracting multiple sequential sources from a mixture sequence. We extend the standard sequence-to-sequence model to a conditional multi-sequence model, which explicitly models the relevance between multiple output sequences with the probabilistic chain rule. Based on this extension, our model can conditionally infer output sequences one-by-one by making use of both input and previously-estimated contextual output sequences. This model additionally has a simple and efficient stop criterion for the end of the transduction, making it able to infer the variable number of output sequences. We take speech data as a primary test field to evaluate our methods since the observed speech data is often composed of multiple sources due to the nature of the superposition principle of sound waves. Experiments on several different tasks including speech separation and multi-speaker speech recognition show that our conditional multi-sequence models lead to consistent improvements over the conventional non-conditional models.

Motivation & Objective

• To address the limitations of existing one-to-many sequence transduction methods in handling mixture signals, particularly in speech processing.
• To model the inherent relationships between multiple output sequences in a way that both serial and parallel mapping approaches fail to capture effectively.
• To enable variable-length output generation with a simple, effective stop criterion for sequence transduction tasks.
• To improve performance in speech separation and multi-speaker ASR by explicitly conditioning each output on prior outputs and the input.
• To provide a unified, generalizable framework applicable to diverse sequence-to-multi-sequence tasks beyond speech.

Proposed method

• The model extends standard seq2seq with attention by using the probabilistic chain rule to factorize the joint distribution of multiple output sequences.
• Each output sequence is generated sequentially, conditioned on the input sequence and all previously generated output sequences.
• The framework supports variable-length output generation through a learnable stop criterion that determines when to terminate the sequence generation process.
• The method is instantiated using TasNet and CTC-aligned supervision for speech separation and ASR, with joint training on waveform and alignment signals.
• Iterative refinement is modeled by chaining multiple generations, where each step conditions on the previous output, mimicking iterative signal re-estimation.
• The model is trained end-to-end with shared architecture and hyperparameters across steps, enabling efficient parameter sharing and adaptation.

Experimental results

Research questions

• RQ1 Can a unified sequence-to-multi-sequence model effectively model dependencies between multiple output sequences in mixture signal tasks?
• RQ2 Does conditioning each output sequence on prior outputs and the input lead to improved performance in speech separation and multi-speaker ASR?
• RQ3 Can the model generalize to tasks with positive correlation between outputs, such as iterative speech denoising?
• RQ4 How does the inclusion of semantic supervision (e.g., CTC alignment) affect the performance of the conditional chain model?
• RQ5 Does the conditional chain approach outperform both serial and parallel mapping paradigms in terms of accuracy and flexibility?

Key findings

• The conditional chain model achieved a 0.3 dB improvement in SI-SNRi when using CTC alignment as a semantic condition, demonstrating the benefit of structured supervision.
• Joint finetuning with both waveform and CTC alignment conditions reduced WER to 14.4%, a significant improvement over the 15.3% WER achieved with waveform-only conditioning.
• In iterative speech denoising, the second step of the conditional chain model achieved SDR of 18.0 dB and SDRi of 8.9 dB, outperforming the first step and the baseline TasNet.
• The model achieved consistent improvements in both speech separation and multi-speaker ASR tasks without a significant increase in model size.
• The results show that SI-SNRi is not always a reliable proxy for ASR performance, highlighting the importance of task-specific evaluation metrics.
• The model's ability to refine outputs through iterative conditioning demonstrates strong adaptability and generalization in modeling sequential dependencies.