[Paper Review] An End-to-end Architecture of Online Multi-channel Speech Separation

Jian Wu, Zhuo Chen+6 more|arXiv (Cornell University)|Sep 7, 2020

Speech and Audio Processing27 references4 citations

TL;DR

This paper proposes an end-to-end online multi-channel speech separation system, E2E-UFE, which integrates speech unmixing, beam selection via an attentional module, and speech extraction into a jointly trainable architecture. By enabling gradient flow through the beam selection module using an attention mechanism, E2E-UFE achieves comparable offline performance to the original modular UFE system and a 12.47% relative WER reduction in block online evaluation, demonstrating improved robustness in real-time settings.

ABSTRACT

Multi-speaker speech recognition has been one of the keychallenges in conversation transcription as it breaks the singleactive speaker assumption employed by most state-of-the-artspeech recognition systems. Speech separation is consideredas a remedy to this problem. Previously, we introduced a sys-tem, calledunmixing,fixed-beamformerandextraction(UFE),that was shown to be effective in addressing the speech over-lap problem in conversation transcription. With UFE, an inputmixed signal is processed by fixed beamformers, followed by aneural network post filtering. Although promising results wereobtained, the system contains multiple individually developedmodules, leading potentially sub-optimum performance. In thiswork, we introduce an end-to-end modeling version of UFE. Toenable gradient propagation all the way, an attentional selectionmodule is proposed, where an attentional weight is learnt foreach beamformer and spatial feature sampled over space. Ex-perimental results show that the proposed system achieves com-parable performance in an offline evaluation with the originalseparate processing-based pipeline, while producing remark-able improvements in an online evaluation.

Motivation & Objective

• To address the limitations of modular, separately optimized speech separation systems like UFE, which suffer from suboptimal performance due to indirect training objectives.
• To enable end-to-end training of multi-channel speech separation by making the beam selection module differentiable through a novel attentional selection mechanism.
• To improve robustness and performance in online, real-time speech separation scenarios, particularly under block-processing constraints.
• To achieve joint optimization of speech unmixing and extraction networks while maintaining low latency for practical deployment in conversation transcription.

Proposed method

• Introduces an end-to-end architecture, E2E-UFE, that replaces the separate sound source localization (SSL) module in UFE with a learnable attentional selection module to enable gradient backpropagation through beam selection.
• Employs a differentiable attention mechanism that computes spatial attention weights over fixed beamformers based on spatial features sampled over frequency and time.
• Uses permutation invariant training (PIT) with scaled-invariant signal-to-noise ratio (Si-SNR) as the objective function for the speech unmixing network to improve separation quality.
• Applies a joint training strategy with pre-training of unmixing and extraction networks followed by end-to-end fine-tuning using a reduced initial learning rate.
• Utilizes short-time Fourier transform (STFT) features with log magnitude spectra and inter-microphone phase differences (cosIPDs) as input features for the neural networks.
• Employs a double-buffering scheme for block online processing, with 2s or 4s look-back context, to simulate real-time constraints.

Experimental results

Research questions

• RQ1 Can an end-to-end trainable architecture be designed for online multi-channel speech separation that enables gradient flow through the beam selection process?
• RQ2 How does joint optimization of unmixing and extraction networks via end-to-end training improve performance compared to modular, separately trained systems?
• RQ3 What is the performance gap between offline and online evaluation in real-time speech separation, and can E2E-UFE close it?
• RQ4 Does end-to-end training make the system more robust to incorrect beam selections during inference compared to the original UFE?

Key findings

• E2E-UFE achieves comparable word error rate (WER) performance to the original UFE system in offline evaluation on both simulated and semi-real datasets.
• On the semi-real dataset, E2E-UFE achieves 4.8% and 4.3% relative WER reduction over UFE on OV35 and OV75, respectively, reaching 33.89% and 35.92% WER.
• In block online evaluation, E2E-UFE achieves a 29.71% relative WER reduction on the simulated dataset and a 12.47% average relative WER reduction on the semi-real dataset compared to the original UFE.
• The original UFE system degrades significantly in online settings, with WER increasing from 16.44% to 24.10% on simulated data and from 35.60% to 44.05% on semi-real data, indicating poor robustness to latency constraints.
• E2E-UFE's robustness is attributed to joint optimization that exposes the model to incorrect beam selections during training, unlike UFE which only sees correct beams.
• The absence of the sparsification trick in both systems may have exacerbated energy leakage in UFE, while E2E-UFE's joint training mitigates this issue through end-to-end optimization.