[Paper Review] Scaling sparsemax based channel selection for speech recognition with ad-hoc microphone arrays

Junqi Chen, Xiao-Lei Zhang|arXiv (Cornell University)|Mar 28, 2021

Speech and Audio ProcessingComputer Science20 references7 citations

TL;DR

This paper proposes Scaling Sparsemax, a novel channel selection method for multichannel end-to-end speech recognition using large-scale ad-hoc microphone arrays. By replacing the Softmax operator in stream attention with Scaling Sparsemax, the model selectively suppresses only the most corrupted channels while preserving useful ones, achieving over 30% relative WER reduction compared to Softmax on simulated data and outperforming even the oracle one-best baseline on semi-real data.

ABSTRACT

Recently, speech recognition with ad-hoc microphone arrays has received much attention. It is known that channel selection is an important problem of ad-hoc microphone arrays, however, this topic seems far from explored in speech recognition yet, particularly with a large-scale ad-hoc microphone array. To address this problem, we propose a Scaling Sparsemax algorithm for the channel selection problem of the speech recognition with large-scale ad-hoc microphone arrays. Specifically, we first replace the conventional Softmax operator in the stream attention mechanism of a multichannel end-to-end speech recognition system with Sparsemax, which conducts channel selection by forcing the channel weights of noisy channels to zero. Because Sparsemax punishes the weights of many channels to zero harshly, we propose Scaling Sparsemax which punishes the channels mildly by setting the weights of very noisy channels to zero only. Experimental results with ad-hoc microphone arrays of over 30 channels under the conformer speech recognition architecture show that the proposed Scaling Sparsemax yields a word error rate of over 30% lower than Softmax on simulation data sets, and over 20% lower on semi-real data sets, in test scenarios with both matched and mismatched channel numbers.

Motivation & Objective

• To address the unexplored challenge of channel selection in large-scale ad-hoc microphone arrays for distant speech recognition.
• To improve automatic speech recognition (ASR) performance by directly optimizing recognition-level metrics rather than relying on signal-quality proxies like SNR.
• To develop a scalable, differentiable channel selection mechanism that can handle arrays with over 30 microphones.
• To train the system in a two-stage fashion: first with clean data for a single-channel conformer, then with multi-channel noisy data to learn channel selection via stream attention.
• To outperform conventional Softmax and existing channel selection baselines, including the oracle one-best method, in both simulated and semi-real environments.

Proposed method

• Replace the Softmax operator in the stream attention mechanism with Sparsemax to enable channel selection by forcing weights of noisy channels to zero.
• Propose Scaling Sparsemax, a differentiable, mild channel pruning method that only sets weights of the most corrupted channels to zero, avoiding over-penalization.
• Design a two-stage training strategy: pre-train a single-channel conformer on clean Librispeech data, then fine-tune the stream attention with multi-channel noisy data from ad-hoc arrays.
• Use a conformer-based ASR architecture with multi-head attention in both encoder and decoder, and integrate the stream attention module to reweight and fuse features from multiple channels.
• Train the stream attention module using multi-channel noisy data to learn optimal channel weights, with the guide vector from the decoder's hidden states used as query input.
• Apply SpecAugment for data augmentation and use greedy decoding without language models for inference.

Experimental results

Research questions

• RQ1 Can a differentiable, attention-based channel selection mechanism improve ASR performance in large-scale ad-hoc microphone arrays?
• RQ2 Does replacing Softmax with Sparsemax or Scaling Sparsemax in stream attention lead to better WER than conventional Softmax or oracle one-best selection?
• RQ3 How does the proposed method perform in mismatched channel scenarios (e.g., 30-channel test with 16-channel training)?
• RQ4 Can Scaling Sparsemax outperform the oracle one-best baseline, which assumes perfect knowledge of the closest microphone?
• RQ5 Does the two-stage training strategy—pre-training on clean data then fine-tuning on noisy multi-channel data—improve model convergence and performance?

Key findings

• Scaling Sparsemax achieved a 33.90% relative WER reduction over Softmax on the 'test-clean' set with 30-channel test data in the simulated Libri-adhoc-simu dataset.
• On the semi-real Libri-adhoc40 dataset, Scaling Sparsemax reduced WER by 17.4% relative to the oracle one-best baseline in the 20-channel test scenario.
• Scaling Sparsemax also achieved a 14.2% relative WER reduction over the oracle baseline in the 30-channel mismatched test scenario on Libri-adhoc40.
• The model outperformed Softmax by over 30% relative WER reduction in the 30-channel simulation test, demonstrating the effectiveness of channel selection in large arrays.
• Visualization confirmed that Softmax only reweights channels, Sparsemax over-penalizes many channels, while Scaling Sparsemax selectively suppresses only the most corrupted ones, leading to optimal performance.
• The two-stage training strategy successfully avoided training failure when including very noisy channels, and improved generalization across mismatched channel counts.