[Paper Review] Multi-Geometry Spatial Acoustic Modeling for Distant Speech Recognition

Kenichi Kumatani, Minhua Wu+3 more|arXiv (Cornell University)|Mar 13, 2019

Speech and Audio Processing30 references4 citations

TL;DR

This paper proposes a multi-geometry spatial acoustic modeling framework that unifies beamforming and LSTM-based acoustic modeling for distant speech recognition. By initializing a spatial filtering layer with beamformer weights for multiple array geometries and jointly optimizing it with an LSTM, the model achieves robust performance across mismatched microphone placements, reducing word error rates by 13.4% (matched) and 12.7% (mismatched) compared to single-channel systems, outperforming even 7-microphone conventional beamforming.

ABSTRACT

The use of spatial information with multiple microphones can improve far-field automatic speech recognition (ASR) accuracy. However, conventional microphone array techniques degrade speech enhancement performance when there is an array geometry mismatch between design and test conditions. Moreover, such speech enhancement techniques do not always yield ASR accuracy improvement due to the difference between speech enhancement and ASR optimization objectives. In this work, we propose to unify an acoustic model framework by optimizing spatial filtering and long short-term memory (LSTM) layers from multi-channel (MC) input. Our acoustic model subsumes beamformers with multiple types of array geometry. In contrast to deep clustering methods that treat a neural network as a black box tool, the network encoding the spatial filters can process streaming audio data in real time without the accumulation of target signal statistics. We demonstrate the effectiveness of such MC neural networks through ASR experiments on the real-world far-field data. We show that our two-channel acoustic model can on average reduce word error rates (WERs) by~13.4 and~12.7% compared to a single channel ASR system with the log-mel filter bank energy (LFBE) feature under the matched and mismatched microphone placement conditions, respectively. Our result also shows that our two-channel network achieves a relative WER reduction of over~7.0% compared to conventional beamforming with seven microphones overall.

Motivation & Objective

• Address the degradation in far-field ASR performance caused by microphone array geometry mismatch between training and test conditions.
• Unify spatial filtering and acoustic modeling into a single end-to-end trainable framework to improve robustness against array configuration variations.
• Enable real-time inference without signal statistic accumulation, unlike batch processing methods such as DNN mask-based beamforming.
• Demonstrate that multi-channel DNNs can subsume multiple beamformer configurations and outperform conventional beamforming, even with fewer microphones.
• Achieve superior ASR accuracy across diverse real-world array geometries by training on multiple microphone configurations simultaneously.

Proposed method

• Propose two multi-channel (MC) neural network architectures: one using fully connected fusion of spatial filter outputs, and another using frequency-tied weights with maximum energy selection.
• Initialize the spatial filtering (SF) layer with beamformer weights designed for multiple array geometries and look directions, enabling the network to subsume various beamformer types.
• Perform all processing in the frequency domain for computational efficiency, using complex-valued operations to model wave propagation and directionality.
• Optimize the entire model end-to-end using the ASR criterion in a stage-wise manner, jointly training spatial filters and LSTM layers.
• Use a shared feature extraction and classification path for both single- and multi-channel inputs, with the MC model initialized from a pre-trained LFBE baseline.
• Apply weight tying across frequency bins in the WTSF architecture to reduce parameters and improve generalization across array configurations.

Experimental results

Research questions

• RQ1 Can a unified deep neural network architecture effectively model multiple microphone array geometries without requiring precise calibration?
• RQ2 Does joint optimization of spatial filtering and acoustic modeling improve ASR robustness under array geometry mismatch?
• RQ3 Can a learnable spatial filtering layer outperform conventional beamforming in terms of word error rate, even with fewer microphones?
• RQ4 How does multi-geometry training affect performance when the test array configuration differs from training configurations?
• RQ5 Can the proposed model process streaming audio in real time without accumulating signal statistics?

Key findings

• The two-channel multi-geometry model reduces word error rate (WER) by 13.4% compared to a single-channel LFBE baseline under matched microphone geometry conditions.
• Under mismatched microphone placements, the same model achieves a 12.7% relative WER reduction, demonstrating robustness to array geometry variation.
• The model outperforms conventional 7-microphone beamforming by over 7.0% relative WER reduction, even with only two microphones.
• The WTSF architecture, which uses tied weights and max-pooling, achieves slightly better performance than the ESF with fully connected fusion, especially in mismatched conditions.
• The multi-geometry model maintains consistent performance across diverse array configurations, while a single-geometry model degrades significantly as test geometry diverges from training.
• The model enables real-time inference without requiring bi-directional processing or signal statistic accumulation, unlike DNN mask-based beamforming.