[Paper Review] Directional ASR: A New Paradigm for E2E Multi-Speaker Speech Recognition with Source Localization

Aswin Shanmugam Subramanian, Chao Weng+5 more|arXiv (Cornell University)|Oct 30, 2020

Speech and Audio Processing31 references4 citations

TL;DR

This paper introduces Directional ASR (D-ASR), a novel end-to-end framework for multi-speaker far-field speech recognition that jointly optimizes source localization, speech separation, and automatic speech recognition (ASR) using only ASR loss. By modeling the azimuth angle of speakers as a latent variable and using it to generate time-frequency masks, D-ASR achieves state-of-the-art performance with less than 3° average DOA error and outperforms MIMO-Speech in both separation quality and ASR accuracy.

ABSTRACT

This paper proposes a new paradigm for handling far-field multi-speaker data in an end-to-end neural network manner, called directional automatic speech recognition (D-ASR), which explicitly models source speaker locations. In D-ASR, the azimuth angle of the sources with respect to the microphone array is defined as a latent variable. This angle controls the quality of separation, which in turn determines the ASR performance. All three functionalities of D-ASR: localization, separation, and recognition are connected as a single differentiable neural network and trained solely based on ASR error minimization objectives. The advantages of D-ASR over existing methods are threefold: (1) it provides explicit speaker locations, (2) it improves the explainability factor, and (3) it achieves better ASR performance as the process is more streamlined. In addition, D-ASR does not require explicit direction of arrival (DOA) supervision like existing data-driven localization models, which makes it more appropriate for realistic data. For the case of two source mixtures, D-ASR achieves an average DOA prediction error of less than three degrees. It also outperforms a strong far-field multi-speaker end-to-end system in both separation quality and ASR performance.

Motivation & Objective

• To develop a unified end-to-end framework that jointly optimizes source localization, speech separation, and automatic speech recognition (ASR) without requiring explicit DOA annotations.
• To improve the explainability and interpretability of multi-speaker ASR systems by explicitly modeling speaker direction as a learnable latent variable.
• To enhance ASR and separation performance in far-field multi-speaker scenarios by leveraging directional information derived from the ASR objective alone.
• To enable robust DOA estimation from only text transcriptions, avoiding the need for parallel ground-truth direction-of-arrival labels in real-world data.

Proposed method

• D-ASR models the azimuth angle of each speaker as a latent variable, which controls the generation of steering vectors and time-frequency (T-F) masks for separation.
• The localization subnetwork uses a CNN to extract phase features from multichannel STFT inputs, followed by a recurrent layer and temporal averaging to produce angle posteriors.
• Angle posteriors are used to compute beamforming weights via a low-complexity mask (LCMP) or MVDR beamformer, which are differentiable and end-to-end trainable.
• The system is trained solely on ASR loss, with no explicit supervision for DOA or separation, enabling joint optimization of all three tasks.
• During inference, the beamformer is replaced with a mask-based MVDR-REF beamformer for improved performance, while DOA predictions remain interpretable.
• A uniform regularization loss is applied to encourage balanced angle posterior distributions, improving generalization and DOA estimation accuracy.

Experimental results

Research questions

• RQ1 Can source localization be effectively learned in an end-to-end manner using only ASR transcriptions as supervision?
• RQ2 How does explicit modeling of speaker direction improve the performance and interpretability of multi-speaker ASR systems?
• RQ3 What is the impact of different beamforming strategies (LCMP vs. MVDR) on DOA estimation and separation quality in the absence of ground-truth DOA?
• RQ4 Can a simplified front-end that predicts only DOA angles achieve better separation and ASR performance than direct T-F mask estimation?
• RQ5 How does the proposed method compare to traditional signal processing (e.g., MUSIC, TOPS) and state-of-the-art end-to-end models like MIMO-Speech in DOA estimation and ASR accuracy?

Key findings

• D-ASR achieves an average DOA prediction error of just 3.0° on the test set when using LCMP beamforming with regularization, significantly outperforming MUSIC (14.4°) and TOPS (10.1°).
• The system reduces WER to 4.1% on the test set, outperforming MIMO-Speech (5.1% WER) and approaching the oracle IBM baseline (3.0% WER).
• SDR scores reach 15.2 dB, nearly matching the oracle IBM (15.7 dB), and surpassing MIMO-Speech (11.7 dB), demonstrating superior separation quality.
• The use of uniform regularization in the localization subnetwork improves DOA estimation accuracy, with the best-performing setup (γ=10) achieving 2.5° average error on the test set.
• The D-ASR system with LCMP beamforming and regularization achieves 3.0° average L1 error, while the MVDR-REF variant maintains strong performance with 7.4 dB SDR and 2.9 PESQ.
• The intermediate outputs (DOA and masks) are interpretable, enabling analysis of the model’s behavior and providing insights into speaker separation dynamics.