[Paper Review] FCN Approach for Dynamically Locating Multiple Speakers

Hodaya Hammer, Shlomo E. Chazan+2 more|arXiv (Cornell University)|Aug 26, 2020

Speech and Audio Processing27 references4 citations

TL;DR

This paper proposes a fully convolutional network (FCN)-based approach for high-resolution, real-time localization of multiple speakers using time-frequency (TF) bin-wise direction-of-arrival (DOA) estimation. By leveraging instantaneous spatial features and exploiting the W-disjoint orthogonality principle, the model achieves superior accuracy and smooth tracking in both static and dynamic scenarios, outperforming state-of-the-art methods in simulated and real-world reverberant environments with up to 720 ms RT60.

ABSTRACT

In this paper, we present a deep neural network-based online multi-speaker localisation algorithm. Following the W-disjoint orthogonality principle in the spectral domain, each time-frequency (TF) bin is dominated by a single speaker, and hence by a single direction of arrival (DOA). A fully convolutional network is trained with instantaneous spatial features to estimate the DOA for each TF bin. The high resolution classification enables the network to accurately and simultaneously localize and track multiple speakers, both static and dynamic. Elaborated experimental study using both simulated and real-life recordings in static and dynamic scenarios, confirms that the proposed algorithm outperforms both classic and recent deep-learning-based algorithms.

Motivation & Objective

• To address the challenge of accurate, real-time localization of multiple moving speakers in highly reverberant acoustic environments.
• To overcome the limitations of frame-based DOA estimation methods that lack temporal resolution and tracking capability.
• To leverage the W-disjoint orthogonality property, where each time-frequency bin is dominated by a single speaker, to enable high-resolution DOA estimation.
• To develop a data-driven, deep learning model robust to varying reverberation and noise conditions, generalizing beyond training environments.
• To establish a publicly available benchmark dataset for multi-speaker DOA estimation using simulated and real recordings.

Proposed method

• A fully convolutional network (FCN) is trained to predict DOA for each time-frequency (TF) bin using instantaneous spatial features derived from microphone signals.
• The spatial features are computed as the real and imaginary parts of the relative transfer function (RTF) in the spectral domain.
• The model is trained with TF-resolution supervision, assigning a single DOA label per TF bin based on the W-disjoint orthogonality principle.
• The network architecture enables end-to-end, online inference with high temporal resolution, supporting simultaneous localization and tracking of multiple speakers.
• Training data is synthesized using real room impulse responses (RIRs) from public datasets, simulating diverse acoustic conditions and speaker dynamics.
• The method is evaluated on both simulated data with measured RIRs and real-life recordings in controlled environments with ground truth from a 3D tracking system.

Experimental results

Research questions

• RQ1 Can a deep learning model trained at the time-frequency bin level achieve superior DOA estimation accuracy compared to frame-level methods in highly reverberant environments?
• RQ2 Does high-resolution TF supervision enable accurate and stable tracking of multiple moving speakers, even under strong reverberation?
• RQ3 How does the proposed FCN-based approach compare to classic methods (e.g., MUSIC, SRP-PHAT) and recent deep learning baselines (e.g., CMS-DOA) in terms of MAE and accuracy across varying RT60 conditions?
• RQ4 What is the impact of different spatial feature representations (e.g., RTF real/imaginary parts vs. cosine/sine of phase) on model performance?
• RQ5 Can the model generalize to unseen rooms and reverberation conditions without fine-tuning, given sufficient data diversity during training?

Key findings

• The proposed TF-DOAnet achieved a mean absolute error (MAE) of only 0.3° in a simulated room with RT60 = 390 ms, compared to 13.1° for the CMS-DOA baseline.
• In a high-reverberation room (RT60 = 720 ms), TF-DOAnet achieved 94.3% accuracy, significantly outperforming CMS-DOA (38.1%) and classic methods (e.g., MUSIC at 16.9%).
• On real-life recordings at 1 m distance with RT60 = 0.61 s, TF-DOAnet achieved 98.3% accuracy, compared to 71.9% for CMS-DOA.
• The ablation study showed that the proposed RTF-based features (real/imaginary parts) outperformed both the cosine-sine features and the addition of spectral features, which slightly degraded performance.
• The model produced smooth, noiseless DOA trajectories even in high-reverberation conditions, unlike the fluctuating outputs of CMS-DOA.
• The method demonstrated robustness across diverse acoustic conditions, including static and dynamic speaker scenarios, with consistent high performance across all test conditions.