[Paper Review] Acoustic Characterization of Environments (ACE) Challenge Results Technical Report

James Eaton, Nikolay D. Gaubitch+2 more|arXiv (Cornell University)|Dec 17, 2015

Speech and Audio Processing15 references3 citations

TL;DR

This technical report evaluates acoustic parameter estimation algorithms for reverberation time (T60) and direct-to-reverberant ratio (DRR) using the ACE Challenge dataset, which features real room impulse responses corrupted by ambient, babble, and fan noise across varying SNRs. The key contribution is a comprehensive benchmarking of 26 algorithms, revealing that frequency-dependent T60 and DRR estimation remain challenging under low SNR and non-stationary noise, with DENBE and Particle Velocity showing robustness in high-frequency bands despite significant bias in low-frequency regions under fan noise at -1 dB SNR.

ABSTRACT

This document provides the results of the tests of acoustic parameter estimation algorithms on the Acoustic Characterization of Environments (ACE) Challenge Evaluation dataset which were subsequently submitted and written up into papers for the Proceedings of the ACE Challenge. This document is supporting material for a forthcoming journal paper on the ACE Challenge which will provide further analysis of the results.

Motivation & Objective

• To evaluate and compare the performance of 26 acoustic parameter estimation algorithms on a standardized dataset of real room impulse responses corrupted by ambient, babble, and fan noise.
• To assess the impact of varying signal-to-noise ratio (SNR) on T60 and DRR estimation accuracy across different noise types and frequency bands.
• To provide a comprehensive benchmark for fullband and frequency-dependent T60 and DRR estimation under realistic acoustic conditions, supporting future algorithm development and evaluation.
• To identify performance bottlenecks in low-SNR and non-stationary noise environments, particularly in low-frequency bands where estimation errors are most pronounced.
• To support the development of robust acoustic scene characterization systems by quantifying algorithmic performance across diverse acoustic scenarios and noise conditions.

Proposed method

• The ACE Challenge dataset comprises 128 room impulse responses recorded in 16 real rooms with varying dimensions, microphone positions, and talker configurations, using anechoic recordings and synthetic noise injection.
• Algorithms were evaluated on fullband and 26 critical-band frequency subbands (centered from 25.12 Hz to 7943.28 Hz) for T60 and DRR estimation using metrics including bias, mean squared error (MSE), and correlation coefficient (ρ).
• Noise types were systematically applied: ambient (stationary), babble (multi-talker speech), and fan (broadband tonal), with SNR levels set at 18 dB, 12 dB, and -1 dB to assess robustness.
• Estimation methods included time-domain energy decay, spectral decay analysis, and advanced techniques such as DENBE (Decoupled Energy Decay Estimation) with FFT- and filter-bank derived subbands, and particle velocity-based modeling.
• Performance was evaluated per room, noise type, SNR, and frequency band, with results aggregated across all test conditions to enable cross-algorithm comparison.
• Statistical analysis focused on bias, MSE, and correlation to assess accuracy and consistency, particularly in low-SNR and high-frequency regimes.

Experimental results

Research questions

• RQ1 How do different acoustic parameter estimation algorithms perform in estimating fullband and frequency-dependent T60 under ambient, babble, and fan noise at varying SNRs?
• RQ2 What is the impact of noise type and SNR on the accuracy of DRR estimation, particularly in low-frequency bands where reverberation dominates?
• RQ3 Which algorithmic approaches demonstrate the most robust performance across diverse acoustic environments and noise conditions, especially under low SNR?
• RQ4 How do frequency-dependent estimation errors vary across the audio spectrum, and are there consistent patterns in bias and MSE across noise types?
• RQ5 To what extent do subband decomposition techniques (e.g., FFT-derived vs. filter-bank subbands) improve estimation accuracy in noisy and reverberant conditions?

Key findings

• Under fan noise at -1 dB SNR, the DENBE algorithm with filtered subbands exhibited a mean bias of -10.14 ms in the highest frequency band (7943.28 Hz), with MSE of 155.3 ms², indicating significant error accumulation in high frequencies.
• For DRR estimation under fan noise at -1 dB SNR, the Particle Velocity algorithm showed a mean bias of -1.244 dB in the 1995.26 Hz band, with MSE of 10.04 dB², and a correlation coefficient (ρ) of 0.4323, suggesting moderate consistency but high variance.
• In ambient noise at 12 dB SNR, the DENBE algorithm with FFT-derived subbands achieved a mean bias of -1.847 ms in the 5011.87 Hz band and MSE of 11.5 ms², showing relatively low error in mid-to-high frequencies.
• The DENBE algorithm with filtered subbands showed the highest MSE (502.3 ms²) in the 398.11 Hz band under fan noise at -1 dB SNR, indicating poor performance in mid-frequency bands under low SNR.
• Across all conditions, DRR estimation exhibited higher bias and MSE in low-frequency bands (e.g., 25.12–50.12 Hz), with some algorithms showing negative bias (e.g., -11.07 ms in DENBE with filtered subbands at 25.12 Hz under fan noise at -1 dB SNR).
• Correlation coefficients (ρ) for DRR estimation were generally low (e.g., ρ = -0.1535 for DENBE with filtered subbands at 25.12 Hz under fan noise at -1 dB SNR), indicating poor alignment between estimated and ground-truth values in low-frequency bands.