[论文解读] Sound Event Detection in Multichannel Audio Using Spatial and Harmonic Features

Sharath Adavanne, Giambattista Parascandolo+3 more|arXiv (Cornell University)|Jun 7, 2017

Music and Audio Processing参考文献 18被引用 87

一句话总结

本论文提出在立体声录音上使用立体声空间特征与基于谐波音高的特征，并结合多标签 RNN-LSTM 以改进多音事件检测。结果显示， binaural features can outperform mono-channel baselines on a real-life dataset.

ABSTRACT

In this paper, we propose the use of spatial and harmonic features in combination with long short term memory (LSTM) recurrent neural network (RNN) for automatic sound event detection (SED) task. Real life sound recordings typically have many overlapping sound events, making it hard to recognize with just mono channel audio. Human listeners have been successfully recognizing the mixture of overlapping sound events using pitch cues and exploiting the stereo (multichannel) audio signal available at their ears to spatially localize these events. Traditionally SED systems have only been using mono channel audio, motivated by the human listener we propose to extend them to use multichannel audio. The proposed SED system is compared against the state of the art mono channel method on the development subset of TUT sound events detection 2016 database. The usage of spatial and harmonic features are shown to improve the performance of SED.

研究动机与目标

• Motivate automatic detection of overlapping sound events in real-life, multichannel audio.
• Extend SED beyond mono audio by leveraging spatial cues and pitch-related features.
• Demonstrate that combining log mel-band energies, pitch, and TDOA in a stereo framework improves detection performance.
• Evaluate the approach on the TUT SED 2016 development subset and compare against mono-channel baselines.

提出的方法

• Extract log mel-band energies for both stereo channels (40 mel-bands).
• Compute harmonic features: absolute pitch and its periodicity; top three dominant pitches per frame per channel.
• Compute multi-band TDOA features using GCC-PHAT across five mel-bands with three window lengths (120, 240, 480 ms) and median-filter the results (tdoa and tdoa3).
• Combine features into multi-label input vectors and train a two-hidden-layer LSTM RNN (2x32 units) with sigmoid outputs for multi-label classification.
• Normalize inputs, split sequences into 25-frame chunks, train with binary cross-entropy loss using Adam, apply early stopping, and threshold outputs at 0.5 for activity decisions.

实验结果

研究问题

• RQ1 Does incorporating spatial (TDOA) and harmonic (pitch) features with stereo log mel-band energies improve polyphonic SED over mono-channel baselines?
• RQ2 How does the proposed multichannel feature set perform across different real-life contexts (home and residential area) relative to mono-channel systems?
• RQ3 What is the impact of different feature combinations on the segment-based error rate and F-score in SED?

主要发现

• Spatial and harmonic features combined with stereo input improve polyphonic SED performance relative to mono baselines on the assessed dataset.
• The proposed binaural features (mel_2 and related combinations) generally achieve competitive or superior F-scores with comparable error rates across contexts.
• Several feature combinations outperform mono-channel baselines, indicating the value of incorporating spatial cues (TDOA) in SED for real-life recordings.
• On a small dataset (≈60 minutes), binaural features show promise, with some configurations achieving top performances in related challenge submissions.