[Paper Review] Multi-Resolution Fully Convolutional Neural Networks for Monaural Audio Source Separation

Emad M. Grais, Hagen Wierstorf+2 more|arXiv (Cornell University)|Oct 28, 2017

Speech and Audio Processing24 references4 citations

TL;DR

This paper proposes Multi-Resolution Fully Convolutional Neural Networks (MR-FCNN) for monaural audio source separation, where each layer uses multiple filter sets of different sizes to extract both global and local time-frequency features. The MR-FCNN outperforms standard DNNs and single-resolution FCNNs, achieving statistically significant improvements in SDR and SAR, with p-values < 0.0025 for all model comparisons.

ABSTRACT

In deep neural networks with convolutional layers, each layer typically has fixed-size/single-resolution receptive field (RF). Convolutional layers with a large RF capture global information from the input features, while layers with small RF size capture local details with high resolution from the input features. In this work, we introduce novel deep multi-resolution fully convolutional neural networks (MR-FCNN), where each layer has different RF sizes to extract multi-resolution features that capture the global and local details information from its input features. The proposed MR-FCNN is applied to separate a target audio source from a mixture of many audio sources. Experimental results show that using MR-FCNN improves the performance compared to feedforward deep neural networks (DNNs) and single resolution deep fully convolutional neural networks (FCNNs) on the audio source separation problem.

Motivation & Objective

• To address the challenge of separating a target audio source from a monaural mixture containing multiple overlapping sources.
• To improve feature extraction in audio source separation by capturing both global context and local details through multi-resolution receptive fields.
• To develop a novel deep learning architecture that applies multi-resolution convolutional filters at every layer, enabling joint learning of diverse spectro-temporal patterns.
• To demonstrate that multi-resolution feature learning enhances separation performance compared to single-resolution or fully connected networks.

Proposed method

• The MR-FCNN uses fully convolutional layers where each layer applies multiple sets of filters with different receptive field sizes (e.g., 3×3, 7×9, 13×21) in parallel.
• Each filter set generates feature maps at different resolutions, enabling the network to capture both fine-grained local details and broader spectral-temporal structures.
• The network architecture follows an encoder-decoder structure with ReLU activation, transposed convolutions in the decoder, and symmetric filter size and channel count design.
• The model is trained end-to-end using Adam optimization with early stopping and learning rate decay on magnitude spectrograms of mixed and target audio signals.
• The number of parameters in the MR-FCNN (558,181) is kept comparable to FCNN (445,173) and DNN (4,206,600) for fair performance comparison.
• The input and output are 2D magnitude spectrograms, and the network learns to reconstruct the target source from the mixture using multi-resolution feature learning.

Experimental results

Research questions

• RQ1 Can multi-resolution convolutional filters at each layer improve audio source separation performance compared to single-resolution or fully connected networks?
• RQ2 Does the joint extraction of global and local features via multiple filter sizes lead to better separation of target sources in monaural mixtures?
• RQ3 How do the SDR, SIR, and SAR metrics of the MR-FCNN compare to those of DNNs and single-resolution FCNNs on a challenging audio separation task?
• RQ4 Is the performance gain of MR-FCNN statistically significant across multiple evaluation metrics?

Key findings

• The MR-FCNN significantly outperforms the DNN in SDR (p = 2×10⁻⁷) and SAR (p = 9×10⁻⁷), with a large effect size.
• The MR-FCNN also outperforms the single-resolution FCNN in SDR (p = 0.0025) and SAR (p = 3×10⁻⁵), demonstrating statistical significance.
• All performance differences between the three models (DNN, FCNN, MR-FCNN) are statistically significant at p < 0.05 after Bonferroni correction.
• The input mixed signal has low SDR and SIR, indicating a highly challenging separation task, which underscores the importance of advanced feature learning.
• The MR-FCNN achieves higher SDR and SAR than both DNN and FCNN, suggesting better reconstruction of the target source and reduced interference.
• The model’s architecture, with varying filter sizes per layer (e.g., 13×21, 7×9, 3×3), enables effective multi-resolution feature extraction across all layers.