[Paper Review] D3Net: Densely connected multidilated DenseNet for music source separation

Naoya Takahashi, Yuki Mitsufuji|arXiv (Cornell University)|Oct 5, 2020

Speech and Audio Processing20 references48 citations

TL;DR

D3Net introduces a densely connected multidilated DenseNet architecture for music source separation, achieving state-of-the-art SDR on MUSDB18 by modeling multi-resolution information within a single layer while mitigating aliasing.

ABSTRACT

Music source separation involves a large input field to model a long-term dependence of an audio signal. Previous convolutional neural network (CNN)-based approaches address the large input field modeling using sequentially down- and up-sampling feature maps or dilated convolution. In this paper, we claim the importance of a rapid growth of a receptive field and a simultaneous modeling of multi-resolution data in a single convolution layer, and propose a novel CNN architecture called densely connected dilated DenseNet (D3Net). D3Net involves a novel multi-dilated convolution that has different dilation factors in a single layer to model different resolutions simultaneously. By combining the multi-dilated convolution with DenseNet architecture, D3Net avoids the aliasing problem that exists when we naively incorporate the dilated convolution in DenseNet. Experimental results on MUSDB18 dataset show that D3Net achieves state-of-the-art performance with an average signal to distortion ratio (SDR) of 6.01 dB.

Motivation & Objective

• Motivate large receptive fields and multi-resolution modeling for music source separation.
• Propose a multidilated convolution within DenseNet to model multiple resolutions in one layer.
• Mitigate aliasing when combining dilation with dense skip connections.
• Introduce a nested D2/D3 block architecture to reuse features across resolutions and depths.

Proposed method

• Define multidilated convolution where each skip-connection channel uses a different dilation (d_i = 2^i).
• Integrate multidilated convolutions into a DenseNet-like densely connected block (D2 block).
• Nest D2 blocks into a D3Net architecture with channel reduction to control growth in feature maps.
• Train four networks (one per source) on MUSDB18 using STFT magnitude inputs and an MWF post-filter.
• Use a multiscale multiband architecture with band-specific and full-band modules.
• Evaluate with SDR on MUSDB18 and perform ablation studies to assess aliasing effects.

Experimental results

Research questions

• RQ1 How can we increase the receptive field rapidly while preserving multi-resolution information in a CNN for MSS?
• RQ2 Does a multidilated convolution within DenseNet mitigate aliasing and improve source separation?
• RQ3 Does a nested D2/D3Net architecture improve MSS performance over standard DenseNets with dilation?
• RQ4 What is the impact of multidilation versus standard dilation and no dilation on SDR in MSS?
• RQ5 How does D3Net compare to state-of-the-art MSS methods on MUSDB18?

Key findings

• D3Net achieves state-of-the-art average SDR (6.01 dB) on MUSDB18.
• Multidilated convolution outperforms standard dilated convolution by reducing aliasing and enhancing feature utilization.
• D3Net improves vocals, drums, and accompaniment SDR compared to baselines, with strongest gains in vocal and drum separation.
• Ablation shows naive dilation in DenseNet causes aliasing; multidilation with dense connections preserves information across resolutions.
• Using extra data further improves D3Net’s SDR, achieving higher vocals and overall performance than several data-augmented baselines.