[论文解读] D3Net: Densely connected multidilated DenseNet for music source separation

Naoya Takahashi, Yuki Mitsufuji|arXiv (Cornell University)|Oct 5, 2020

Speech and Audio Processing参考文献 20被引用 48

一句话总结

D3Net 引入了一种密集连接的多扩张 DenseNet 架构用于音乐源分离，通过在单层内建模多分辨率信息同时减小混叠，达到 MUSDB18 的最先进 SDR。

ABSTRACT

Music source separation involves a large input field to model a long-term dependence of an audio signal. Previous convolutional neural network (CNN)-based approaches address the large input field modeling using sequentially down- and up-sampling feature maps or dilated convolution. In this paper, we claim the importance of a rapid growth of a receptive field and a simultaneous modeling of multi-resolution data in a single convolution layer, and propose a novel CNN architecture called densely connected dilated DenseNet (D3Net). D3Net involves a novel multi-dilated convolution that has different dilation factors in a single layer to model different resolutions simultaneously. By combining the multi-dilated convolution with DenseNet architecture, D3Net avoids the aliasing problem that exists when we naively incorporate the dilated convolution in DenseNet. Experimental results on MUSDB18 dataset show that D3Net achieves state-of-the-art performance with an average signal to distortion ratio (SDR) of 6.01 dB.

研究动机与目标

• Motivate large receptive fields and multi-resolution modeling for music source separation.
• Propose a multidilated convolution within DenseNet to model multiple resolutions in one layer.
• Mitigate aliasing when combining dilation with dense skip connections.
• Introduce a nested D2/D3 block architecture to reuse features across resolutions and depths.

提出的方法

• Define multidilated convolution where each skip-connection channel uses a different dilation (d_i = 2^i).
• Integrate multidilated convolutions into a DenseNet-like densely connected block (D2 block).
• Nest D2 blocks into a D3Net architecture with channel reduction to control growth in feature maps.
• Train four networks (one per source) on MUSDB18 using STFT magnitude inputs and an MWF post-filter.
• Use a multiscale multiband architecture with band-specific and full-band modules.
• Evaluate with SDR on MUSDB18 and perform ablation studies to assess aliasing effects.

实验结果

研究问题

• RQ1 How can we increase the receptive field rapidly while preserving multi-resolution information in a CNN for MSS?
• RQ2 Does a multidilated convolution within DenseNet mitigate aliasing and improve source separation?
• RQ3 Does a nested D2/D3Net architecture improve MSS performance over standard DenseNets with dilation?
• RQ4 What is the impact of multidilation versus standard dilation and no dilation on SDR in MSS?
• RQ5 How does D3Net compare to state-of-the-art MSS methods on MUSDB18?

主要发现

SDR in dB	Method	Vocals	Drums	Bass	Other	Acco.	Avg.
6.60	TAK1 (MMDenseLSTM) [ 10 ]	6.60	6.43	5.16	4.15	12.83	5.59
5.93	UHL2 (BLSTM ensemble) [ 3 ]	5.93	5.92	5.03	4.19	12.23	5.27
6.85	GRU dilation 1 [ 11 ]	6.85	5.86	4.86	4.65	13.40	5.56
6.32	UMX [ 19 ]	6.32	5.73	5.23	4.02	-	5.33
6.29	demucs* [ 7 ]	6.29	6.08	5.83	4.12	-	5.58
6.40	Meta-TasNet* [ 8 ]	6.40	5.91	5.58	4.19	-	5.52
6.92	Nachmani et. al.* [ 20 ]	6.92	6.15	5.88	4.32	-	5.82
6.86	D3Net w/o dilation	6.86	6.37	4.97	4.21	13.19	5.60
7.12	D3Net standard dilation	7.12	6.61	5.19	4.53	13.39	5.86
7.24	D3Net (proposed)	7.24	7.01	5.25	4.53	13.52	6.01

• D3Net achieves state-of-the-art average SDR (6.01 dB) on MUSDB18.
• Multidilated convolution outperforms standard dilated convolution by reducing aliasing and enhancing feature utilization.
• D3Net improves vocals, drums, and accompaniment SDR compared to baselines, with strongest gains in vocal and drum separation.
• Ablation shows naive dilation in DenseNet causes aliasing; multidilation with dense connections preserves information across resolutions.
• Using extra data further improves D3Net’s SDR, achieving higher vocals and overall performance than several data-augmented baselines.