[Paper Review] Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation

Qiuqiang Kong, Yin Cao+3 more|arXiv (Cornell University)|Sep 12, 2021

Speech and Audio Processing38 citations

TL;DR

The paper decouples magnitude and phase in complex ideal ratio masks (cIRMs), allows unbounded mask magnitudes by combining with direct magnitude prediction, and introduces a 143-layer residual UNet (ResUNet) for state-of-the-art music source separation on MUSDB18, especially vocals (SDR 8.98 dB).

ABSTRACT

Deep neural network based methods have been successfully applied to music source separation. They typically learn a mapping from a mixture spectrogram to a set of source spectrograms, all with magnitudes only. This approach has several limitations: 1) its incorrect phase reconstruction degrades the performance, 2) it limits the magnitude of masks between 0 and 1 while we observe that 22% of time-frequency bins have ideal ratio mask values of over~1 in a popular dataset, MUSDB18, 3) its potential on very deep architectures is under-explored. Our proposed system is designed to overcome these. First, we propose to estimate phases by estimating complex ideal ratio masks (cIRMs) where we decouple the estimation of cIRMs into magnitude and phase estimations. Second, we extend the separation method to effectively allow the magnitude of the mask to be larger than 1. Finally, we propose a residual UNet architecture with up to 143 layers. Our proposed system achieves a state-of-the-art MSS result on the MUSDB18 dataset, especially, a SDR of 8.98~dB on vocals, outperforming the previous best performance of 7.24~dB. The source code is available at: https://github.com/bytedance/music_source_separation

Motivation & Objective

• Motivate improving MSS by addressing phase estimation limitations and magnitude mask bounds.
• Propose decoupled magnitude and phase estimation for complex masks (cIRMs).
• Enable mask magnitudes >1 by combining bounded masks with direct magnitude prediction.
• Introduce a very deep residual UNet (143 layers) to enhance MSS performance.
• Demonstrate state-of-the-art results on MUSDB18, especially vocals.

Proposed method

• Decouple magnitude and phase for cIRM estimation by predicting a bounded magnitude mask, plus real/imaginary components to derive cosine and sine of the mask phase.
• Compute cIRM from M_mag, cos(angle M), sin(angle M) and X to recover S via S = M X with magnitude scaling and phase rotation.
• Combine the bounded mask with a direct magnitude prediction term to estimate the source magnitude as |S| = ReLU(M_mag ⊙ |X| + Q).
• Predict four outputs via a shared backbone: M_mag, Q, P_r, P_i; apply appropriate activations and construct the complex mask M = M_r + j M_i for separation.
• Develop a 143-layer residual UNet (ResUNet) with residual encoder blocks, residual decoder blocks, and intermediate convolutional blocks to enable deep architectures for MSS.

Experimental results

Research questions

• RQ1 Can decoupling magnitude and phase improve complex mask-based MSS compared to traditional magnitude-based approaches?
• RQ2 How does allowing mask magnitudes > 1 affect MSS performance and upper bounds?
• RQ3 Does a very deep residual UNet provide tangible gains for MSS over shallower architectures?
• RQ4 Does combining bounded mask estimation with direct magnitude prediction yield better SDRs than either approach alone?
• RQ5 What are the performance gains on MUSDB18 across vocal and instrumental sources when applying these techniques?

Key findings

• The proposed decoupled cIRM estimation improves performance over phase-less/magnitude-only approaches, enhancing vocals SDR.
• Allowing unbounded mask magnitudes (via cIRM with magnitude >1) yields higher theoretical upper bounds and practical gains in SDR.
• Combining a bounded mask with a direct magnitude prediction term improves MSS performance over using either mechanism alone.
• A 143-layer residual UNet (ResUNet) significantly outperforms shallower UNets, confirming deep architectures help MSS.
• On MUSDB18, the final ResUNetDecouple+ system achieves 8.98 dB SDR for vocals, outperforming previous best (7.24 dB) and other baselines; other sources also show improvements (e.g., bass 6.04 dB, drums 6.62 dB, other 5.29 dB, accompaniment 16.63 dB).