[Paper Review] Conditioned-U-Net: Introducing a Control Mechanism in the U-Net for Multiple Source Separations

Gabriel Meseguer-Brocal, Geoffroy Peeters|arXiv (Cornell University)|Jul 2, 2019

Speech and Audio Processing19 references36 citations

TL;DR

The paper presents Conditioned-U-Net (C-U-Net), a single U-Net augmented with FiLM-based conditioning to perform multi-instrument source separation guided by a one-hot instrument vector.

ABSTRACT

Data-driven models for audio source separation such as U-Net or Wave-U-Net are usually models dedicated to and specifically trained for a single task, e.g. a particular instrument isolation. Training them for various tasks at once commonly results in worse performances than training them for a single specialized task. In this work, we introduce the Conditioned-U-Net (C-U-Net) which adds a control mechanism to the standard U-Net. The control mechanism allows us to train a unique and generic U-Net to perform the separation of various instruments. The C-U-Net decides the instrument to isolate according to a one-hot-encoding input vector. The input vector is embedded to obtain the parameters that control Feature-wise Linear Modulation (FiLM) layers. FiLM layers modify the U-Net feature maps in order to separate the desired instrument via affine transformations. The C-U-Net performs different instrument separations, all with a single model achieving the same performances as the dedicated ones at a lower cost.

Motivation & Objective

• Motivate multi-task music source separation without training separate models per instrument.
• Introduce a conditioning mechanism that adapts a generic U-Net to isolate a chosen instrument.
• Propose FiLM-based conditioning (simple and complex variants) and a lightweight condition generator.
• Demonstrate parameter efficiency and competitive performance on MUSDB18 compared to dedicated models.

Proposed method

• Incorporate FiLM layers into the U-Net encoder to condition feature maps based on external data.
• Use a one-hot instrument vector z to guide the conditioning via a condition generator that outputs gamma and beta parameters.
• Compare FiLM simple (shared gamma/beta across feature maps) versus FiLM complex (per-feature-map gamma/beta) layers.
• Explore two embedding strategies for the condition generator: fully connected (FC) and CNN-based embeddings.
• Train the conditioned network and the condition generator jointly on the MUSDB18 dataset with progressive training.
• Evaluate using magnitude spectrogram masks to reconstruct separated sources and compute SIR, SAR, and SDR.

Figure 1 : [Left part] Traditional approach: a dedicated U-Net is trained to separate a specific source. [Right part] Our proposition based on conditioning learning. The problem is divided in two: a standard U-Net (which provides generic source separation filters) and a control mechanism. This divis

Experimental results

Research questions

• RQ1 Can a single conditioned U-Net match the performance of four dedicated U-Nets for vocals, drums, bass, and rest on MUSDB18?
• RQ2 What is the impact of using FiLM simple versus FiLM complex layers on multi-instrument separation performance?
• RQ3 Does progressive training improve multi-task C-U-Net performance compared to non-progressive training?
• RQ4 How does the C-U-Net’s parameter count and scalability compare to dedicated models as the number of instruments grows?

Key findings

• C-U-Net achieves performance comparable to dedicated Fix-U-Nets across instruments, with FiLM simple layers performing as well as complex ones.
• Progressive training improves performance, especially for FiLM simple configurations.
• C-U-Nets maintain a fixed parameter count regardless of the number of instruments, offering substantial parameter efficiency over separate models.
• Pearson correlation between C-U-Net and dedicated models is consistently high (global corr > 0.9, p < 0.001), indicating similar behavior across tasks and metrics.
• In vocal, drums, and bass separation, C-U-Net-CoF variants show results on par with fixed dedicated models; Rest shows more variability, reflecting broader instrument combinations.

Figure 2 : [Top part] FiLM simple layer applies the same affine transformation to all the input feature maps $x$ . [Bottom part] In the FiLM complex layer, independent affine transformations are applied to each feature map $c$ .