[Paper Review] Automatic music mixing with deep learning and out-of-domain data

Marco A. Martínez-Ramírez, Wei‐Hsiang Liao+4 more|arXiv (Cornell University)|Aug 24, 2022

Music and Audio Processing12 citations

TL;DR

This paper proposes a novel data preprocessing method that enables training deep learning models for automatic music mixing using out-of-domain, wet multitrack recordings—commonly available but previously unusable—by normalizing audio effect features (loudness, EQ, panning, DRC, reverb). The approach achieves mixes indistinguishable from professional human-made mixes in subjective tests with expert engineers, demonstrating a significant leap in automated mixing quality without requiring dry multitrack data.

ABSTRACT

Music mixing traditionally involves recording instruments in the form of clean, individual tracks and blending them into a final mixture using audio effects and expert knowledge (e.g., a mixing engineer). The automation of music production tasks has become an emerging field in recent years, where rule-based methods and machine learning approaches have been explored. Nevertheless, the lack of dry or clean instrument recordings limits the performance of such models, which is still far from professional human-made mixes. We explore whether we can use out-of-domain data such as wet or processed multitrack music recordings and repurpose it to train supervised deep learning models that can bridge the current gap in automatic mixing quality. To achieve this we propose a novel data preprocessing method that allows the models to perform automatic music mixing. We also redesigned a listening test method for evaluating music mixing systems. We validate our results through such subjective tests using highly experienced mixing engineers as participants.

Motivation & Objective

• To address the critical lack of dry multitrack recordings, which limits the performance of deep learning models in automatic music mixing.
• To explore whether out-of-domain, wet multitrack data can be repurposed for supervised training of automatic mixing systems.
• To develop a data normalization technique that enables models to learn to reverse audio effects and reconstruct original mixes.
• To design a perceptually grounded listening test with experienced mixing engineers to evaluate system performance objectively.
• To achieve automatic mixing quality on par with professional human-made mixes using only existing wet datasets.

Proposed method

• Proposes a novel data preprocessing pipeline that normalizes each audio effect class (loudness, EQ, panning, DRC, reverb) across stems using average features computed from the out-of-domain dataset.
• Computes average loudness (LUFS) and frequency magnitude spectra (via STFT) per instrument type to establish target normalization profiles.
• Applies effect-specific normalization: loudness via gain adjustment, EQ via filter matching to target spectra, panning via inter-channel level normalization, DRC via dynamic range equalization, and reverb via spectral decay normalization.
• Trains a deep neural network to predict mixing parameters (gain, EQ, panning, DRC, reverb) by denormalizing the normalized stems, effectively learning to reverse the effects.
• Uses stereo-invariant loss functions to improve generalization and perceptual quality in stereo output.
• Applies the same preprocessing to real dry multitrack data at inference, enabling end-to-end automatic mixing.

Experimental results

Research questions

• RQ1 Can deep learning models trained on out-of-domain, wet multitrack data produce music mixes comparable to professional human-made mixes?
• RQ2 Can a data normalization technique effectively bridge the distribution gap between wet training data and dry inference data for automatic mixing?
• RQ3 How do expert mixing engineers rate the perceptual quality of model-generated mixes compared to professional human mixes?
• RQ4 Does the proposed stereo-invariant loss function improve the perceptual quality of the final mix?
• RQ5 Can the model generalize across diverse music genres and maintain high-quality mixing without access to dry training data?

Key findings

• The proposed method achieved music mixes that were indistinguishable from professional human-made mixes in terms of Production Value and Excitement in subjective listening tests with experienced engineers.
• Model-generated mixes scored significantly higher than professional mixes in Clarity, indicating superior perceptual transparency.
• The listening test revealed that professional mixes were rarely rated as 'very good' by experts, highlighting the difficulty of the evaluation task and the high bar achieved by the proposed system.
• The model outperformed baseline Wave-U-Net models (WUN) in all perceptual criteria, with statistically significant improvements (p < 0.001) in Production Value and Excitement.
• The stereo-invariant loss function contributed to improved generalization and perceptual quality, particularly in spatial imaging and balance.
• The method successfully enabled training on existing wet datasets like MUSDB18, bypassing the need for expensive dry multitrack collection.