[Paper Review] Separation of Instrument Sounds using Non-negative Matrix Factorization with Spectral Envelope Constraints

Jeongsoo Park, Jaeyoung Shin+1 more|arXiv (Cornell University)|Jan 12, 2018

Speech and Audio Processing30 references3 citations

TL;DR

This paper proposes a non-negative matrix factorization (NMF) method for music source separation that incorporates spectral envelope constraints via linear prediction to improve timbral accuracy. By constraining the spectral envelope components of NMF bases using instrument-specific or learned envelopes, the method achieves superior separation performance, especially in blind scenarios, outperforming conventional NMF and state-of-the-art methods in SDR, SIR, and SAR metrics.

ABSTRACT

Spectral envelope is one of the most important features that characterize the timbre of an instrument sound. However, it is difficult to use spectral information in the framework of conventional spectrogram decomposition methods. We overcome this problem by suggesting a simple way to provide a constraint on the spectral envelope calculated by linear prediction. In the first part of this study, we use a pre-trained spectral envelope of known instruments as the constraint. Then we apply the same idea to a blind scenario in which the instruments are unknown. The experimental results reveal that the proposed method outperforms the conventional methods.

Motivation & Objective

• To address the challenge of accurately modeling instrument timbre in spectrogram-based source separation, where spectral envelope is a key timbral feature.
• To overcome the limitation of conventional NMF methods in effectively utilizing spectral envelope information during decomposition.
• To develop an informed source separation method using pre-trained instrument-specific spectral envelopes as constraints.
• To extend the informed approach to a blind scenario where true instrument envelopes are unknown, by learning representative envelopes from data.
• To evaluate the proposed method against state-of-the-art techniques using objective metrics such as SDR, SIR, and SAR.

Proposed method

• The method decomposes the spectrogram into basis vectors and activation matrices using NMF, with bases explicitly separated into spectral envelope and excitation components via linear prediction.
• Spectral envelope constraints are applied by enforcing the envelope part of the basis vectors to remain close to a target envelope during iterative NMF optimization.
• In the informed case, the target spectral envelopes are derived from pre-trained models of known instruments (e.g., violin, clarinet).
• In the blind case, the method learns representative spectral envelopes directly from the mixture data using a self-regularization strategy based on activation magnitude.
• The weighting of the envelope constraint is dynamically adjusted using the L1-norm of the activation vector, with optimal exponent p=5 found to maximize performance.
• Sparse initialization of basis vectors is applied to improve convergence and performance, especially with limited numbers of bases.

Experimental results

Research questions

• RQ1 Can spectral envelope constraints improve the accuracy of instrument timbre representation in NMF-based music source separation?
• RQ2 How does the use of pre-trained instrument-specific spectral envelopes affect separation performance compared to unconstrained NMF?
• RQ3 Can the proposed method generalize to blind scenarios where true instrument envelopes are not available?
• RQ4 What is the optimal weighting strategy for spectral envelope constraints based on activation energy?
• RQ5 How does the number of bases and initialization method influence the performance of the proposed NMF framework?

Key findings

• The proposed informed NMF method achieved the highest SDR (5.55 dB), SIR (9.84 dB), and SAR (11.20 dB) on the RWC database, significantly outperforming baseline methods.
• The proposed blind NMF method achieved 3.16 dB SDR, which was the highest among all blind methods tested, including Spiertz et al.'s hierarchical, MFCC, and NMF-based approaches.
• The performance of the proposed blind method with sparse initialization reached 3.34 dB SDR at 100 bases, showing consistent improvement over normal initialization and baseline methods.
• The optimal exponent for the activation-based weighting was found to be p=5, as it maximized SDR and stabilized performance across different configurations.
• The proposed informed method was robust to the number of bases, maintaining SDR above 5.5 dB regardless of base count (20–100), indicating strong generalization.
• The use of sparse initialization consistently improved SDR across all base counts and methods, demonstrating its effectiveness in shaping more meaningful basis vectors.