[Paper Review] Determined BSS based on time-frequency masking and its application to harmonic vector analysis

Kohei Yatabe, Daichi Kitamura|arXiv (Cornell University)|Apr 29, 2020

Speech and Audio Processing68 references4 citations

TL;DR

This paper proposes Harmonic Vector Analysis (HVA), a determined blind source separation (BSS) method that models audio sources using a time-frequency mask based on cepstral sparsity to enhance harmonic structures. By integrating this mask into a plug-and-play primal-dual splitting framework, HVA achieves performance comparable to state-of-the-art ILRMA with lower computational cost, especially in speech and music separation tasks.

ABSTRACT

This paper proposes harmonic vector analysis (HVA) based on a general algorithmic framework of audio blind source separation (BSS) that is also presented in this paper. BSS for a convolutive audio mixture is usually performed by multichannel linear filtering when the numbers of microphones and sources are equal (determined situation). This paper addresses such determined BSS based on batch processing. To estimate the demixing filters, effective modeling of the source signals is important. One successful example is independent vector analysis (IVA) that models the signals via co-occurrence among the frequency components in each source. To give more freedom to the source modeling, a general framework of determined BSS is presented in this paper. It is based on the plug-and-play scheme using a primal-dual splitting algorithm and enables us to model the source signals implicitly through a time-frequency mask. By using the proposed framework, determined BSS algorithms can be developed by designing masks that enhance the source signals. As an example of its application, we propose HVA by defining a time-frequency mask that enhances the harmonic structure of audio signals via sparsity of cepstrum. The experiments showed that HVA outperforms IVA and independent low-rank matrix analysis (ILRMA) for both speech and music signals. A MATLAB code is provided along with the paper for a reference ( https://doi.org/10.24433/CO.9507820.v1 ).

Motivation & Objective

• To develop a general framework for determined BSS using time-frequency masking that enables flexible source modeling.
• To address the limitation of existing BSS methods like IVA and ILRMA in explicitly modeling harmonic structures in audio signals.
• To propose a novel BSS algorithm, HVA, that enhances harmonic content via cepstrum sparsity while maintaining computational efficiency.
• To demonstrate that the proposed masking-based BSS framework allows easy integration of multiple source priors through mask design.
• To provide a computationally efficient alternative to ILRMA that performs well on both speech and music signals without requiring iterative rank optimization.

Proposed method

• The paper introduces a plug-and-play BSS framework based on primal-dual splitting algorithms, enabling modular integration of any time-frequency mask for source enhancement.
• The demixing matrix is estimated by minimizing a cost function that combines a log-determinant term (for independence) and a penalty function derived from a source-specific mask.
• HVA is implemented by designing a custom time-frequency mask that promotes harmonic structure through sparsity in the cepstrum domain.
• The mask is computed via a frequency-directional Fourier transform pair and exponential mapping to emphasize harmonic components in the time-frequency representation.
• The optimization is solved using a proximal splitting algorithm, allowing efficient and stable convergence even with complex mask functions.
• The framework supports batch processing and allows the use of multiple masks in combination, enabling future extension to multi-criteria source separation.

Experimental results

Research questions

• RQ1 Can a general BSS framework based on time-frequency masking enable more flexible and effective source modeling than existing methods like IVA and ILRMA?
• RQ2 How well can a cepstrum-based mask that promotes harmonic structure perform in separating speech and music signals compared to state-of-the-art BSS methods?
• RQ3 Does the proposed plug-and-play masking framework allow for efficient and modular development of new BSS algorithms without redesigning the entire optimization pipeline?
• RQ4 What is the computational trade-off between HVA and ILRMA in terms of iteration time and convergence speed, especially in multi-channel scenarios?
• RQ5 To what extent can HVA generalize to non-harmonic signals, and what modifications are needed to extend its applicability?

Key findings

• HVA achieved performance comparable to ILRMA on both speech and music separation tasks, with a 200-iteration setting showing strong convergence and low error rates.
• For 2-channel mixtures, HVA with λ = 0.08 performed similarly to ILRMA despite ILRMA’s ability to model repetitive spectral patterns, indicating effective harmonic modeling.
• In 3-channel mixtures containing drum signals (non-harmonic), IVA outperformed ILRMA, but HVA still achieved comparable performance, suggesting robustness to mixed source types.
• HVA required 60.8 ms per iteration for speech and 164.3 ms for music on a Core i7-8700, making it more efficient than ILRMA (116.1 ms and 271.2 ms, respectively), though less efficient than IVA.
• Due to faster convergence (e.g., 20–50 iterations for Mixture A and B), HVA’s total runtime was typically lower than ILRMA’s, despite higher per-iteration cost.
• The mask-based framework allows easy extension by combining multiple masks, as demonstrated in prior work, suggesting strong potential for future improvements through multi-criteria masking.