[Paper Review] Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures

John R. Hershey, Jonathan Le Roux+1 more|arXiv (Cornell University)|Sep 9, 2014

Domain Adaptation and Few-Shot Learning26 references254 citations

TL;DR

This paper introduces deep unfolding, a method that transforms iterative model-based inference algorithms into deep neural network architectures by untying parameters across layers. By applying this to non-negative matrix factorization for speech enhancement, the authors create a parameter-efficient, interpretable deep network that outperforms standard DNNs with significantly fewer parameters while preserving domain-specific constraints like signal additivity.

ABSTRACT

Model-based methods and deep neural networks have both been tremendously successful paradigms in machine learning. In model-based methods, problem domain knowledge can be built into the constraints of the model, typically at the expense of difficulties during inference. In contrast, deterministic deep neural networks are constructed in such a way that inference is straightforward, but their architectures are generic and it is unclear how to incorporate knowledge. This work aims to obtain the advantages of both approaches. To do so, we start with a model-based approach and an associated inference algorithm, and \emph{unfold} the inference iterations as layers in a deep network. Rather than optimizing the original model, we \emph{untie} the model parameters across layers, in order to create a more powerful network. The resulting architecture can be trained discriminatively to perform accurate inference within a fixed network size. We show how this framework allows us to interpret conventional networks as mean-field inference in Markov random fields, and to obtain new architectures by instead using belief propagation as the inference algorithm. We then show its application to a non-negative matrix factorization model that incorporates the problem-domain knowledge that sound sources are additive. Deep unfolding of this model yields a new kind of non-negative deep neural network, that can be trained using a multiplicative backpropagation-style update algorithm. We present speech enhancement experiments showing that our approach is competitive with conventional neural networks despite using far fewer parameters.

Motivation & Objective

• To bridge the gap between model-based methods, which embed domain knowledge but suffer from slow inference, and deep neural networks, which are fast but lack interpretability.
• To develop a general framework that transforms iterative inference algorithms into trainable, layered deep architectures.
• To enable discriminative training of these architectures while preserving the structural constraints of the original model-based approach.
• To demonstrate that deep unfolding yields novel, efficient, and interpretable neural networks for real-world applications like speech enhancement.

Proposed method

• Unfold the iterations of an iterative inference algorithm (e.g., multiplicative updates in NMF) into a sequence of layers in a deep network.
• Untie the model parameters across layers to allow discriminative training, increasing representational capacity beyond the original model.
• Use gradient-based backpropagation to train the network, with a multiplicative update rule derived from the original inference algorithm.
• Apply the framework to Markov random fields and belief propagation to unify conventional sigmoid networks and alternative deep architectures.
• Design a non-negative deep network by unfolding the NMF inference process, preserving the additivity constraint of sound sources.
• Train the resulting architecture using a multiplicative backpropagation-style algorithm tailored to non-negative parameters.

Experimental results

Research questions

• RQ1 Can iterative model-based inference algorithms be systematically transformed into deep neural network architectures with improved expressivity and trainability?
• RQ2 How can domain-specific constraints—such as signal additivity in audio—be embedded into deep learning models through model-based design?
• RQ3 Can deep unfolding produce architectures that outperform standard DNNs in accuracy while using significantly fewer parameters?
• RQ4 What is the impact of layer-wise parameter untied training on performance and generalization in deep unfolding architectures?
• RQ5 How does the choice of inference algorithm (e.g., mean field vs. belief propagation) influence the resulting deep network architecture?

Key findings

• The deep NMF architecture with K=25, C=2 achieves an SDR of 9.64 dB using only 440K parameters, outperforming a DNN with 5.5M parameters that achieves 9.57 dB.
• The smallest deep NMF topology (K=25, C=2) outperforms the best DNN despite using an order of magnitude fewer parameters.
• Discriminative training of the first layer yields the largest performance gain, and training deeper layers consistently improves performance, especially in low SNR conditions.
• Increasing the number of layers from R^l=100 to R^l=1000 yields only modest gains, suggesting diminishing returns or data/optimization bottlenecks.
• The framework unifies conventional sigmoid networks as unfolded mean-field inference and enables new architectures via belief propagation-based unfolding.
• The multiplicative backpropagation algorithm successfully trains the non-negative deep network, preserving the non-negativity constraint and enabling effective optimization.