[Paper Review] Deep-RLS: A Model-Inspired Deep Learning Approach to Nonlinear PCA

Zahra Esmaeilbeig, Shahin Khobahi+1 more|arXiv (Cornell University)|Nov 15, 2020

Blind Source Separation Techniques15 references4 citations

TL;DR

This paper proposes Deep-RLS, a deep learning framework inspired by the recursive least squares (RLS) algorithm for nonlinear principal component analysis (PCA) in blind source separation (BSS). By unfolding RLS iterations into a deep neural network with learnable parameters, including a forgetting factor regularized via ReLU constraints, Deep-RLS achieves significantly lower mean squared error (MSE) in recovering source signals compared to traditional RLS, especially with fewer layers and increasing source dimensions.

ABSTRACT

In this work, we consider the application of model-based deep learning in nonlinear principal component analysis (PCA). Inspired by the deep unfolding methodology, we propose a task-based deep learning approach, referred to as Deep-RLS, that unfolds the iterations of the well-known recursive least squares (RLS) algorithm into the layers of a deep neural network in order to perform nonlinear PCA. In particular, we formulate the nonlinear PCA for the blind source separation (BSS) problem and show through numerical analysis that Deep-RLS results in a significant improvement in the accuracy of recovering the source signals in BSS when compared to the traditional RLS algorithm.

Motivation & Objective

• To address the limitations of linear PCA in separating statistically independent source signals due to its reliance on second-order statistics.
• To improve blind source separation (BSS) performance by incorporating higher-order statistics through nonlinear PCA.
• To develop a deep learning architecture that integrates the structure of the classical RLS algorithm for adaptive, real-time signal recovery.
• To leverage deep unfolding to create a differentiable, end-to-end trainable network that enhances convergence and accuracy over standard RLS.

Proposed method

• The Deep-RLS network unfolds the recursive least squares (RLS) algorithm into a deep neural network with T layers, each corresponding to one RLS iteration.
• Each layer computes an estimate of the source signals using a separating matrix W(k), a nonlinear activation function g(·), and a forgetting factor ωk.
• The forgetting factor ωk is constrained via a ReLU-based regularization term in the loss function to ensure 0 < ωk ≤ 1, maintaining numerical stability.
• The network is trained using a cumulative mean squared error (MSE) loss over all layers, with backpropagation through time to optimize parameters {Γk} = {Hk, ωk, bk}.
• The training procedure uses batch learning with Adam optimizer and a fixed mixing matrix A drawn from a standard normal distribution.
• The architecture is designed to mimic the recursive structure of RLS while enabling end-to-end learning of the separating matrix and nonlinear transformation.

Experimental results

Research questions

• RQ1 Can the recursive least squares (RLS) algorithm be effectively unfolded into a deep neural network to improve nonlinear PCA performance in blind source separation?
• RQ2 How does the performance of the proposed Deep-RLS framework compare to traditional RLS in terms of source signal recovery accuracy?
• RQ3 To what extent does the inclusion of learnable parameters, such as the forgetting factor, enhance the robustness and convergence of nonlinear PCA?
• RQ4 Can the deep unfolding approach with ReLU-regularized forgetting factors yield better generalization and lower MSE than fixed-parameter RLS?

Key findings

• Deep-RLS achieves significantly lower mean squared error (MSE) in recovering source signals than traditional RLS, even with fewer layers.
• For m=2 source signals, Deep-RLS outperforms RLS with β=0.99, achieving lower MSE across all tested numbers of layers/iterations.
• As the number of source signals m increases, Deep-RLS maintains superior performance compared to RLS, demonstrating scalability.
• The ReLU-based regularization of the forgetting factor ωk ensures feasibility (0 < ωk ≤ 1) and contributes to stable training and improved convergence.
• The cumulative MSE loss across all layers enables effective backpropagation and parameter optimization, leading to better signal reconstruction.
• Empirical results show that the Deep-RLS network generalizes well to unseen test data, with a test set size of 100 and training set of 1,000 samples.