[Paper Review] Regularized Least-Mean-Square Algorithms

Yilun Chen, Yuantao Gu+1 more|arXiv (Cornell University)|Dec 22, 2010

Advanced Adaptive Filtering TechniquesEngineering20 references83 citations

TL;DR

This paper proposes a family of regularized Least-Mean-Square (LMS) algorithms that incorporate convex regularization to improve performance in adaptive system identification, especially for sparse or group-sparse systems. By deriving a closed-form expression for the regularization parameter, the method provably dominates conventional LMS in terms of mean square deviation, with demonstrated advantages in convergence speed and steady-state error under sparsity assumptions.

ABSTRACT

We consider adaptive system identification problems with convex constraints and propose a family of regularized Least-Mean-Square (LMS) algorithms. We show that with a properly selected regularization parameter the regularized LMS provably dominates its conventional counterpart in terms of mean square deviations. We establish simple and closed-form expressions for choosing this regularization parameter. For identifying an unknown sparse system we propose sparse and group-sparse LMS algorithms, which are special examples of the regularized LMS family. Simulation results demonstrate the advantages of the proposed filters in both convergence rate and steady-state error under sparsity assumptions on the true coefficient vector.

Motivation & Objective

• To develop a general framework for regularized LMS algorithms that incorporate convex constraints and time-varying regularization.
• To establish a systematic, closed-form method for selecting the regularization parameter to ensure performance dominance over conventional LMS.
• To extend the framework to sparse and group-sparse system identification using ℓ1 and ℓ1,2 regularization, respectively.
• To demonstrate provable performance gains in convergence rate and steady-state error under both white and correlated input signals.
• To show robustness to model mis-specification and superior performance compared to projection-based methods at equivalent computational cost.

Proposed method

• Introduces a regularized LMS update equation with an additional sub-gradient term derived from convex regularization functions.
• Derives a closed-form expression for the regularization parameter ρn* that ensures the regularized LMS dominates the conventional LMS in mean square deviation.
• Applies ℓ1 regularization to enforce sparsity, recovering ZA-LMS and RZA-LMS as special cases.
• Uses ℓ1,2 regularization (group sparsity) to promote structured sparsity in block-structured systems.
• Establishes theoretical dominance via induction by showing the expected mean square deviation of the regularized LMS is bounded above by that of the conventional LMS under the derived parameter condition.
• Analyzes performance under both white and correlated input signals, proving robustness and consistent superiority.

Experimental results

Research questions

• RQ1 Can a general family of regularized LMS algorithms be developed that provably dominates conventional LMS under convex constraints?
• RQ2 What closed-form expression for the regularization parameter ensures dominance of the regularized LMS over the conventional LMS?
• RQ3 How can ℓ1 and ℓ1,2 regularization be effectively integrated into the LMS framework to exploit sparsity in system coefficients?
• RQ4 Does the proposed method maintain performance advantages under correlated input signals, not just white noise?
• RQ5 How does the regularized LMS compare to contemporary projection-based adaptive filtering methods in terms of convergence and steady-state error?

Key findings

• The regularized LMS algorithm provably dominates the conventional LMS in mean square deviation when the regularization parameter ρn is selected within the interval [0, 2ρn*], where ρn* is derived in closed form.
• For white input signals, the derived parameter selection guarantees that the regularized LMS achieves lower mean square deviation than the conventional LMS.
• The sparse LMS filters ZA-LMS and RZA-LMS are shown to be special cases of the proposed regularized LMS family.
• The group-sparse LMS with ℓ1,2 regularization achieves provable dominance over conventional LMS for both white and correlated inputs, with a derived closed-form parameter selection rule.
• Numerical simulations confirm faster convergence and lower steady-state error for the proposed filters under sparsity assumptions.
• The proposed method outperforms equivalent-cost projection-based adaptive filters and is robust to model mis-specification.