[Paper Review] Geometric Optimization Methods for Adaptive Filtering

Steven T. Smith|arXiv (Cornell University)|May 8, 2013

Advanced Adaptive Filtering Techniques27 references140 citations

TL;DR

This paper introduces Riemannian geometric optimization methods—specifically Riemannian Newton and conjugate gradient algorithms—on manifolds such as spheres, Stiefel, and Grassmann manifolds to solve eigenvalue and singular value problems in adaptive filtering. It demonstrates superlinear convergence for the conjugate gradient method on symmetric spaces, achieving $O(nk^2)$ operations and $O(k)$ matrix-vector multiplications per step, with numerical experiments confirming convergence to machine precision within 50 iterations on a 294-dimensional manifold.

ABSTRACT

The techniques and analysis presented in this thesis provide new methods to solve optimization problems posed on Riemannian manifolds. These methods are applied to the subspace tracking problem found in adaptive signal processing and adaptive control. A new point of view is offered for the constrained optimization problem. Some classical optimization techniques on Euclidean space are generalized to Riemannian manifolds. Several algorithms are presented and their convergence properties are analyzed employing the Riemannian structure of the manifold. Specifically, two new algorithms, which can be thought of as Newton's method and the conjugate gradient method on Riemannian manifolds, are presented and shown to possess quadratic and superlinear convergence, respectively. These methods are applied to several eigenvalue and singular value problems, which are posed as constrained optimization problems. ...

Motivation & Objective

• To develop efficient optimization algorithms for constrained problems on Riemannian manifolds, particularly in adaptive signal processing.
• To generalize classical optimization techniques—Newton’s method and conjugate gradient—onto Riemannian manifolds such as spheres and Stiefel manifolds.
• To solve the subspace tracking problem in adaptive filtering by formulating it as a constrained optimization on the Stiefel manifold.
• To ensure orthonormality of estimated eigenvectors throughout iterative updates, preserving numerical stability.
• To evaluate the convergence and tracking performance of the proposed algorithms on time-varying matrices with discontinuous changes.

Proposed method

• Formulates the eigenvalue and singular value problems as constrained optimization tasks on Riemannian manifolds, particularly the sphere and Stiefel manifold.
• Applies Riemannian Newton’s method to the Rayleigh quotient on the sphere, showing cubic convergence and equivalence to Rayleigh quotient iteration.
• Develops a Riemannian conjugate gradient method on symmetric spaces, using geodesic flow and parallel transport to maintain manifold structure.
• Exploits the homogeneous space structure of the Stiefel manifold to derive an efficient algorithm requiring $O(nk^2)$ operations and $O(k)$ matrix-vector products per iteration.
• Uses the generalized Rayleigh quotient on the Stiefel manifold as the cost function for subspace tracking.
• Implements numerical experiments on time-varying symmetric matrices to test convergence and tracking behavior under sudden changes in subspace orientation and eigenvalues.

Experimental results

Research questions

• RQ1 Can Riemannian Newton’s method on the sphere achieve cubic convergence when optimizing the Rayleigh quotient, and how does it relate to Rayleigh quotient iteration?
• RQ2 Does the Riemannian conjugate gradient method on symmetric spaces yield superlinear convergence for computing $k$ extreme eigenvectors of a symmetric matrix?
• RQ3 How efficiently can the Riemannian conjugate gradient method be implemented on the Stiefel manifold for subspace tracking, and what is its computational complexity?
• RQ4 How does the algorithm perform in tracking subspaces undergoing abrupt changes, such as rotating to an orthogonal plane or transitioning from a maximum to a saddle point?
• RQ5 Can the algorithm maintain orthonormality of the estimated subspace basis throughout iterations, and what is the impact on numerical stability?

Key findings

• The Riemannian Newton method applied to the Rayleigh quotient on the sphere converges cubically, and Rayleigh quotient iteration is shown to be an efficient approximation of this method.
• The Riemannian conjugate gradient method applied to the Rayleigh quotient on the sphere yields a new algorithm for computing extreme eigenvectors with superlinear convergence and $O(n)$ operations per iteration.
• For the $k$-extreme eigenvector problem on the Stiefel manifold, the algorithm requires $O(nk^2)$ operations and $O(k)$ matrix-vector multiplications per step, with convergence to machine precision within 50 iterations on a 294-dimensional manifold.
• Numerical experiments show that the algorithm converges to machine accuracy in less than 20 steps when the conjugate gradient is reset after a discontinuous subspace change.
• In tracking scenarios involving rotating subspaces with fixed or changing eigenvalues, the algorithm achieves accurate eigenvalue estimates in fewer than 10–25 iterations, depending on the scenario.
• When the true maximum becomes a saddle point, the algorithm remains near the solution for approximately 15 iterations, indicating robustness in transient conditions.