[Paper Review] Eigenvalue and Generalized Eigenvalue Problems: Tutorial

Benyamin Ghojogh, Fakhri Karray+1 more|arXiv (Cornell University)|Mar 25, 2019

Blind Source Separation Techniques8 references46 citations

TL;DR

A tutorial that introduces eigenvalue and generalized eigenvalue problems, their optimization formulations, solution methods, and machine learning examples like PCA and Fisher discriminant analysis.

ABSTRACT

This paper is a tutorial for eigenvalue and generalized eigenvalue problems. We first introduce eigenvalue problem, eigen-decomposition (spectral decomposition), and generalized eigenvalue problem. Then, we mention the optimization problems which yield to the eigenvalue and generalized eigenvalue problems. We also provide examples from machine learning, including principal component analysis, kernel supervised principal component analysis, and Fisher discriminant analysis, which result in eigenvalue and generalized eigenvalue problems. Finally, we introduce the solutions to both eigenvalue and generalized eigenvalue problems.

Motivation & Objective

• Explain the eigenvalue and generalized eigenvalue problems and their spectral decompositions.
• Show how common ML problems yield eigenvalue or generalized eigenvalue formulations.
• Present step-by-step optimization formulations and their corresponding solutions.
• Illustrate with machine learning examples including PCA, kernel SPCA, and Fisher discriminant analysis.

Proposed method

• Define the eigenvalue problem and spectral decomposition for symmetric and non-symmetric matrices.
• Define the generalized eigenvalue problem and the pencil (A, B) concept.
• Derive optimization forms that lead to eigenvalue problems (Forms 1–5) and generalized eigenvalue problems (Forms 1–3).
• Explain the Rayleigh-Ritz quotient and its role in optimization.
• Show how reconstruction error and projection relate to eigenvalue problems in PCA.
• Provide explicit solutions via determinant conditions and matrix transformations (e.g., C = B^{-1}A) for solving eigenpairs.
• Discuss rigorous vs quick-and-dirty methods for generalized eigenvalue problems.

Experimental results

Research questions

• RQ1 What are the precise definitions and properties of eigenvalue and generalized eigenvalue problems?
• RQ2 How do optimization formulations yield eigenvalue and generalized eigenvalue problems in practice?
• RQ3 What are representative machine learning problems that reduce to these eigenvalue problems (e.g., PCA, kernel SPCA, FDA)?

Key findings

• The paper presents eigenvalue and generalized eigenvalue problems with their matrix forms and spectral decompositions.
• It systematically derives multiple optimization forms that yield eigenvalue and generalized eigenvalue problems, including both single-vector and multi-vector (Φ) formulations.
• It relates optimization solutions to eigenpairs and discusses the ordering of eigenvalues under different optimization goals (maximize/minimize).
• It connects eigenvalue problems to PCA variants, kernel SPCA, and Fisher discriminant analysis through explicit formulations.
• It explains both quick-and-dirty and rigorous approaches for solving generalized eigenvalue problems and outlines how to diagonalize B and transform to an ordinary eigenproblem.
• It clarifies the relationship between eigenvalue problems and the generalized Rayleigh quotient.