[Paper Review] Support Vector Machine Classification with Indefinite Kernels

Ronny Luss, Alexandre d’Aspremont|ArXiv.org|Apr 1, 2008

Sparse and Compressive Sensing Techniques19 references17 citations

TL;DR

This paper proposes a convex optimization framework for support vector machine (SVM) classification using indefinite kernels—kernels that are not positive semidefinite—by simultaneously learning support vector weights and a proxy positive semidefinite kernel matrix. The method treats the indefinite kernel as a noisy observation of a true Mercer kernel, penalizing the deviation from it, which transforms the nonconvex problem into a convex one solvable via projected gradient or analytic center cutting plane methods with guaranteed convergence and efficient scalability on large datasets.

ABSTRACT

We propose a method for support vector machine classification using indefinite kernels. Instead of directly minimizing or stabilizing a nonconvex loss function, our algorithm simultaneously computes support vectors and a proxy kernel matrix used in forming the loss. This can be interpreted as a penalized kernel learning problem where indefinite kernel matrices are treated as a noisy observations of a true Mercer kernel. Our formulation keeps the problem convex and relatively large problems can be solved efficiently using the projected gradient or analytic center cutting plane methods. We compare the performance of our technique with other methods on several classic data sets.

Motivation & Objective

• To address the challenge of using indefinite kernels in SVM classification, which are empirically effective but violate Mercer's condition due to lack of positive semidefiniteness.
• To develop a method that avoids nonconvex optimization or heuristic kernel transformations, which may compromise performance or interpretation.
• To frame indefinite kernel learning as a penalized kernel learning problem where the proxy kernel is a low-rank approximation minimizing deviation from the original indefinite kernel.
• To ensure computational efficiency and convergence guarantees by reformulating the problem as a convex optimization task with explicit solution for the kernel learning component.
• To extend the framework to other kernel methods such as support vector regression and one-class SVMs.

Proposed method

• The method introduces a proxy kernel matrix as a convex relaxation of the indefinite kernel, minimizing the Frobenius norm of the difference between the proxy and original kernel.
• It formulates the SVM problem as a penalized optimization problem where the loss is based on the proxy kernel, ensuring convexity and tractability.
• The kernel learning subproblem is solved explicitly, decoupling it from the SVM dual optimization, which reduces computational complexity compared to standard kernel learning.
• Two algorithms are employed: projected gradient method and analytic center cutting plane method (ACCPM), both with provable convergence and complexity bounds.
• The approach treats the indefinite kernel as a noisy observation of a true positive semidefinite kernel, modeling uncertainty in kernel input.
• The framework allows for efficient large-scale training by leveraging rank-one updates in eigenvalue decomposition during iterative optimization.

Experimental results

Research questions

• RQ1 Can a convex optimization framework be designed for SVM classification with indefinite kernels, avoiding nonconvexity and heuristic kernel fixes?
• RQ2 How can a proxy kernel be learned explicitly to approximate an indefinite kernel while preserving classification performance?
• RQ3 What are the convergence and computational complexity properties of algorithms solving this reformulated convex problem?
• RQ4 How does the performance of this method compare to existing approaches like spectral kernel transformation or pE-space embedding in terms of accuracy and efficiency?
• RQ5 Can this framework be extended to other kernel methods such as support vector regression and one-class SVM?

Key findings

• The proposed method transforms a nonconvex SVM problem with indefinite kernels into a convex optimization problem, enabling global convergence and efficient solution.
• The kernel learning component is solved explicitly, reducing computational overhead compared to iterative kernel learning methods.
• Projected gradient and ACCPM algorithms both converge to optimal solutions, with ACCPM achieving higher precision linearly, while projected gradient converges faster initially but stalls at higher precision.
• On the USPS-SS-3-5 dataset, the projected gradient method outperforms SIQCLP in total runtime and is more efficient than ACCPM due to lower per-iteration cost, despite higher iteration counts for larger problems.
• The duality gap computed via the projected gradient method provides a reliable stopping criterion, though SIQCLP yields a tighter gap at the cost of significantly higher per-iteration complexity.
• The framework provides a principled interpretation for heuristic methods used in indefinite kernel SVMs, showing they implicitly learn a proxy kernel under uncertainty.