[Paper Review] TrIK-SVM : an alternative decomposition for kernel methods in Krein spaces

Gaëlle Loosli|arXiv (Cornell University)|Feb 27, 2019

Neural Networks and Applications9 references4 citations

TL;DR

TrIK-SVM proposes a novel kernel decomposition for kernel methods in Kreĭn spaces that avoids eigen-decomposition, enabling sparse solutions without precomputing the full kernel matrix. It achieves performance competitive with KSVM while eliminating numerical instabilities and maintaining computational efficiency.

ABSTRACT

The proposed work aims at proposing a alternative kernel decomposition in the context of kernel machines with indefinite kernels. The original paper of KSVM (SVM in Kreǐn spaces) uses the eigen-decomposition, our proposition avoids this decompostion. We explain how it can help in designing an algorithm that won't require to compute the full kernel matrix. Finally we illustrate the good behavior of the proposed method compared to KSVM.

Motivation & Objective

• Address the practical limitations of KSVM, including high computational cost and loss of sparsity due to eigen-decomposition.
• Develop a kernel decomposition method that avoids full kernel matrix computation and preserves sparsity in the solution.
• Ensure numerical stability and robustness across varying hyperparameters, particularly λ.
• Enable efficient implementation using cache strategies by computing kernel terms on-the-fly.
• Demonstrate competitive performance against KSVM on synthetic datasets with varying degrees of overlap and separability.

Proposed method

• Proposes a TrIK (Trace-based Indefinite Kernel) decomposition that replaces eigen-decomposition with a trace-based formulation, avoiding the need to compute full spectral decompositions.
• Reformulates the optimization problem in a Hilbert space (H) by splitting the kernel into positive and negative parts: K = K₊ + K₋, where K₊ and K₋ are positive-definite kernels.
• Uses a modified dual optimization problem in the RKHS (H) with a modified kernel matrix G = G₊ + G₋, where G₊ and G₋ are derived from K₊ and K₋.
• Applies a projection mechanism to recover the solution in the original RKKS (K) via α_sv = P_sv,sv * α̃_sv, ensuring sparsity and stability.
• Employs a three-step algorithm: (1) solve in RKHS using modified kernel matrix, (2) check dual feasibility and project to admissible set, (3) verify primal optimality on support vectors.
• Utilizes on-the-fly kernel computation or caching strategies to avoid storing the full kernel matrix, enhancing memory efficiency.

Experimental results

Research questions

• RQ1 Can a kernel decomposition be designed that avoids eigen-decomposition while preserving the solution quality in indefinite kernel learning?
• RQ2 Does the proposed method maintain sparsity and numerical stability compared to KSVM, especially under varying λ values?
• RQ3 Can TrIK-SVM achieve competitive accuracy with KSVM without precomputing the full kernel matrix?
• RQ4 How does the choice of λ affect the solution quality and computational efficiency in TrIK-SVM?
• RQ5 Can the algorithm be efficiently implemented using cache strategies to reduce memory overhead?

Key findings

• TrIK-SVM produces solutions in the RKKS that are nearly identical across different λ values, as long as λ is below the smallest eigenvalue, demonstrating robustness.
• The method achieves training accuracy comparable to KSVM across 30 randomly generated datasets of increasing size (100 to 3000 examples), confirming competitive performance.
• Despite being implemented in Python without optimization, TrIK-SVM shows training times on a log-log scale that are comparable to KSVM, which uses highly optimized libsvm.
• The solution remains sparse because the algorithm computes only the support vector components, avoiding full kernel matrix storage.
• Numerical instabilities common in KSVM—especially in datasets with high point similarity—are significantly reduced in TrIK-SVM due to the avoidance of eigen-decomposition.
• The hyperparameter λ does not affect solution quality or training time as long as it is sufficiently negative, making the method parameter-free in practice.