[Paper Review] On $\ell_p$-Support Vector Machines and Multidimensional Kernels

Víctor Blanco, Justo Puerto+1 more|arXiv (Cornell University)|Nov 28, 2017

Tensor decomposition and applications20 references3 citations

TL;DR

This paper extends Support Vector Machines to $β$-norms for $p > 1$, formulating primal and dual problems as second-order cone and polynomial optimization. It introduces multidimensional kernels—based on homogeneous polynomials and real tensors—that generalize the kernel trick, enabling efficient classification via moment-SOCP and Schauder space approximations, with empirical results showing competitive or superior accuracy over $β$-SVM, especially with $\ell_{4/3}$-norm.

ABSTRACT

In this paper, we extend the methodology developed for Support Vector Machines (SVM) using $\ell_2$-norm ($\ell_2$-SVM) to the more general case of $\ell_p$-norms with $p\ge 1$ ($\ell_p$-SVM). The resulting primal and dual problems are formulated as mathematical programming problems; namely, in the primal case, as a second order cone optimization problem and in the dual case, as a polynomial optimization problem involving homogeneous polynomials. Scalability of the primal problem is obtained via general transformations based on the expansion of functionals in Schauder spaces. The concept of Kernel function, widely applied in $\ell_2$-SVM, is extended to the more general case by defining a new operator called multidimensional Kernel. This object gives rise to reformulations of dual problems, in a transformed space of the original data, which are solved by a moment-sdp based approach. The results of some computational experiments on real-world datasets are presented showing rather good behavior in terms of standard indicators such a extit{accuracy index} and its ability to classify new data.

Motivation & Objective

• To develop a unifying theoretical framework for $β$-norm Support Vector Machines ($β$-SVM) with $p > 1$, generalizing the standard $β$-SVM.
• To extend the kernel trick—central to $β$-SVM—beyond the Euclidean norm by introducing multidimensional kernel functions.
• To provide scalable primal and dual formulations using second-order cone programming and polynomial optimization.
• To enable efficient solution via moment-SOCP and Schauder space-based functional approximations without explicit data mapping.
• To empirically evaluate the performance of $β$-SVM across different norms and kernel types on real-world datasets.

Proposed method

• Formulates $β$-SVM as a second-order cone program (SOCP) in the primal, enabling scalable solution via conic optimization.
• Reformulates the dual as a polynomial optimization problem involving homogeneous polynomials, enabling the use of moment-SOCP techniques.
• Introduces the concept of multidimensional kernel as a generalization of the standard kernel function, defined via symmetric real tensors of appropriate order and dimension.
• Establishes sufficient conditions for a symmetric real tensor to induce a valid multidimensional kernel function.
• Employs a moment-SOCP approach to solve the dual problem via a sequence of semidefinite programs converging to the optimal solution.
• Applies limited expansions in Schauder spaces to approximate functional transformations without explicit data mapping, preserving the kernel trick in the $β$-SVM context.

Experimental results

Research questions

• RQ1 How can the $β$-SVM framework be generalized to arbitrary $β$-norms with $p > 1$ while maintaining computational tractability?
• RQ2 What is the appropriate generalization of the kernel trick for $β$-SVM beyond the $β$-norm case?
• RQ3 Can multidimensional kernel functions based on homogeneous polynomials and real tensors effectively enable nonlinear classification in $β$-SVM?
• RQ4 How do different $β$-norms ($\ell_{4/3}$, $\ell_2$, etc.) compare in terms of classification accuracy and sparsity?
• RQ5 To what extent can the kernel trick be preserved in $β$-SVM without explicit data transformation, using functional approximation in Schauder spaces?

Key findings

• The $\ell_{4/3}$-norm consistently achieved the best or near-best test accuracy across multiple datasets, outperforming the standard $\ell_2$-SVM in several cases.
• For the cleveland and german credit datasets, non-linear transformations ($\eta = 4$ and $\eta = 3$) under $\widetilde{\Phi}[\eta]$ yielded higher test accuracy than linear ones, indicating benefit from nonlinearity.
• The $\ell_2$-SVM was solved fastest due to its direct quadratic formulation, but other norms like $\ell_{4/3}$ showed superior sparsity, with fewer non-zero $\omega$-coefficients.
• Overfitting was observed: perfect training accuracy ($100\%$) did not guarantee better test performance, especially for higher-degree transformations.
• The multidimensional kernel framework successfully enabled kernel-based classification in the $\ell_p$-SVM context, with convergence guarantees via moment-SOCP.
• The Schauder space-based approach allowed effective approximation of functional transformations without explicit data mapping, maintaining the kernel trick's efficiency.