[Paper Review] Behavioral analysis of support vector machine classifier with Gaussian kernel and imbalanced data

Alaa Tharwat|arXiv (Cornell University)|Jul 9, 2020

Imbalanced Data Classification Techniques18 references4 citations

TL;DR

This paper proposes a novel two-stage search algorithm for optimizing SVM parameters with Gaussian kernels on imbalanced datasets, reducing computational time by searching in two one-dimensional spaces instead of one two-dimensional space. The method improves efficiency and effectiveness by leveraging data analysis to narrow the kernel parameter search space, demonstrating faster and more accurate classification than traditional grid search.

ABSTRACT

The parameters of support vector machines (SVMs) such as the penalty parameter and the kernel parameters have a great impact on the classification accuracy and the complexity of the SVM model. Therefore, the model selection in SVM involves the tuning of these parameters. However, these parameters are usually tuned and used as a black box, without understanding the mathematical background or internal details. In this paper, the behavior of the SVM classification model is analyzed when these parameters take different values with balanced and imbalanced data. This analysis including visualization, mathematical and geometrical interpretations and illustrative numerical examples with the aim of providing the basics of the Gaussian and linear kernel functions with SVM. From this analysis, we proposed a novel search algorithm. In this algorithm, we search for the optimal SVM parameters into two one-dimensional spaces instead of searching into one two-dimensional space. This reduces the computational time significantly. Moreover, in our algorithm, from the analysis of the data, the range of kernel function can be expected. This also reduces the search space and hence reduces the required computational time. Different experiments were conducted to evaluate our search algorithm using different balanced and imbalanced datasets. The results demonstrated how the proposed strategy is fast and effective than other searching strategies.

Motivation & Objective

• To analyze the behavior of SVM with Gaussian and linear kernels under varying penalty and kernel parameters on both balanced and imbalanced datasets.
• To provide intuitive understanding through visualizations, mathematical interpretations, and numerical examples of SVM performance.
• To develop a more efficient parameter search strategy that reduces computational cost while maintaining or improving classification accuracy.
• To address the challenge of imbalanced data in SVM by tailoring parameter search to data characteristics.

Proposed method

• The method decomposes the two-dimensional parameter search (C, γ) into two sequential one-dimensional searches, significantly reducing computational complexity.
• It introduces a data-driven estimation of the kernel parameter (γ) range based on data distribution, narrowing the search space.
• The algorithm uses Lagrangian optimization and duality theory to solve the primal and dual SVM problems, ensuring theoretical consistency.
• It applies the kernel trick to map data into a higher-dimensional space where nonlinearly separable data become linearly separable.
• The approach is validated using both synthetic and real-world balanced and imbalanced datasets.
• The dual problem is solved to find the optimal Lagrange multipliers, which are used to compute the decision boundary and support vectors.

Experimental results

Research questions

• RQ1 How do varying values of the penalty parameter C and kernel parameter γ affect SVM performance on imbalanced datasets?
• RQ2 What is the geometric and mathematical behavior of SVM with Gaussian and linear kernels under different parameter settings?
• RQ3 Can a two-stage one-dimensional search strategy outperform traditional two-dimensional grid search in terms of speed and accuracy?
• RQ4 How can data characteristics be used to estimate a meaningful range for the kernel parameter γ to reduce search space?
• RQ5 What impact does data imbalance have on the number of support vectors, margin size, and classification error?

Key findings

• The proposed algorithm reduces computational time significantly by searching in two one-dimensional spaces instead of a single two-dimensional space.
• The method achieves higher classification accuracy than grid search on multiple imbalanced datasets, particularly in identifying minority class samples.
• Data-driven estimation of the kernel parameter range reduces the search space and improves efficiency without sacrificing accuracy.
• The analysis reveals that multiple lines exist in the 'good region' of parameter space, justifying the need for a broader search than previously assumed.
• The dual problem solution confirms strong duality with zero duality gap, validating the optimality of the computed solution.
• Visualizations and numerical examples demonstrate how parameter changes affect margin size, support vectors, and decision boundaries.