[Paper Review] Distance-weighted Support Vector Machine

Xingye Qiao, Lingsong Zhang|arXiv (Cornell University)|Oct 11, 2013

Face and Expression Recognition23 references3 citations

TL;DR

This paper proposes the Distance-weighted Support Vector Machine (DWSVM), a hybrid linear classifier that combines the robustness of Distance-weighted Discrimination (DWD) against data-piling with the balanced performance of SVM in imbalanced data settings. By minimizing a novel loss function that integrates the Hinge loss (SVM) and DWD loss via an auxiliary hyperplane, DWSVM achieves Fisher consistency, asymptotic normality, and superior classification accuracy and interpretability in high-dimensional, low-sample-size and imbalanced data scenarios.

ABSTRACT

A novel linear classification method that possesses the merits of both the Support Vector Machine (SVM) and the Distance-weighted Discrimination (DWD) is proposed in this article. The proposed Distance-weighted Support Vector Machine method can be viewed as a hybrid of SVM and DWD that finds the classification direction by minimizing mainly the DWD loss, and determines the intercept term in the SVM manner. We show that our method inheres the merit of DWD, and hence, overcomes the data-piling and overfitting issue of SVM. On the other hand, the new method is not subject to imbalanced data issue which was a main advantage of SVM over DWD. It uses an unusual loss which combines the Hinge loss (of SVM) and the DWD loss through a trick of axillary hyperplane. Several theoretical properties, including Fisher consistency and asymptotic normality of the DWSVM solution are developed. We use some simulated examples to show that the new method can compete DWD and SVM on both classification performance and interpretability. A real data application further establishes the usefulness of our approach.

Motivation & Objective

• To address the data-piling phenomenon in high-dimensional, low-sample-size (HDLSS) data that plagues traditional SVM and leads to overfitting and unstable classification directions.
• To resolve the sensitivity of DWD to imbalanced class sizes, where the decision boundary is pushed toward the minority class, causing misclassification of all new observations as the majority class.
• To develop a unified linear classifier that inherits the robustness of DWD against data-piling and the balance of SVM in handling imbalanced data.
• To establish theoretical properties such as Fisher consistency and asymptotic normality for the proposed method under regularity conditions.
• To demonstrate through simulations and real data that DWSVM achieves competitive classification performance and improved interpretability compared to SVM and DWD.

Proposed method

• The DWSVM method minimizes a hybrid loss function that combines the Hinge loss from SVM and the DWD loss through an auxiliary hyperplane, balancing classification margin and distance to class centroids.
• The classification direction is determined by minimizing the DWD loss, ensuring robustness to data-piling in HDLSS settings.
• The intercept term is estimated in the SVM manner, preserving its insensitivity to class imbalance and avoiding the bias toward the majority class seen in pure DWD.
• The method employs a reparameterization using an auxiliary hyperplane to smoothly blend the two loss functions, enabling stable optimization and theoretical analysis.
• Theoretical analysis includes proving Fisher consistency and asymptotic normality of the DWSVM estimator under regularity conditions, establishing its statistical reliability.
• The optimization is conducted via a modified subgradient method, with convergence and consistency results derived using empirical process theory and convexity arguments.

Experimental results

Research questions

• RQ1 Can a linear classifier be designed that simultaneously overcomes the data-piling issue of SVM and the class imbalance sensitivity of DWD in high-dimensional settings?
• RQ2 Does the proposed hybrid loss function based on Hinge and DWD losses yield a classifier with better classification accuracy and stability than SVM and DWD in both simulated and real-world HDLSS data?
• RQ3 Is the DWSVM estimator Fisher consistent and asymptotically normal, ensuring its reliability as a statistical estimator in large samples?
• RQ4 How does the DWSVM method perform in terms of classification accuracy and interpretability when compared to SVM and DWD across varying degrees of class imbalance and dimensionality?
• RQ5 Can the theoretical properties of DWSVM, such as consistency and asymptotic normality, be rigorously established under standard regularity conditions?

Key findings

• DWSVM successfully mitigates the data-piling phenomenon observed in SVM, preserving a Gaussian-like projection pattern of data points along the classification direction, indicating reduced overfitting.
• The method maintains robustness to class imbalance, avoiding the tendency of DWD to push the decision boundary toward the minority class, thus ensuring balanced classification performance.
• In simulated HDLSS data with $ d = 300 $, DWSVM achieves classification accuracy comparable to both SVM and DWD, while maintaining better alignment with the Bayes rule direction.
• Theoretical analysis confirms that the DWSVM estimator is Fisher consistent and asymptotically normal, supporting its use as a reliable statistical classifier.
• Empirical results on real data demonstrate that DWSVM outperforms or matches SVM and DWD in classification accuracy while offering improved interpretability due to stable and well-aligned classification directions.
• The intercept term in DWSVM does not diverge to $-ty$ under extreme imbalance, as seen in SVM, confirming its robustness to class size disparities.