[Paper Review] Deep Learning using Linear Support Vector Machines

Yichuan Tang|arXiv (Cornell University)|Jun 2, 2013

Anomaly Detection Techniques and Applications19 references741 citations

TL;DR

This paper proposes replacing the softmax layer in deep neural networks with a linear Support Vector Machine (SVM) to improve classification performance. By using the differentiable L2-SVM loss instead of cross-entropy, the model achieves better generalization through superior regularization, resulting in state-of-the-art error rates of 0.87% on MNIST and 11.9% on CIFAR-10, outperforming standard softmax-based networks.

ABSTRACT

Recently, fully-connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification, natural language processing, and bioinformatics. For classification tasks, most of these "deep learning" models employ the softmax activation function for prediction and minimize cross-entropy loss. In this paper, we demonstrate a small but consistent advantage of replacing the softmax layer with a linear support vector machine. Learning minimizes a margin-based loss instead of the cross-entropy loss. While there have been various combinations of neural nets and SVMs in prior art, our results using L2-SVMs show that by simply replacing softmax with linear SVMs gives significant gains on popular deep learning datasets MNIST, CIFAR-10, and the ICML 2013 Representation Learning Workshop's face expression recognition challenge.

Motivation & Objective

• To investigate whether replacing the softmax layer with a linear SVM improves classification performance in deep learning models.
• To evaluate the impact of using margin-based L2-SVM loss instead of cross-entropy loss on generalization and robustness.
• To determine whether performance gains stem from the objective function or improved optimization.
• To demonstrate the effectiveness of end-to-end training with L2-SVM in deep architectures without requiring feature fine-tuning or multi-stage pipelines.

Proposed method

• Replace the final softmax layer in deep neural networks with a linear L2-SVM layer that minimizes a differentiable, margin-based hinge loss.
• Optimize the primal form of the L2-SVM using stochastic gradient descent, enabling backpropagation through the entire network.
• Use the squared hinge loss (L2-SVM) to penalize misclassified examples more heavily than the standard hinge loss (L1-SVM).
• Train the full network end-to-end, allowing lower-layer features to be jointly optimized with the SVM objective.
• Apply data augmentation (random horizontal flips and jittering) and input-level Gaussian noise (σ = 1.0, decayed to 0) to improve generalization.
• Tune hyperparameters (learning rate, weight decay, SVM C) separately for each model using validation sets.

Experimental results

Research questions

• RQ1 Does replacing the softmax layer with a linear SVM lead to improved test accuracy on standard deep learning benchmarks?
• RQ2 Is the performance gain due to the L2-SVM's objective function or better optimization capabilities?
• RQ3 How does the L2-SVM loss compare to cross-entropy in terms of regularization and generalization?
• RQ4 Can end-to-end training of deep networks with L2-SVM layers achieve state-of-the-art results without unsupervised pretraining or complex architectures?

Key findings

• The DLSVM model achieved a test error of 0.87% on MNIST, outperforming the softmax baseline of 0.99% under identical training conditions.
• On CIFAR-10, the DLSVM model achieved 11.9% test error, compared to 14.0% for the softmax model, demonstrating consistent improvement.
• The L2-SVM objective led to significantly lower hinge loss (0.313) compared to the cross-entropy loss (0.353) in the DLSVM model, indicating better margin learning.
• When initialized with DLSVM weights, the softmax model's error increased to 14.0% after further training, suggesting the DLSVM objective is more effective for generalization.
• The performance gain is primarily attributed to the regularization properties of the L2-SVM loss rather than optimization advantages.
• Visual inspection of learned filters showed that the DLSVM-trained network produced more textured filters, suggesting differences in feature learning.