[Paper Review] An Architecture Combining Convolutional Neural Network (CNN) and Support Vector Machine (SVM) for Image Classification

Abien Fred Agarap|arXiv (Cornell University)|Dec 10, 2017

Neural Networks and Applications11 references143 citations

TL;DR

The paper implements a CNN-SVM architecture for image classification and compares it to CNN-Softmax on MNIST and Fashion-MNIST, reporting similar or slightly different accuracies.

ABSTRACT

Convolutional neural networks (CNNs) are similar to "ordinary" neural networks in the sense that they are made up of hidden layers consisting of neurons with "learnable" parameters. These neurons receive inputs, performs a dot product, and then follows it with a non-linearity. The whole network expresses the mapping between raw image pixels and their class scores. Conventionally, the Softmax function is the classifier used at the last layer of this network. However, there have been studies (Alalshekmubarak and Smith, 2013; Agarap, 2017; Tang, 2013) conducted to challenge this norm. The cited studies introduce the usage of linear support vector machine (SVM) in an artificial neural network architecture. This project is yet another take on the subject, and is inspired by (Tang, 2013). Empirical data has shown that the CNN-SVM model was able to achieve a test accuracy of ~99.04% using the MNIST dataset (LeCun, Cortes, and Burges, 2010). On the other hand, the CNN-Softmax was able to achieve a test accuracy of ~99.23% using the same dataset. Both models were also tested on the recently-published Fashion-MNIST dataset (Xiao, Rasul, and Vollgraf, 2017), which is suppose to be a more difficult image classification dataset than MNIST (Zalandoresearch, 2017). This proved to be the case as CNN-SVM reached a test accuracy of ~90.72%, while the CNN-Softmax reached a test accuracy of ~91.86%. The said results may be improved if data preprocessing techniques were employed on the datasets, and if the base CNN model was a relatively more sophisticated than the one used in this study.

Motivation & Objective

• Motivate exploration of SVM as an alternative classifier in CNN architectures.
• Evaluate CNN-SVM performance against CNN-Softmax on standard benchmarks MNIST and Fashion-MNIST.
• Analyze training dynamics and final test accuracy without data preprocessing.
• Discuss implications and limitations relative to previous work (Tang, 2013).

Proposed method

• Use a base CNN with two convolutional layers followed by fully connected layers and dropout.
• Replace the final softmax classifier with an L2-SVM loss in the last layer and train with Adam optimization.
• Compare two setups: CNN-Softmax and CNN-SVM on MNIST and Fashion-MNIST without explicit preprocessing.
• Report training accuracy, training loss, and test accuracy after 10,000 steps.
• Hyper-parameters: batch size 128, dropout 0.5, learning rate 1e-3, SVM C = 1 for CNN-SVM.
• Code available at: https://github.com/AFAgarap/cnn-svm

Experimental results

Research questions

• RQ1 Does CNN-SVM achieve comparable or better test accuracy than CNN-Softmax on MNIST and Fashion-MNIST?
• RQ2 How does removing data preprocessing influence CNN-SVM performance on these datasets?
• RQ3 What are the training dynamics (accuracy and loss) for CNN-SVM versus CNN-Softmax under the same architecture and training regime?
• RQ4 Are results consistent with prior work suggesting CNN-SVM can be competitive with softmax-based classifiers?

Key findings

• On MNIST, CNN-Softmax slightly outperforms CNN-SVM in test accuracy (99.23% vs 99.04%).
• On Fashion-MNIST, CNN-Softmax also outperforms CNN-SVM (91.86% vs 90.72%).
• Training times for both models on MNIST and Fashion-MNIST were around 4 minutes per run (approximate from training steps).
• CNN-SVM achieves competitive results but does not surpass CNN-Softmax on the datasets tested without data preprocessing.
• The study notes that more sophisticated base CNN models and preprocessing might reproduce or improve results reported in related work (Tang, 2013).