[Paper Review] Large-Margin Softmax Loss for Convolutional Neural Networks

Weiyang Liu, Yandong Wen+2 more|arXiv (Cornell University)|Dec 7, 2016

Advanced Neural Network Applications29 references432 citations

TL;DR

The paper introduces Large-Margin Softmax (L-Softmax) loss, an angular-margin version of softmax, to learn more discriminative CNN features for classification and verification, showing consistent gains on MNIST, CIFAR, and LFW datasets.

ABSTRACT

Cross-entropy loss together with softmax is arguably one of the most common used supervision components in convolutional neural networks (CNNs). Despite its simplicity, popularity and excellent performance, the component does not explicitly encourage discriminative learning of features. In this paper, we propose a generalized large-margin softmax (L-Softmax) loss which explicitly encourages intra-class compactness and inter-class separability between learned features. Moreover, L-Softmax not only can adjust the desired margin but also can avoid overfitting. We also show that the L-Softmax loss can be optimized by typical stochastic gradient descent. Extensive experiments on four benchmark datasets demonstrate that the deeply-learned features with L-softmax loss become more discriminative, hence significantly boosting the performance on a variety of visual classification and verification tasks.

Motivation & Objective

• Motivate the need for more discriminative features beyond standard softmax in CNNs.
• Propose a generalized large-margin softmax loss that enforces intra-class compactness and inter-class separability in angular space.
• Show that L-Softmax can be optimized with standard SGD and can be a drop-in replacement for softmax.
• Demonstrate improvements in image classification and face verification through extensive experiments.

Proposed method

• Represent class scores as f_j = W_j^T x_i and interpret as f_j = ||W_j|| ||x_i|| cos(theta_j).
• Define a margin by replacing cos(theta_yi) with psi(theta_yi) that encodes an angular margin m (ψ(θ) = cos(mθ) for 0 ≤ θ ≤ π/m, extended monotonically beyond).
• Use a tractable ψ(θ) formulation to enable forward/backward propagation (e.g., ψ(θ) = (-1)^k cos(mθ) - 2k for θ ∈ [kπ/m, (k+1)π/m]).
• Derive gradients with respect to x_i and W_yi to enable SGD updates (with special handling for m ≥ 2 via polynomial expansions).
• Provide a practical look-up approach for cos(θ_yi) and outline forward/backward computations for m=2 as a concrete example.

Experimental results

Research questions

• RQ1 Can an angular margin in the softmax objective improve intra-class compactness and inter-class separability in CNN features?
• RQ2 Does L-Softmax provide tangible gains for both visual classification and verification tasks across standard benchmarks?
• RQ3 Is L-Softmax compatible with standard SGD training and existing CNN architectures without excessive overfitting?
• RQ4 How does the margin parameter m affect discriminability and learning difficulty?

Key findings

• L-Softmax yields more discriminative features with larger angular margins, improving class separability in experiments.
• On MNIST, L-Softmax with m=2/3/4 achieves lower error rates than softmax and several baselines (e.g., 0.40% softmax vs. 0.31% for m=3 on MNIST).
• On CIFAR-10 without augmentation, L-Softmax with m=2/4 improves CIFAR-10 error to 7.58% (m=4) from 9.05% with softmax; with augmentation, CIFAR-10+ drops to 5.92% (m=4).
• On CIFAR-100, L-Softmax reduces error to 29.53% (m=4) versus 32.74% for softmax and other baselines.
• On LFW face verification, L-Softmax achieves higher accuracy (e.g., 98.71% with m=4) using CASIA-WebFace for training and standard evaluation protocol.
• The method can mitigate overfitting and scales with network capacity, as larger models further improve performance under L-Softmax.