[Paper Review] TV-SVM: Total Variation Support Vector Machine for Semi-Supervised Data Classification

Xavier Bresson, Ruiliang Zhang|arXiv (Cornell University)|Oct 2, 2012

Face and Expression Recognition9 references3 citations

TL;DR

This paper proposes TV-SVM, a novel semi-supervised classification framework that integrates total variation (TV) regularization with support vector machines (SVM) and reproducing kernel Hilbert spaces (RKHS). By leveraging TV's ability to preserve discontinuities and promote piecewise constant solutions, TV-SVM achieves significantly lower classification error than Laplacian-based methods, especially when labeled data are scarce, with error rates dropping to 1.75% on the USPS dataset using only 1 label per class.

ABSTRACT

We introduce semi-supervised data classification algorithms based on total variation (TV), Reproducing Kernel Hilbert Space (RKHS), support vector machine (SVM), Cheeger cut, labeled and unlabeled data points. We design binary and multi-class semi-supervised classification algorithms. We compare the TV-based classification algorithms with the related Laplacian-based algorithms, and show that TV classification perform significantly better when the number of labeled data is small.

Motivation & Objective

• To address the challenge of semi-supervised classification when labeled data are extremely limited.
• To improve generalization by incorporating total variation (TV) regularization into the SVM framework, promoting piecewise constant solutions on data graphs.
• To outperform existing Laplacian-based semi-supervised methods in low-label regimes.
• To develop efficient optimization algorithms for non-differentiable TV terms using augmented Lagrangian and ADMM-like splitting.
• To extend the TV-SVM framework to both binary and multi-class classification problems.

Proposed method

• Formulates a TV-regularized optimization problem combining kernel-based function estimation, SVM slack variables, and total variation on graph-structured data.
• Uses a two-splitting-variable augmented Lagrangian method to handle the non-differentiable TV term, enabling convergence via primal-dual optimization.
• Applies the primal-dual method to solve the TV subproblem with O(1/k²) convergence rate.
• Implements normalization and zero-mean constraints on the function estimates to stabilize training.
• For multi-class problems, uses a consensus constraint on soft labels and projects them onto the probability simplex.
• Employs a Cheeger cut-inspired formulation to further enhance class separation in multi-class settings.

Experimental results

Research questions

• RQ1 Can total variation regularization improve semi-supervised classification performance compared to Laplacian-based methods, especially with few labeled examples?
• RQ2 How does TV-SVM perform in comparison to Lap-SVM and Lap-RLS in low-label regimes?
• RQ3 Can the non-differentiable TV term be efficiently optimized in a kernel-based semi-supervised learning framework?
• RQ4 Does the TV-SVM framework generalize well to multi-class classification tasks?
• RQ5 Can the Cheeger-based formulation further improve classification accuracy by emphasizing intrinsic data structure?

Key findings

• TV-SVM achieves a test error of 1.75% on the USPS dataset with only 1 label per class, significantly outperforming Lap-SVM (49.95%) and Lap-RLS (20.06%).
• TV-RLS achieves 1.91% error with 10 labels per class, outperforming Lap-RLS (4.03%) and showing robustness in low-data regimes.
• TV-SVM maintains consistent performance across all label settings, with error rates below 1.85% even at 50 labels per class.
• The Cheeger-based SVM variant achieves 1.72% error with 10 labels per class, indicating that geometric structure-aware regularization enhances generalization.
• TV-based methods consistently outperform Laplacian-based counterparts when labeled data are scarce, demonstrating the advantage of TV in preserving sharp decision boundaries.
• The proposed ADMM-based optimization scheme effectively handles the non-differentiable TV term, enabling stable and accurate training on large-scale semi-supervised problems.