[Paper Review] A Simple Algorithm for Maximum Margin Classification, Revisited

Sariel Har-Peled|arXiv (Cornell University)|Jul 6, 2015

Machine Learning and Algorithms1 references3 citations

TL;DR

This paper revisits a simple iterative algorithm for computing a linear maximum margin classifier in high-dimensional spaces. Using a labeling and counterexample oracle, the algorithm incrementally refines a separating slab between two classes, achieving a (1−ε)-approximation of the optimal margin in O((Δ/γ)²/ε²) iterations, with O(Idxn) total time, where Δ is the point set diameter and γ is the true margin.

ABSTRACT

In this note, we revisit the algorithm of Har-Peled et. al. [HRZ07] for computing a linear maximum margin classifier. Our presentation is self contained, and the algorithm itself is slightly simpler than the original algorithm. The algorithm itself is a simple Perceptron like iterative algorithm. For more details and background, the reader is referred to the original paper.

Motivation & Objective

• To present a self-contained, simplified version of the maximum margin classification algorithm from Har-Peled et al. (2007).
• To minimize the number of expensive oracle queries (labeling and counterexample) in active learning settings.
• To achieve a (1−ε)-approximation of the maximum margin classifier for linearly separable data in ℝᵈ.
• To analyze the convergence rate and iteration complexity of the algorithm under margin and diameter constraints.

Proposed method

• The algorithm uses a slab defined by two points, bᵢ and wᵢ, forming a (1−ε)-shrunk slab around their line segment.
• In each iteration, it checks if the current slab separates all points using the counterexample oracle; if not, it selects the closest point to the slab's middle hyperplane.
• It queries the labeling oracle for the label of the closest misclassified point and updates one of the slab-defining points via projection to improve separation.
• The algorithm maintains a decreasing sequence of slab widths ℓᵢ = ||bᵢ − wᵢ||, with each update reducing the width by a factor bounded by 1 − (εℓᵢ/(4Δ))².
• Convergence is analyzed by showing that each epoch (where ℓᵢ halves) takes O((2ʲ/ε)²) iterations, with the final epoch dominating the total complexity.
• The total number of iterations is bounded by O((Δ/γ)²/ε²), ensuring efficient convergence to a (1−ε)-approximate maximum margin classifier.

Experimental results

Research questions

• RQ1 Can a simpler, self-contained algorithm achieve (1−ε)-approximate maximum margin classification with minimal oracle queries?
• RQ2 How does the iteration count scale with the data diameter Δ and margin γ in a margin-based active learning setting?
• RQ3 What is the theoretical convergence rate of an iterative slab refinement algorithm using only labeling and counterexample oracles?
• RQ4 Can the algorithm be interpreted as an approximation to the underlying quadratic program for maximum margin separation?
• RQ5 How does the geometric structure of the slab updates ensure monotonic improvement toward the optimal margin?

Key findings

• The algorithm computes a (1−ε)-approximate maximum margin classifier in O((Δ/γ)²/ε²) iterations, with O(Idn) total time complexity.
• Each iteration involves at most one call to the labeling oracle and one to the counterexample oracle, minimizing expensive queries.
• The number of iterations is dominated by the final epoch, which begins when the slab width drops below 2γ and ends at γ, taking O((Δ/γ)²/ε²) steps.
• The width ℓᵢ of the slab decreases geometrically over epochs, with each epoch lasting O((2ʲ/ε)²) iterations for the j-th epoch.
• The algorithm guarantees that after I = O((Δ/γ)²/ε²) iterations, the resulting slab has width at least (1−ε)γ and correctly separates all points.
• The method provides a practical and efficient approximation to the optimal margin classifier, interpretable as a first-order method for solving the associated quadratic program.