[Paper Review] The Pessimistic Limits and Possibilities of Margin-based Losses in Semi-supervised Learning

Jesse H. Krijthe, Marco Loog|arXiv (Cornell University)|Dec 28, 2016

Machine Learning and Data Classification4 citations

TL;DR

This paper investigates the theoretical limits of semi-supervised learning using convex margin-based surrogate losses. It proves that for decreasing convex margin-based losses (e.g., hinge loss), no semi-supervised method can guarantee performance improvement over supervised learning for all possible labelings of unlabeled data. In contrast, for increasing losses (e.g., quadratic loss), safe improvements are possible under specific conditions, revealing a fundamental dichotomy in the feasibility of pessimistically safe semi-supervised learning.

ABSTRACT

Consider a classification problem where we have both labeled and unlabeled data available. We show that for linear classifiers defined by convex margin-based surrogate losses that are decreasing, it is impossible to construct any semi-supervised approach that is able to guarantee an improvement over the supervised classifier measured by this surrogate loss on the labeled and unlabeled data. For convex margin-based loss functions that also increase, we demonstrate safe improvements are possible.

Motivation & Objective

• To determine whether semi-supervised learning can be made strictly safe—guaranteeing no performance degradation compared to supervised learning—under worst-case labelings of unlabeled data.
• To analyze the role of convex margin-based surrogate losses in determining the feasibility of safe semi-supervised learning.
• To distinguish between losses that are decreasing versus increasing in the margin, and assess their implications for performance guarantees.
• To formalize a pessimistic, finite-sample criterion for safe semi-supervised learning that holds for all possible labelings of unlabeled data.
• To challenge assumptions in prior work by avoiding untestable conditions such as low-density separators or distributional assumptions.

Proposed method

• Formalizes a transductive setting where performance is measured on a fixed set of labeled and unlabeled data, using the same surrogate loss for both supervised and semi-supervised classifiers.
• Defines 'safe' semi-supervised learning as a method that never performs worse than the supervised classifier on the combined data, under any possible labeling of the unlabeled set.
• Analyzes convex margin-based surrogate losses, focusing on their monotonicity: whether they are decreasing or increasing in the margin.
• Uses a partial assignment argument to show that for decreasing losses, any deviation from the supervised solution can be reversed by some labeling, making guaranteed improvement impossible.
• Derives conditions under which increasing losses (e.g., quadratic loss) allow for safe improvements by ensuring the semi-supervised objective can strictly dominate the supervised one.
• Fixes the regularization parameter across supervised and semi-supervised models to isolate the effect of unlabeled data on performance.

Experimental results

Research questions

• RQ1 Can any semi-supervised learning method guarantee performance improvement over supervised learning for all possible labelings of unlabeled data, under a pessimistic, finite-sample criterion?
• RQ2 Why are decreasing convex margin-based losses (e.g., hinge loss) fundamentally incapable of supporting safe semi-supervised improvements?
• RQ3 Under what conditions can increasing convex margin-based losses (e.g., quadratic loss) support safe semi-supervised improvements?
• RQ4 How does the use of the same surrogate loss for both supervised and semi-supervised learning affect the validity of performance comparisons?
• RQ5 To what extent do prior assumptions—such as low-density separators or distributional constraints—limit the generality of safe semi-supervised learning guarantees?

Key findings

• For convex margin-based surrogate losses that are decreasing, no semi-supervised method can guarantee performance improvement over the supervised classifier for all possible labelings of the unlabeled data.
• The impossibility arises because any semi-supervised solution that deviates from the supervised solution can be reversed by a labeling of the unlabeled data that recovers the supervised objective value.
• For losses that are increasing over some interval (e.g., quadratic loss), safe improvements are possible under specific conditions, demonstrating that the monotonicity of the loss is a key determinant of feasibility.
• The results contradict the claim in Ben-David et al. (2008) that improvements are impossible without strong distributional assumptions, by showing that safe improvements are possible even with finite VC-dimension when the loss is increasing.
• The study reveals that surrogate loss monotonicity—specifically whether it increases or decreases with margin—determines whether pessimistically safe semi-supervised learning is theoretically possible.
• The findings imply that while many standard methods (e.g., SVMs with hinge loss) cannot be made strictly safe in this sense, others (e.g., least-squares classifiers with quadratic loss) may be viable under the same criterion.