[Paper Review] A statistical theory of semi-supervised learning

Laurence Aitchison|arXiv (Cornell University)|Aug 13, 2020

Cell Image Analysis Techniques8 references4 citations

TL;DR

This paper proposes a generative model of data curation in standard benchmarks like CIFAR-10, showing that semi-supervised learning methods—including entropy minimization, pseudo-labelling, and FixMatch—emerge naturally as variational lower bounds on the log-likelihood under this model. The key contribution is a principled statistical foundation for semi-supervised learning, enabling integration with Bayesian methods.

ABSTRACT

We currently lack a solid statistical understanding of semi-supervised learning methods, instead treating them as a collection of highly effective tricks. This precludes the principled combination e.g. of Bayesian methods and semi-supervised learning, as semi-supervised learning objectives are not currently formulated as likelihoods for an underlying generative model of the data. Here, we note that standard image benchmark datasets such as CIFAR-10 are carefully curated, and we provide a generative model describing the curation process. Under this generative model, several state-of-the-art semi-supervised learning techniques, including entropy minimization, pseudo-labelling and the FixMatch family emerge naturally as variational lower-bounds on the log-likelihood.

Motivation & Objective

• To address the lack of a principled statistical foundation for semi-supervised learning, which is currently treated as a set of heuristic tricks.
• To formalize the data curation process in standard benchmarks like CIFAR-10 as a generative model.
• To show that established semi-supervised learning techniques arise naturally as variational lower bounds under this model.
• To enable the principled integration of semi-supervised learning with Bayesian inference by formulating it as a likelihood-based generative model.

Proposed method

• Propose a generative model that describes how standard image datasets like CIFAR-10 are curated, incorporating selection and labeling processes.
• Formulate the semi-supervised learning objective as a variational lower bound on the log-likelihood of the observed data under the generative model.
• Demonstrate that entropy minimization, pseudo-labelling, and FixMatch correspond to specific approximations of this lower bound.
• Use the generative model to derive the connection between semi-supervised objectives and probabilistic inference under uncertainty.
• Show that optimizing the variational lower bound yields state-of-the-art performance in semi-supervised learning settings.

Experimental results

Research questions

• RQ1 How can semi-supervised learning methods be formally grounded in a statistical generative model?
• RQ2 What underlying data curation process in benchmarks like CIFAR-10 explains the success of current semi-supervised learning techniques?
• RQ3 Can entropy minimization and pseudo-labelling be derived as approximations of a single principled objective?
• RQ4 To what extent do methods like FixMatch emerge naturally from a unified probabilistic framework?

Key findings

• Semi-supervised learning methods such as entropy minimization and pseudo-labelling are shown to be natural approximations of a variational lower bound on the log-likelihood under a generative model of data curation.
• The FixMatch family of methods also emerges as a specific instance of this variational lower bound, providing a unified statistical interpretation.
• The generative model of data curation explains why these methods are effective on benchmarks like CIFAR-10, where data are carefully curated.
• The framework enables the principled combination of semi-supervised learning with Bayesian inference, as the method is now based on a likelihood model.