[Paper Review] Universum Prescription: Regularization using Unlabeled Data

Xiang Zhang, Yann LeCun|arXiv (Cornell University)|Nov 11, 2015

Advanced Neural Network Applications57 references3 citations

TL;DR

This paper proposes Universum prescription—a simple yet effective regularization technique that improves deep learning generalization by assigning 'none of the above' labels to unlabeled data. By treating unlabeled samples as belonging to a universal set outside the supervised classes, the method reduces generalization error across CIFAR-10, CIFAR-100, STL-10, and ImageNet, with the dustbin class variant outperforming others due to its learnable parameterization.

ABSTRACT

This paper shows that simply prescribing "none of the above" labels to unlabeled data has a beneficial regularization effect to supervised learning. We call it universum prescription by the fact that the prescribed labels cannot be one of the supervised labels. In spite of its simplicity, universum prescription obtained competitive results in training deep convolutional networks for CIFAR-10, CIFAR-100, STL-10 and ImageNet datasets. A qualitative justification of these approaches using Rademacher complexity is presented. The effect of a regularization parameter -- probability of sampling from unlabeled data -- is also studied empirically.

Motivation & Objective

• To investigate whether assigning 'none of the above' labels to unlabeled data can serve as an effective regularization technique in deep learning.
• To evaluate the impact of different 'none of the above' prescription strategies—uniform, dustbin class, and background class—on model generalization.
• To theoretically justify the method using Rademacher complexity and empirically validate its effectiveness across multiple benchmark datasets.
• To analyze the effect of the regularization parameter, i.e., the probability of sampling from unlabeled data, on model performance and stability.

Proposed method

• The method prescribes unlabeled data with agnostic targets: uniform distribution over all classes, a dedicated 'dustbin' class, or a thresholded background class.
• It assumes that labeled classes are negligible in the unlabeled distribution, ensuring that unlabeled samples do not belong to any supervised class.
• The loss function is based on negative log-likelihood, and during training, unlabeled samples are randomly sampled with probability p to update the model.
• The dustbin class extends the output layer to include an extra class, allowing the model to learn a dedicated representation for unlabeled data.
• The background class uses a constant threshold to avoid parameterization, providing a simpler but less adaptive alternative.
• Theoretical justification is provided via Rademacher complexity, showing that the generalization error is bounded by the joint problem including unlabeled data.

Experimental results

Research questions

• RQ1 Can prescribing 'none of the above' labels to unlabeled data reduce the generalization gap in deep neural networks?
• RQ2 How do different prescription strategies—uniform, dustbin class, and background class—affect model performance and generalization?
• RQ3 What is the optimal value of the regularization parameter p, the probability of sampling from unlabeled data?
• RQ4 Does Universum prescription improve model stability and variance, especially with large amounts of unlabeled data?
• RQ5 How does Universum prescription compare in effect to dropout and other standard regularization techniques?

Key findings

• The dustbin class method consistently achieved the best test error across all datasets, outperforming uniform and background class prescriptions.
• On CIFAR-10, CIFAR-100, and STL-10, Universum prescription reduced the generalization gap, with the effect being more pronounced when using large amounts of unlabeled data.
• Increasing the regularization parameter p reduced the generalization gap up to a point, after which training collapse occurred due to over-reliance on unlabeled data.
• The method’s regularization effect was comparable to that of dropout, with no significant additional benefit when both were used together.
• Empirical results showed that the variance of training and test errors decreased with larger unlabeled datasets, indicating improved model stability.
• Theoretical analysis using Rademacher complexity supports the claim that adding unlabeled data with Universum prescription improves generalization bounds.