[論文レビュー] MixMatch: A Holistic Approach to Semi-Supervised Learning

David Berthelot, Nicholas Carlini+4 more|arXiv (Cornell University)|May 6, 2019

Privacy-Preserving Technologies in Data参考文献 47被引用数 604

ひとこと要約

MixMatch はエントロピー最小化、整合性正規化、および MixUp を統合して、SSL のパフォーマンスを向上させ、はるかに少ないラベルで最先端の結果を達成し、プライバシー-有用性のトレードオフをより良く可能にします。

ABSTRACT

Semi-supervised learning has proven to be a powerful paradigm for leveraging unlabeled data to mitigate the reliance on large labeled datasets. In this work, we unify the current dominant approaches for semi-supervised learning to produce a new algorithm, MixMatch, that works by guessing low-entropy labels for data-augmented unlabeled examples and mixing labeled and unlabeled data using MixUp. We show that MixMatch obtains state-of-the-art results by a large margin across many datasets and labeled data amounts. For example, on CIFAR-10 with 250 labels, we reduce error rate by a factor of 4 (from 38% to 11%) and by a factor of 2 on STL-10. We also demonstrate how MixMatch can help achieve a dramatically better accuracy-privacy trade-off for differential privacy. Finally, we perform an ablation study to tease apart which components of MixMatch are most important for its success.

研究の動機と目的

• Motivate reducing labeled data requirements in SSL by unifying dominant SSL paradigms.
• Develop a single holistic loss that leverages unlabeled data effectively.
• Demonstrate strong empirical gains on standard image SSL benchmarks.
• Explore privacy-utility benefits in privacy-preserving learning (PATE) using MixMatch.

提案手法

• Guess low-entropy labels for augmented unlabeled data via averaging predictions over K augmentations and sharpening with temperature T.
• Combine labeled and unlabeled data through a modified MixUp that preserves batch order and blends labels as probability distributions.
• Compute a supervised loss on augmented labeled data using cross-entropy and an unsupervised loss on guessed-label unlabeled data using a bounded L2 (Brier) loss.
• Train with a combined loss L = L_X + λ_U L_U, with hyperparameters T, K, α (for Beta distribution in MixUp), and λ_U.
• Provide algorithmic description (Algorithm 1) and a label-guessing step diagram (Figure 1) to illustrate the process.

実験結果

リサーチクエスチョン

• RQ1 Can a unified loss that combines entropy minimization, consistency regularization, and MixUp improve SSL performance across standard benchmarks?
• RQ2 How do data augmentation, label guessing, and mixing unlabeled data with labeled data contribute to performance gains?
• RQ3 What are the effects of key hyperparameters (T, K, α, λ_U) on semi-supervised accuracy and stability?
• RQ4 Is MixMatch beneficial for privacy-preserving learning settings (e.g., PATE) and differential privacy budgets?

主な発見

• Achieves state-of-the-art results across standard SSL image benchmarks.
• On CIFAR-10 with 250 labeled examples, MixMatch reduces error rate substantially (e.g., from 38% to 11% in the abstract).
• On CIFAR-10 with 4000 labels, MixMatch attains 6.24% error, approaching supervised performance with 50,000 labels.
• Demonstrates strong results on SVHN and STL-10, including competitive or superior performance with limited labeled data.
• Showcases improved accuracy-privacy trade-offs in privacy-preserving learning, achieving 95.21% test accuracy at ε ≈ 0.97 (vs. VAT baseline at ε ≈ 4.96).
• Ablation studies show each component (augmentation averaging, sharpening, EMA, MixUp, and cross-component mixing) contributes to performance, especially at very low label regimes.