[论文解读] Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data

Colin Wei, Kendrick Shen+2 more|arXiv (Cornell University)|Oct 7, 2020

Machine Learning and Data Classification参考文献 68被引用 74

一句话总结

本文首次对深度神经网络在半监督学习、域自适应和无监督学习中的自训练方法提供了统一的理论分析。它引入了一个“扩展”假设，并证明在该假设下，自训练和输入一致性正则化能够最小化总体目标函数，从而实现高准确率，且对神经网络的样本复杂度界为边缘和Lipschitz常数的多项式形式。

ABSTRACT

Self-training algorithms, which train a model to fit pseudolabels predicted by another previously-learned model, have been very successful for learning with unlabeled data using neural networks. However, the current theoretical understanding of self-training only applies to linear models. This work provides a unified theoretical analysis of self-training with deep networks for semi-supervised learning, unsupervised domain adaptation, and unsupervised learning. At the core of our analysis is a simple but realistic "expansion" assumption, which states that a low probability subset of the data must expand to a neighborhood with large probability relative to the subset. We also assume that neighborhoods of examples in different classes have minimal overlap. We prove that under these assumptions, the minimizers of population objectives based on self-training and input-consistency regularization will achieve high accuracy with respect to ground-truth labels. By using off-the-shelf generalization bounds, we immediately convert this result to sample complexity guarantees for neural nets that are polynomial in the margin and Lipschitzness. Our results help explain the empirical successes of recently proposed self-training algorithms which use input consistency regularization.

研究动机与目标

• 为深度神经网络的自训练建立超越线性模型的理论基础。
• 统一分析自训练在半监督学习、无监督域自适应和无监督学习中的表现。
• 识别确保收敛至高准确率模型的最小且现实的假设，特别是“扩展”假设。
• 将理论保证与神经网络的实际泛化界联系起来。
• 解释近期自训练算法中输入一致性正则化的经验成功原因。

提出的方法

• 引入“扩展”假设：低概率数据子集必须扩展为高概率邻域。
• 在不同类别样本的邻域之间施加最小重叠条件，以确保可分性。
• 基于自训练和输入一致性正则化分析总体目标函数。
• 使用现成的泛化界推导深度网络的样本复杂度保证。
• 证明在给定假设下，这些目标函数的最小化器可达到高准确率。
• 通过与边缘和Lipschitz常数相关的多项式界，将理论收敛性与实际性能联系起来。

实验结果

研究问题

• RQ1 在半监督和无监督设置下，深度神经网络的自训练在何种条件下收敛至高准确率模型？
• RQ2 如何形式化“扩展”假设，以确保自训练中可靠的泛化性能？
• RQ3 在所提假设下，输入一致性正则化在提升模型准确率方面发挥何种作用？
• RQ4 理论保证如何转化为深度神经网络的样本复杂度界？
• RQ5 为何近期采用输入一致性正则化的自训练算法能取得优异的实证结果？

主要发现

• 在扩展假设和最小类邻域重叠条件下，自训练和输入一致性正则化能实现相对于真实标签的高准确率。
• 自训练和输入一致性正则化中总体目标函数的最小化器收敛至高准确率解。
• 泛化界表明样本复杂度为网络边缘和Lipschitz常数的多项式形式。
• 该理论框架解释了近期自训练方法中输入一致性正则化经验成功的原因。
• 该分析在半监督学习、无监督域自适应和无监督学习设置中具有统一适用性。
• 结果为在低数据量场景下利用无标签数据时自训练的鲁棒性提供了理论依据。