[论文解读] Semi-Supervised Learning with Balanced Deep Representation Distributions

Changchun Li, Ximing Li+3 more|arXiv (Cornell University)|Mar 22, 2026

Text and Document Classification Technologies被引用 0

一句话总结

简述：引入 S2tc-bdd，一种半监督文本分类方法，在深度表示中平衡标签角度方差以提高伪标签准确性和在标注稀缺情况下的性能。

ABSTRACT

Semi-Supervised Text Classification (SSTC) mainly works under the spirit of self-training. They initialize the deep classifier by training over labeled texts; and then alternatively predict unlabeled texts as their pseudo-labels and train the deep classifier over the mixture of labeled and pseudo-labeled texts. Naturally, their performance is largely affected by the accuracy of pseudo-labels for unlabeled texts. Unfortunately, they often suffer from low accuracy because of the margin bias problem caused by the large difference between representation distributions of labels in SSTC. To alleviate this problem, we apply the angular margin loss, and perform several Gaussian linear transformations to achieve balanced label angle variances, i.e., the variance of label angles of texts within the same label. More accuracy of predicted pseudo-labels can be achieved by constraining all label angle variances balanced, where they are estimated over both labeled and pseudo-labeled texts during self-training loops. With this insight, we propose a novel SSTC method, namely Semi-Supervised Text Classification with Balanced Deep representation Distributions (S2TC-BDD). We implement both multi-class classification and multi-label classification versions of S2TC-BDD by introducing some pseudo-labeling tricks and regularization terms. To evaluate S2 TC-BDD, we compare it against the state-of-the-art SSTC methods. Empirical results demonstrate the effectiveness of S2 TC-BDD, especially when the labeled texts are scarce.

研究动机与目标

• 通过不平衡标签表示分布在半监督文本分类（SSTC）中识别边缘偏差。
• 通过基于高斯的角度变换开发平衡深度表示分布（BDD）损失以平衡标签角度。
• 在自训练过程中扩展 SSTC，使其具备多类和多标签变体，利用平衡分布。
• 证明 S2tc-bdd 相比最先进的 SSTC 方法在标注数据有限时具有更优性能。

提出的方法

• 在 BERT 的角度边距（AM）损失基础上建立模型，以学习判别性深度表示。
• 假设标签角度来自标签特定的高斯分布，并应用高斯线性变换在标签之间平衡方差（ψk(θik)）。
• 定义并优化平衡深度表示分布（BDDL）损失 L_bdd，在 AM 损失中用 ψk(·) 替代 θik。
• 在自训练过程中同时从有标签文本和伪标签文本估计标签角度分布（均值 μk 和方差 σk^2）以及原型 ck。
• 对于多类，使用锐化和信息熵正则化；对于多标签，应用基于类别分布的伪标签（CAP）及通过 ADMM 的低秩正则化。
• 给出完整目标函数：多类的 L_mcc 和多标签的 L_mlc，整合有监督、无监督和正则化项。

Figure 1: The average difference of label angle variances (Avg.DLAV) computed in semi-supervised and supervised manners across AG News ( Multi-Class Case ) and AAPD ( Multi-Label Case ), respectively.

实验结果

研究问题

• RQ1 不平衡的标签表示分布造成的边缘偏差如何影响 SSTC 的伪标签准确性？
• RQ2 将角度转化为平衡的标签分布是否能提高伪标签质量和最终性能？
• RQ3 在标注数据稀缺时，使用 S2tc-bdd 的多类和多标签 SSTC 设置表现如何？
• RQ4 锐化、CAP（Cap）和正则化对使用平衡表示进行学习的影响？
• RQ5 在标准基准上提出的方法是否优于现有的 SSTC 基线？

主要发现

• S2tc-bdd 相较于最先进的 SSTC 方法具有更优性能，尤其是在标注数据稀缺时。
• 通过高斯线性变换平衡标签角度方差，消除边缘偏差并提升伪标签准确性。
• 该方法对多类和多标签文本分类任务均有效扩展。
• 结合锐化或 CAP 与信息熵正则化，在自训练下提升学习效果。
• 在 AG News、Yelp、Yahoo（多类）以及 Ohsumed、AAPD、RCV1-V2（多标签）的实验中显示出稳健的提升。

Figure 2: Let solid circles and triangles denote labeled positive and negative texts, and hollow ones denote corresponding unlabeled texts. (a) The large difference between label angle variances results in the margin bias. Many unlabeled texts (in red) can be misclassified. (b) Balancing the label a