[Paper Review] Context-based Virtual Adversarial Training for Text Classification with Noisy Labels

Do-Myoung Lee, Yeachan Kim+1 more|arXiv (Cornell University)|May 29, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes Context-based Virtual Adversarial Training (ConVAT), a network-agnostic method that performs adversarial training on contextual feature representations rather than word-level inputs to improve robustness against noisy labels in text classification. ConVAT achieves state-of-the-art performance with significantly reduced time and memory costs compared to standard VAT, effectively preventing memorization of noisy labels while maintaining strong generalization on four benchmark datasets under high noise rates.

ABSTRACT

Deep neural networks (DNNs) have a high capacity to completely memorize noisy labels given sufficient training time, and its memorization, unfortunately, leads to performance degradation. Recently, virtual adversarial training (VAT) attracts attention as it could further improve the generalization of DNNs in semi-supervised learning. The driving force behind VAT is to prevent the models from overfitting data points by enforcing consistency between the inputs and the perturbed inputs. This strategy could be helpful in learning from noisy labels if it prevents neural models from learning noisy samples while encouraging the models to generalize clean samples. In this paper, we propose context-based virtual adversarial training (ConVAT) to prevent a text classifier from overfitting to noisy labels. Unlike the previous works, the proposed method performs the adversarial training at the context level rather than the inputs. It makes the classifier not only learn its label but also its contextual neighbors, which alleviates the learning from noisy labels by preserving contextual semantics on each data point. We conduct extensive experiments on four text classification datasets with two types of label noises. Comprehensive experimental results clearly show that the proposed method works quite well even with extremely noisy settings.

Motivation & Objective

• To address the problem of deep neural networks overfitting to noisy labels in text classification.
• To improve model generalization by enforcing consistency between clean samples and their contextual neighbors.
• To reduce computational and memory overhead compared to standard virtual adversarial training (VAT).
• To develop a network-agnostic method that does not require additional training parameters.
• To demonstrate effectiveness under extreme label noise settings across multiple text classification datasets.

Proposed method

• ConVAT performs adversarial training in the context-level feature space rather than at the word-level input, focusing on contextual representations.
• It formulates a min-max optimization problem to find the worst-case perturbation that maximizes the classification loss on a given sample.
• The method minimizes the distributional distance between original and perturbed context vectors to enforce robustness and smoothness.
• Perturbations are computed using gradients only from the softmax layer, reducing memory and computation compared to full-network backpropagation in VAT.
• The approach is applied during training without modifying the network architecture or adding extra parameters.
• It leverages label smoothing implicitly through adversarial perturbations, reducing overconfidence and memorization of noisy labels.

Experimental results

Research questions

• RQ1 Can adversarial training at the context level improve robustness to label noise in text classification?
• RQ2 Does context-level adversarial training reduce overfitting to noisy labels compared to word-level VAT?
• RQ3 Can ConVAT achieve better performance with lower time and memory complexity than standard VAT?
• RQ4 How does the choice of perturbation magnitude (epsilon) affect performance under varying noise rates?
• RQ5 Is ConVAT effective across diverse text classification datasets with different noise patterns?

Key findings

• ConVAT outperforms state-of-the-art methods on four text classification datasets under both uniform and class-conditional label noise settings.
• The model trained with ConVAT shows stable and gradually improving validation accuracy, indicating no overfitting to noisy labels, even at 50% noise rate.
• ConVAT reduces training time by up to 5x and memory usage significantly compared to standard VAT, especially on deeper networks.
• t-SNE visualization confirms that context vectors from ConVAT remain well-clustered and class-discriminative throughout training, unlike CNNs that lose cluster structure.
• Optimal epsilon values increase with noise rate, and grid search confirms that higher noise rates require larger perturbation magnitudes for best performance.
• The method maintains strong generalization even with extremely high noise levels, demonstrating robustness through consistent performance gains.