[Paper Review] Label Confusion Learning to Enhance Text Classification Models

Biyang Guo, Songqiao Han+3 more|arXiv (Cornell University)|Dec 9, 2020

Text and Document Classification Technologies4 citations

TL;DR

This paper proposes Label Confusion Learning (LCM), a novel enhancement for text classification models that replaces one-hot label vectors with a learned label confusion distribution (LCD) based on semantic similarity between instances and labels. By modeling label overlaps and dependencies, LCM improves performance—especially on confused or noisy datasets—outperforming label smoothing and boosting models like LSTM, CNN, and BERT across five benchmark datasets in both English and Chinese.

ABSTRACT

Representing a true label as a one-hot vector is a common practice in training text classification models. However, the one-hot representation may not adequately reflect the relation between the instances and labels, as labels are often not completely independent and instances may relate to multiple labels in practice. The inadequate one-hot representations tend to train the model to be over-confident, which may result in arbitrary prediction and model overfitting, especially for confused datasets (datasets with very similar labels) or noisy datasets (datasets with labeling errors). While training models with label smoothing (LS) can ease this problem in some degree, it still fails to capture the realistic relation among labels. In this paper, we propose a novel Label Confusion Model (LCM) as an enhancement component to current popular text classification models. LCM can learn label confusion to capture semantic overlap among labels by calculating the similarity between instances and labels during training and generate a better label distribution to replace the original one-hot label vector, thus improving the final classification performance. Extensive experiments on five text classification benchmark datasets reveal the effectiveness of LCM for several widely used deep learning classification models. Further experiments also verify that LCM is especially helpful for confused or noisy datasets and superior to the label smoothing method.

Motivation & Objective

• Address the limitations of one-hot label representations in text classification, which assume label independence and fail to capture semantic overlaps between similar labels.
• Mitigate model overconfidence and overfitting caused by noisy or ambiguous labels in human-annotated datasets.
• Improve generalization in text classification by learning a more realistic label distribution that reflects true label dependencies.
• Develop a plug-in component (LCM) that enhances existing deep learning models without modifying their architecture or increasing inference cost.
• Demonstrate the effectiveness of LCM across diverse datasets, including those with high confusion or labeling noise, and extend its applicability to image classification tasks.

Proposed method

• Learn label representations using a shared embedding layer and compute semantic similarity between input text representations and label embeddings.
• Construct a label confusion distribution (LCD) by aggregating similarity scores across all labels, capturing the dependency of each label with the input instance.
• Combine the original one-hot label vector with the LCD using a learnable parameter α, then apply softmax to normalize into a simulated label distribution (SLD).
• Use the SLD as the target for cross-entropy loss during training, replacing the standard one-hot vector to provide more informative supervision.
• Integrate LCM as a lightweight, trainable component during training only—no change to inference or model structure—enabling plug-and-play enhancement.
• Apply early stopping to LCM to prevent overfitting, especially in later training phases where the model may start to degrade.

Experimental results

Research questions

• RQ1 Can modeling label confusion through learned semantic similarity improve text classification performance beyond standard one-hot labeling?
• RQ2 How does LCM perform on datasets with high label similarity (i.e., confused datasets) compared to label smoothing and standard training?
• RQ3 Does LCM effectively reduce overfitting and improve robustness on noisy datasets with labeling errors?
• RQ4 Can LCM be generalized beyond text classification to other tasks such as image classification?
• RQ5 What hyperparameters (e.g., α) and training strategies (e.g., early stopping) optimize LCM performance across different datasets?

Key findings

• LCM significantly improves classification accuracy on five benchmark text classification datasets (including IMDB, 20NG, and Chinese datasets) across LSTM, CNN, and BERT models.
• On highly confused datasets (e.g., 8NG-H and 4NG-H), LCM achieves substantial gains—up to 3.5% improvement—while showing minimal or negative gains on easier, less confused sets (e.g., 4NG-E).
• LCM outperforms label smoothing (LS) consistently, especially in noisy settings: on 20NG with 20% noise, LCM improves accuracy by over 2% compared to LS.
• In image classification, LCM boosts test accuracy from 98.22% (baseline CNN) to 98.41% on MNIST and from 89.29% to 90.28% on Fashion MNIST, proving cross-domain applicability.
• The use of early stopping for LCM training prevents overfitting and further improves performance, with optimal results observed when LCM is deactivated after ~10 epochs.
• A smaller α value (e.g., 0.5) generally yields better results on confused or noisy datasets, indicating sensitivity to label similarity and error rates.