[Paper Review] Exploiting Context for Robustness to Label Noise in Active Learning

Sudipta Paul, Shivkumar Chandrasekaran+2 more|arXiv (Cornell University)|Oct 18, 2020

Machine Learning and Algorithms54 references4 citations

TL;DR

This paper proposes a context-aware noisy label detection (CNLD) method for robust multi-class active learning under label noise. By modeling inter-relationships among data points via graphical representations and comparing new label-induced relations with prior relational knowledge, the method detects incorrect labels through a dissimilarity score, significantly improving recognition accuracy even with noisy annotations across scene, activity, and document classification tasks.

ABSTRACT

Several works in computer vision have demonstrated the effectiveness of active learning for adapting the recognition model when new unlabeled data becomes available. Most of these works consider that labels obtained from the annotator are correct. However, in a practical scenario, as the quality of the labels depends on the annotator, some of the labels might be wrong, which results in degraded recognition performance. In this paper, we address the problems of i) how a system can identify which of the queried labels are wrong and ii) how a multi-class active learning system can be adapted to minimize the negative impact of label noise. Towards solving the problems, we propose a noisy label filtering based learning approach where the inter-relationship (context) that is quite common in natural data is utilized to detect the wrong labels. We construct a graphical representation of the unlabeled data to encode these relationships and obtain new beliefs on the graph when noisy labels are available. Comparing the new beliefs with the prior relational information, we generate a dissimilarity score to detect the incorrect labels and update the recognition model with correct labels which result in better recognition performance. This is demonstrated in three different applications: scene classification, activity classification, and document classification.

Motivation & Objective

• Address the critical challenge of label noise in active learning, where human-annotated labels are often incorrect due to annotator variability.
• Develop a method to detect which queried labels are likely incorrect, especially in multi-class settings where prior work focuses mainly on binary classification.
• Formalize an active learning framework that integrates noisy label detection to improve model robustness and performance despite unreliable annotations.
• Enable effective learning in low-resource and large-scale scenarios where multiple annotators are unavailable and label quality is uncertain.

Proposed method

• Construct a graphical representation of unlabeled data to encode contextual relationships between data points, such as object-scene or action-activity co-occurrences.
• Use prior relational information from an initial seed model to define expected relationships among classes, serving as a baseline for consistency checks.
• Perform conditional inference on the graph to compute new edge beliefs when noisy labels are introduced, capturing updated relational structures.
• Compute a dissimilarity score between prior and updated relational beliefs to quantify inconsistency, identifying labels that deviate significantly from expected context.
• Filter out labels with high dissimilarity scores (likely incorrect) and update the classification model only with high-confidence, context-consistent labels.
• Integrate CNLD into an active learning pipeline that combines human-verified labels and pseudo-labeled data, improving performance by removing noisy pseudo-labels.

Experimental results

Research questions

• RQ1 How can we detect incorrect labels in a multi-class active learning setting where label noise is prevalent and annotators are non-experts?
• RQ2 To what extent can contextual relationships among data points be leveraged to identify inconsistent or noisy labels?
• RQ3 Can a context-aware label filtering mechanism improve recognition performance in the presence of label noise across diverse visual and text classification tasks?
• RQ4 How does the performance of the proposed framework compare to standard active learning and pseudo-labeling approaches when noisy labels are present?

Key findings

• The proposed CNLD method achieves a 1.0% to 3.81% absolute accuracy improvement in scene classification and up to 6.51% improvement in document classification under high noise rates (Ω = 0.50), compared to baseline active learning with noisy labels.
• In activity classification, the method achieves a 1.21% to 4.25% absolute accuracy gain when noise rates are high, demonstrating robustness across different data modalities.
• Filtering pseudo-labels using CNLD significantly improves performance over direct pseudo-labeling, with the best results observed in document and scene classification tasks.
• The framework maintains high performance across diverse applications—scene, activity, and document classification—demonstrating its generalization and independence from specific features or classifiers.
• The β parameter, controlling the precision-recall trade-off in label detection, shows consistent performance gains across a wide range of values (β ∈ {0.80, 0.85, 0.90}) under high noise conditions.
• Qualitative results confirm that low dissimilarity scores correlate strongly with correct labels, validating the method’s ability to detect contextually inconsistent labels.