[Paper Review] When Labelled Data Hurts: Deep Semi-Supervised Classification with the Graph 1-Laplacian

Angelica I. Avilés-Rivero, Nicolas Papadakis+4 more|arXiv (Cornell University)|Jun 20, 2019

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a novel deep semi-supervised classification framework using the non-smooth ℓ₁ norm of the normalized graph 1-Laplacian to improve performance with limited labelled data. By enforcing smoothness and leveraging intrinsic data relationships through class priors, the method achieves state-of-the-art results on CIFAR-10, CIFAR-100, and ChestX-Ray14, outperforming classic methods and matching deep learning baselines.

ABSTRACT

We consider the task of classifying when a significantly reduced amount of labelled data is available. This problem is of a great interest, in several real-world problems, as obtaining large amounts of labelled data is expensive and time consuming. We present a novel semi-supervised framework for multi-class classification that is based on the non-smooth $\ell_1$ norm of the normalised graph 1-Laplacian. Our transductive framework is framed under a novel functional with carefully selected class priors - that enforces a sufficiently smooth solution and strengthens the intrinsic relation between the labelled and unlabelled data. We provide theoretical results of our new optimisation model and show its connections with deep learning for handling large-scale datasets. We demonstrate through extensive experimental results on large datasets - CIFAR-10, CIFAR-100 and ChestX-Ray14 - that our method outperforms classic methods and readily competes with recent deep-learning approaches.

Motivation & Objective

• Address the challenge of limited labelled data in real-world classification tasks, where data annotation is costly and time-consuming.
• Develop a transductive semi-supervised framework that effectively leverages both labelled and unlabelled data to improve classification performance.
• Introduce a novel functional with class priors that strengthens the intrinsic relationship between labelled and unlabelled data.
• Ensure solution smoothness through the non-smooth ℓ₁ norm of the normalized graph 1-Laplacian to enhance generalization.
• Demonstrate strong performance on large-scale datasets comparable to deep learning models, despite minimal use of labelled data.

Proposed method

• Formulate a new optimization model based on the normalized graph 1-Laplacian using the ℓ₁ norm to promote sparsity and robustness in label propagation.
• Incorporate class priors into the functional to guide the model toward more coherent and intrinsic data structure.
• Enforce solution smoothness by minimizing the ℓ₁ norm of the graph 1-Laplacian, which better preserves local data geometry than traditional ℓ₂-based methods.
• Design a transductive framework that predicts labels for unlabelled samples based on the joint structure of labelled and unlabelled data.
• Establish theoretical connections between the proposed model and deep learning architectures to support scalability on large datasets.
• Optimize the functional using a novel algorithmic approach that ensures convergence and stability in high-dimensional settings.

Experimental results

Research questions

• RQ1 Can a non-smooth ℓ₁-based graph regularizer improve semi-supervised classification when labelled data is scarce?
• RQ2 How does incorporating class priors into the graph 1-Laplacian framework enhance the relationship between labelled and unlabelled data?
• RQ3 To what extent does the proposed method outperform classic semi-supervised learning methods on benchmark datasets?
• RQ4 Can the proposed framework achieve performance competitive with deep learning models despite using minimal labelled data?
• RQ5 What is the theoretical and empirical connection between the proposed model and deep learning architectures in handling large-scale data?

Key findings

• The proposed method achieves state-of-the-art performance on CIFAR-10 and CIFAR-100 with significantly reduced labelled data, outperforming classic semi-supervised learning methods.
• On the ChestX-Ray14 dataset, the method demonstrates strong generalization and robustness, achieving competitive results with deep learning baselines.
• The use of the ℓ₁ norm in the graph 1-Laplacian leads to sparser and more robust label propagation compared to ℓ₂-based alternatives.
• The integration of class priors strengthens the intrinsic data structure and improves the consistency of predictions across unlabelled samples.
• Theoretical analysis confirms the model's compatibility with deep learning frameworks, enabling scalability to large-scale datasets.
• Extensive experiments confirm that the method maintains high accuracy even when only a small fraction of training data is labelled.