[Paper Review] Local Label Propagation for Large-Scale Semi-Supervised Learning

Chengxu Zhuang, Xuehao Ding+2 more|arXiv (Cornell University)|May 28, 2019

Domain Adaptation and Few-Shot Learning34 references7 citations

TL;DR

This paper proposes Local Label Propagation (LLP), a scalable semi-supervised learning method that trains a deep neural network to embed labeled and unlabeled data in a shared latent space, then propagates pseudo-labels using local geometric structure—prioritizing proximity and local density—while jointly optimizing classification accuracy and cluster cohesion. LLP achieves state-of-the-art performance on ImageNet across diverse training regimes and generalizes well to transfer learning on Places205.

ABSTRACT

A significant issue in training deep neural networks to solve supervised learning tasks is the need for large numbers of labelled datapoints. The goal of semi-supervised learning is to leverage ubiquitous unlabelled data, together with small quantities of labelled data, to achieve high task performance. Though substantial recent progress has been made in developing semi-supervised algorithms that are effective for comparatively small datasets, many of these techniques do not scale readily to the large (unlaballed) datasets characteristic of real-world applications. In this paper we introduce a novel approach to scalable semi-supervised learning, called Local Label Propagation (LLP). Extending ideas from recent work on unsupervised embedding learning, LLP first embeds datapoints, labelled and otherwise, in a common latent space using a deep neural network. It then propagates pseudolabels from known to unknown datapoints in a manner that depends on the local geometry of the embedding, taking into account both inter-point distance and local data density as a weighting on propagation likelihood. The parameters of the deep embedding are then trained to simultaneously maximize pseudolabel categorization performance as well as a metric of the clustering of datapoints within each psuedo-label group, iteratively alternating stages of network training and label propagation. We illustrate the utility of the LLP method on the ImageNet dataset, achieving results that outperform previous state-of-the-art scalable semi-supervised learning algorithms by large margins, consistently across a wide variety of training regimes. We also show that the feature representation learned with LLP transfers well to scene recognition in the Places 205 dataset.

Motivation & Objective

• Address the scalability limitations of existing semi-supervised learning methods that rely on global similarity computations, which scale poorly with large datasets.
• Develop a method that effectively leverages sparse labeled data and abundant unlabeled data in large-scale learning scenarios, such as ImageNet.
• Improve representation learning by co-optimizing classification performance and cluster compactness of pseudo-labeled data in the embedding space.
• Enable strong transferability of learned features to downstream tasks beyond object classification, such as scene recognition.

Proposed method

• Train a deep neural network to embed both labeled and unlabeled data into a shared low-dimensional latent space.
• Perform label propagation from known to unknown examples based on local neighborhood structure, weighted by inter-point distance and local data density.
• Use a density-weighted probability model to modulate propagation likelihood, favoring regions of higher known-label density.
• Optimize the network via a dual objective: (1) cross-entropy loss on pseudo-labels (with higher weight on true labels), and (2) a clustering loss that encourages points with the same pseudo-label to cluster together in the embedding space.
• Iteratively alternate between label propagation and network training to refine both representations and pseudo-label assignments.
• Use a memory bank of features and nearest-neighbor inference for efficient evaluation during training.

Experimental results

Research questions

• RQ1 Can local geometric structure in a learned embedding space be effectively exploited to scale label propagation to large datasets like ImageNet?
• RQ2 How does incorporating local density into label propagation improve performance compared to global similarity-based methods?
• RQ3 To what extent does joint optimization of classification and clustering objectives enhance representation quality and downstream accuracy?
• RQ4 How well do LLP-trained features generalize to transfer learning tasks outside of object classification, such as scene recognition?

Key findings

• LLP achieves state-of-the-art performance on ImageNet across all tested training regimes, including settings with very few labeled examples.
• The method significantly outperforms previous scalable semi-supervised learning algorithms such as those in [29, 18, 24, 16, 6, 22] on ImageNet.
• LLP features transfer effectively to the Places205 dataset, improving scene recognition performance.
• Ablation studies confirm that local neighborhood selection (K), confidence weighting, combined loss, and density-weighted propagation are all critical to performance.
• The learned embeddings capture meaningful sub-clusters in multi-modal categories (e.g., 'labrador' splitting into 'black' and 'yellow' subtypes), even without such labels in the original annotation.
• The method maintains strong performance on small subsets (50 categories, 50 images per category), showing competitiveness with global propagation methods despite its local design.