[Paper Review] Data-Efficient Image Recognition with Contrastive Predictive Coding

Olivier J. Hénaff, Aravind Srinivas+5 more|arXiv (Cornell University)|May 22, 2019

Advanced Image and Video Retrieval Techniques99 references936 citations

TL;DR

This paper revisits and improves Contrastive Predictive Coding (CPC) to learn unsupervised image representations that enable data-efficient recognition, achieving state-of-the-art linear ImageNet performance and strong transfer to object detection, with markedly better performance using few labels than pixels-based training.

ABSTRACT

Human observers can learn to recognize new categories of images from a handful of examples, yet doing so with artificial ones remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable. We therefore revisit and improve Contrastive Predictive Coding, an unsupervised objective for learning such representations. This new implementation produces features which support state-of-the-art linear classification accuracy on the ImageNet dataset. When used as input for non-linear classification with deep neural networks, this representation allows us to use 2-5x less labels than classifiers trained directly on image pixels. Finally, this unsupervised representation substantially improves transfer learning to object detection on the PASCAL VOC dataset, surpassing fully supervised pre-trained ImageNet classifiers.

Motivation & Objective

• Motivate data-efficient recognition by learning representations that make natural signal variability more predictable.
• Improve CPC architecture and training to maximize downstream linear separability and data efficiency.
• Demonstrate that CPC-pretrained features enable far fewer labels to achieve competitive or superior performance compared to pixel-based supervised models.
• Show transferability of CPC representations to object detection on PASCAL VOC 2007, surpassing supervised pre-training.
• Provide practical guidance on architectural and augmentation choices that enhance CPC performance.

Proposed method

• Reformulate CPC with an enhanced encoder (ResNet-161) and expanded patch-based prediction tasks to increase supervisory signal.
• Use layer normalization instead of batch normalization to improve training efficiency and performance.
• Predict across multiple directions (top, bottom, left, right) to increase prediction tasks and accuracy.
• Apply patch-based augmentations, including color dropping and random geometric/color transformations, to deter reliance on low-level cues.
• Train a linear classifier on CPC features for linear evaluation, and optionally fine-tune the encoder and classifier jointly for efficient classification.
• Transfer CPC representations to Faster-RCNN for object detection on PASCAL VOC 2007 to assess generalization.

Experimental results

Research questions

• RQ1 Can CPC be scaled and modified to yield representations that improve data-efficient image recognition?
• RQ2 Do architectural changes, normalization choices, and multi-directional predictions boost CPC performance?
• RQ3 Do CPC representations enable competitive or superior performance with limited labeled data compared to pixel-based supervised models?
• RQ4 Do CPC-pretrained features transfer effectively to object detection on a different dataset (PASCAL VOC 2007)?

Key findings

• CPC v2 achieves 71.5% Top-1 accuracy on linear ImageNet classification, up from 48.7% with CPC v1.
• With 1% of ImageNet labels, a classifier on CPC features attains 78.3% Top-5 accuracy (vs. 44.1% Top-5 for pixel-based supervised training with 1% labels).
• When fully trained with all labels, CPC-based models reach 83.4% Top-1 and 96.5% Top-5, surpassing the supervised ResNet-200 baseline.
• CPC representations enable strong transfer, yielding 76.6% mAP on PASCAL VOC 2007 object detection, outperforming supervised pre-training (74.7% mAP).
• CPC improvements stem from larger capacity (ResNet-161), larger receptive fields, layer normalization, multi-directional predictions, and extensive patch-based augmentation.
• CPC-based methods are competitive with or surpass label-propagation and other self-supervised approaches in data-efficient recognition.