[Paper Review] Representation Learning by Learning to Count

Mehdi Noroozi, Hamed Pirsiavash+1 more|arXiv (Cornell University)|Aug 22, 2017

Domain Adaptation and Few-Shot LearningComputer Science40 references30 citations

TL;DR

This paper proposes a self-supervised representation learning method that trains deep networks to count visual primitives—such as objects or parts—by exploiting invariance under scaling and tiling transformations. Using a contrastive loss on transformed image patches, the model learns semantically meaningful features that match or exceed state-of-the-art performance on transfer learning benchmarks without any manual annotations.

ABSTRACT

We introduce a novel method for representation learning that uses an artificial supervision signal based on counting visual primitives. This supervision signal is obtained from an equivariance relation, which does not require any manual annotation. We relate transformations of images to transformations of the representations. More specifically, we look for the representation that satisfies such relation rather than the transformations that match a given representation. In this paper, we use two image transformations in the context of counting: scaling and tiling. The first transformation exploits the fact that the number of visual primitives should be invariant to scale. The second transformation allows us to equate the total number of visual primitives in each tile to that in the whole image. These two transformations are combined in one constraint and used to train a neural network with a contrastive loss. The proposed task produces representations that perform on par or exceed the state of the art in transfer learning benchmarks.

Motivation & Objective

• To develop a self-supervised representation learning method that avoids manual annotation by using a novel pretext task based on counting visual primitives.
• To formalize a supervision signal using equivariance between image transformations (scaling and tiling) and feature transformations.
• To demonstrate that counting-based self-supervision leads to features discriminative for downstream tasks like classification and detection.
• To validate that the learned features capture high-level semantic content rather than low-level textures or edges.

Proposed method

• The method uses two image transformations: scaling (to enforce scale invariance of visual primitive counts) and tiling (to enforce additive consistency of counts across image regions).
• It formulates a contrastive loss that encourages the network to produce similar representations for transformed image patches that preserve the same total number of visual primitives.
• The supervision signal is derived from an equivariance principle: if the total number of visual primitives is preserved across transformations, the representation must reflect this arithmetic consistency.
• The network is trained end-to-end using a contrastive loss on pairs of transformed images, where positive pairs are those satisfying the counting constraint.
• The counting vector output by the network is used as a representation for downstream transfer learning.
• The approach is generalizable to other transformation relationships beyond counting, provided they can be expressed as functional relationships in feature space.

Experimental results

Research questions

• RQ1 Can counting visual primitives serve as a meaningful pretext task for self-supervised representation learning?
• RQ2 Does enforcing invariance under scaling and tiling transformations lead to representations that capture high-level semantic content?
• RQ3 Can a contrastive loss based on counting consistency outperform existing self-supervised methods on standard transfer learning benchmarks?
• RQ4 To what extent do the learned features reflect semantic concepts rather than low-level image statistics?

Key findings

• The proposed method achieves state-of-the-art performance on standard transfer learning benchmarks, outperforming or matching prior self-supervised methods.
• The counting vector magnitude increases with image region size, indicating sensitivity to the number of visual primitives rather than low-level texture.
• Images with high counting feature magnitude contain multiple objects or large objects, while low-magnitude images are often textures without salient primitives.
• Nearest neighbor retrieval in the counting feature space retrieves semantically similar images with similar scene outlines, confirming the semantic relevance of the features.
• Visualization of neuron activations shows that individual neurons respond to semantically coherent image clusters, such as dogs in ImageNet and people playing baseball in COCO.
• The model performs well even when color is preserved in image crops, suggesting that color does not disrupt the counting signal, though complete color removal degrades performance.