[Paper Review] Unsupervised Learning of Dense Visual Representations

Pedro O. Pinheiro, Amjad Almahairi+3 more|arXiv (Cornell University)|Nov 11, 2020

Domain Adaptation and Few-Shot LearningComputer Science50 citations

TL;DR

VADeR learns pixel-level dense visual representations via pixelwise contrastive learning, outperforming strong baselines on dense prediction tasks and sometimes surpassing ImageNet supervised pretraining.

ABSTRACT

Contrastive self-supervised learning has emerged as a promising approach to unsupervised visual representation learning. In general, these methods learn global (image-level) representations that are invariant to different views (i.e., compositions of data augmentation) of the same image. However, many visual understanding tasks require dense (pixel-level) representations. In this paper, we propose View-Agnostic Dense Representation (VADeR) for unsupervised learning of dense representations. VADeR learns pixelwise representations by forcing local features to remain constant over different viewing conditions. Specifically, this is achieved through pixel-level contrastive learning: matching features (that is, features that describes the same location of the scene on different views) should be close in an embedding space, while non-matching features should be apart. VADeR provides a natural representation for dense prediction tasks and transfers well to downstream tasks. Our method outperforms ImageNet supervised pretraining (and strong unsupervised baselines) in multiple dense prediction tasks.

Motivation & Objective

• Motivate learning dense (pixel-level) representations for visual understanding tasks beyond global image representations.
• Propose View-Agnostic Dense Representation (VADeR) that enforces perceptual constancy across views at the pixel level.
• Develop an encoder-decoder architecture to produce per-pixel embeddings suitable for dense prediction tasks.
• Evaluate transferability of VADeR features to segmentation, depth estimation, object detection, and keypoint tasks.

Proposed method

• Use an encoder-decoder network (FPN with ResNet-50 backbone) to produce per-pixel embeddings at 1/4 resolution.
• Train with pixel-wise contrastive loss (NCE) to pull together embeddings of the same pixel across different views and push apart different pixels.
• Generate positive pixel pairs from known pixel correspondences between two augmented views; negatives drawn from a large momentum queue (MoCo-style).
• Represent pixel similarity with temperature-scaled cosine similarity c(x1,x2)= (1/τ) x1^T x2 / (||x1|| ||x2||).
• Train using two-view pairs created by random appearance and geometric transformations; use 32 matching pixel pairs per image for the loss.
• Initialize encoder with MoCo2 weights; train decoder from scratch; use a 65,536-sized dictionary with momentum 0.999 for negatives.

Experimental results

Research questions

• RQ1 Can dense (pixel-level) representations learned in an unsupervised fashion transfer effectively to dense prediction tasks such as segmentation, depth, and keypoint detection?
• RQ2 Do pixel-level contrastive objectives that enforce viewpoint invariance improve over global (image-level) representations for dense tasks?
• RQ3 What is the impact of pixel correspondence quality and view sampling strategies on downstream performance?
• RQ4 How does VADeR perform relative to MoCo and ImageNet-supervised pretraining when fine-tuning with varying amounts of labeled data?

Key findings

• VADeR outperforms MoCo on semantic segmentation and depth prediction when using fixed features (e.g., mIoU 56.7 vs 43.0 on VOC; depth RMSE 0.964 vs 1.136 on NYU-d v2).
• VADeR also surpasses supervised ImageNet pretraining on at least one semantic segmentation task and depth prediction under fixed representation settings.
• In object detection, instance segmentation, and keypoint detection, VADeR consistently outperforms MoCo and the supervised baseline under the same FPN architecture.
• Fine-tuning with VADeR initializes both encoder and decoder and shows advantages when labeled data is limited, often matching or exceeding MoCo performance and surpassing supervised pretraining as labeled data decreases.
• For dense correspondence tasks (DAVIS-2017), VADeR achieves higher J and F metrics compared to several baselines, indicating strong pixel-level transfer.