[Paper Review] DINOv2: Learning Robust Visual Features without Supervision

Maxime Oquab, Timothée Darcet+24 more|arXiv (Cornell University)|Apr 14, 2023

Multimodal Machine Learning Applications131 references1,011 citations

TL;DR

DINOv2 trains large-scale self-supervised vision transformers on a curated, diverse dataset to produce transferable, out-of-the-box visual features that rival weakly-supervised models across image- and pixel-level tasks.

ABSTRACT

The recent breakthroughs in natural language processing for model pretraining on large quantities of data have opened the way for similar foundation models in computer vision. These models could greatly simplify the use of images in any system by producing all-purpose visual features, i.e., features that work across image distributions and tasks without finetuning. This work shows that existing pretraining methods, especially self-supervised methods, can produce such features if trained on enough curated data from diverse sources. We revisit existing approaches and combine different techniques to scale our pretraining in terms of data and model size. Most of the technical contributions aim at accelerating and stabilizing the training at scale. In terms of data, we propose an automatic pipeline to build a dedicated, diverse, and curated image dataset instead of uncurated data, as typically done in the self-supervised literature. In terms of models, we train a ViT model (Dosovitskiy et al., 2020) with 1B parameters and distill it into a series of smaller models that surpass the best available all-purpose features, OpenCLIP (Ilharco et al., 2021) on most of the benchmarks at image and pixel levels.

Motivation & Objective

• Motivate foundation-model-style, task-agnostic visual representations in vision similar to NLP foundations.
• Show that self-supervised pretraining on curated, diverse data can yield transferable features without finetuning.
• Develop scalable training techniques and data pipelines to enable large-model self-supervision.
• Demonstrate that distillation can transfer knowledge from large to smaller models while retaining quality.

Proposed method

• Combine DINO and iBOT losses with Sinkhorn-Knopp centering and KoLeo regularizer to train discriminative self-supervised models.
• Use image- and patch-level objectives to learn features from ViT backbones.
• Construct LVD-142M as a curated, diverse pretraining dataset via a retrieval-based data-augmentation pipeline without relying on text or metadata.
• Employ data-efficient training enhancements (sequence packing, efficient attention, FSDP, stochastic depth) to scale to 1B-parameter ViT models.
• Apply distillation: train smaller models from a frozen large teacher to improve small-model performance.
• Increase resolution briefly at the end of training to boost pixel-level tasks without full-cost high-res training.

Experimental results

Research questions

• RQ1 Can self-supervised pretraining on a curated, large-scale image dataset produce general-purpose visual features that perform well out-of-the-box across diverse tasks?
• RQ2 How do data curation, model scaling, and training optimizations impact the quality and transferability of self-supervised visual features?
• RQ3 Does distillation from a large self-supervised teacher improve small-model performance across vision benchmarks?
• RQ4 What is the effect of high-resolution finetuning at late training stages on pixel-level tasks?
• RQ5 How do image- and patch-level objectives interact to support both global and local vision tasks?

Key findings

• DINOv2 significantly improves over prior self-supervised methods on frozen features across multiple benchmarks.
• A ViT-g billion-parameter model trained on LVD-142M matches ImageNet-1k performance of models trained on ImageNet-22k while outperforming on other benchmarks.
• Knowledge distillation from a large DINOv2 teacher yields better ViT-L performance than training from scratch.
• Using a curated dataset (LVD-142M) yields better transfer to diverse domains than using uncurated data, and often surpasses ImageNet-22k baselines for non-ImageNet domains.
• High-resolution training at the end of training provides most of the gains for pixel-level tasks at a fraction of full high-res training cost.