[Paper Review] Into the Rabbit Hull: From Task-Relevant Concepts in DINO to Minkowski Geometry

Thomas Fel, Binxu Wang+9 more|ArXiv.org|Oct 8, 2025

Face Recognition and Perception3 citations

TL;DR

This study builds a large overcomplete dictionary of 32,000 visual concepts from DINOv2 using stable sparse autoencoders, analyzes how downstream tasks recruit these concepts, and proposes the Minkowski Representation Hypothesis to describe a convex-archetype geometry of activations.

ABSTRACT

DINOv2 is routinely deployed to recognize objects, scenes, and actions; yet the nature of what it perceives remains unknown. As a working baseline, we adopt the Linear Representation Hypothesis (LRH) and operationalize it using SAEs, producing a 32,000-unit dictionary that serves as the interpretability backbone of our study, which unfolds in three parts. In the first part, we analyze how different downstream tasks recruit concepts from our learned dictionary, revealing functional specialization: classification exploits "Elsewhere" concepts that fire everywhere except on target objects, implementing learned negations; segmentation relies on boundary detectors forming coherent subspaces; depth estimation draws on three distinct monocular depth cues matching visual neuroscience principles. Following these functional results, we analyze the geometry and statistics of the concepts learned by the SAE. We found that representations are partly dense rather than strictly sparse. The dictionary evolves toward greater coherence and departs from maximally orthogonal ideals (Grassmannian frames). Within an image, tokens occupy a low dimensional, locally connected set persisting after removing position. These signs suggest representations are organized beyond linear sparsity alone. Synthesizing these observations, we propose a refined view: tokens are formed by combining convex mixtures of archetypes (e.g., a rabbit among animals, brown among colors, fluffy among textures). This structure is grounded in Gardenfors' conceptual spaces and in the model's mechanism as multi-head attention produces sums of convex mixtures, defining regions bounded by archetypes. We introduce the Minkowski Representation Hypothesis (MRH) and examine its empirical signatures and implications for interpreting vision-transformer representations.

Motivation & Objective

• Motivate and operationalize the Linear Representation Hypothesis (LRH) for interpretability of vision transformers.
• Create a large, stable dictionary of visual concepts (32,000 atoms) from DINOv2 activations using sparse autoencoders.
• Characterize how downstream tasks (classification, segmentation, depth) selectively recruit concepts.
• Examine the geometry, sparsity, and coherence of the concept dictionary beyond strict sparsity.
• Propose the Minkowski Representation Hypothesis (MRH) to describe token formation as convex mixtures around archetypes.

Proposed method

• Operationalize LRH with a stable sparse autoencoder that factorizes DINOv2 activations into a nonnegative code Z and a dictionary D, with D constrained to conv(A) for stability.
• Use a dictionary of c = 32,000 atoms and enforce k = 8 active codes per token, approximating conv(A) via 128,000 centroids from 1.4M ImageNet images.
• Train with Adam for 50 epochs achieving reconstruction fidelity R^2 > 88%.
• Analyze downstream task alignment by computing expected concept importance E(Z W') as a measure of concept-task relevance.
• Visualize and cluster concept activations to identify task-specific subspaces and archetype-like structures.

Experimental results

Research questions

• RQ1 What internal features (concepts) does DINOv2 encode, and how are they organized geometrically?
• RQ2 How do downstream tasks (classification, segmentation, depth estimation) recruit different subsets of learned concepts?
• RQ3 Do concepts form functional subspaces or more general convex archetypes rather than strictly orthogonal directions?
• RQ4 What is the role of token types (cls, reg, spatial) in concept activation patterns?
• RQ5 What are the empirical signatures of the Minkowski Representation Hypothesis in vision transformers?

Key findings

• Downstream tasks recruit distinct concept subsets, with classification using a broad concept set, while segmentation and depth rely on more localized, low-dimensional subspaces.
• Concepts show partial density and coherence; their inner products exhibit heavier tails than orthogonal models, and task subspaces are low-dimensional and more aligned than random subsets.
• Top task-aligned concepts per head display intra-task similarity, indicating functional subspaces rather than purely orthogonal directions.
• Classification yields “Elsewhere” concepts that activate off-object regions conditioned on object presence, suggesting a structured, posited negation logic.
• Segmentation relies on border concepts localized along object boundaries, forming tight clusters indicative of specialized border detectors.
• Depth concepts cluster into three families: projective-geometry cues, shadow-based cues, and local frequency transitions, reflecting monocular depth cues learned from 2D data.
• Register tokens reveal global scene properties via register-only concepts, including lighting, motion blur, and camera effects, indicating global non-local features.