[Paper Review] Unsupervised learning of object frames by dense equivariant image labelling

James Thewlis, Hakan Bilen+1 more|arXiv (Cornell University)|Jun 9, 2017

Advanced Image and Video Retrieval TechniquesComputer Science49 references64 citations

TL;DR

The paper trains a CNN to map every image pixel to a dense, object-centric coordinate space that is invariant to deformations, enabling unsupervised learning of consistent object frames for articulated and deformable objects (e.g., faces, robot arms) using only image transformations or optical flow as supervision.

ABSTRACT

One of the key challenges of visual perception is to extract abstract models of 3D objects and object categories from visual measurements, which are affected by complex nuisance factors such as viewpoint, occlusion, motion, and deformations. Starting from the recent idea of viewpoint factorization, we propose a new approach that, given a large number of images of an object and no other supervision, can extract a dense object-centric coordinate frame. This coordinate frame is invariant to deformations of the images and comes with a dense equivariant labelling neural network that can map image pixels to their corresponding object coordinates. We demonstrate the applicability of this method to simple articulated objects and deformable objects such as human faces, learning embeddings from random synthetic transformations or optical flow correspondences, all without any manual supervision.

Motivation & Objective

• Motivate unsupervised discovery of intrinsic object structure by removing nuisance factors such as viewpoint and deformations.
• Extend landmark-based viewpoint factorization to dense, per-pixel object coordinates.
• Introduce a neural labelling function that assigns pixels to a canonical object-centric space.
• Incorporate equivariance with a distinctiveness constraint to avoid trivial solutions.
• Demonstrate applicability to faces, textured spheres, and simple articulated objects.

Proposed method

• Formulate dense equivariant labelling Φ that maps image pixels to a low-dimensional object-centric space Z (homeomorphic to a sphere).
• Enforce equivariance: Φ(x, u) = Φ(gx, gu) for image deformations g (or gwarp on coordinates).
• Impose distinctiveness to prevent constant labellings, e.g., using a loss that aligns Φ with a max-overlap criterion on the sphere.
• Represent uncertainties by embedding labels in R3, enabling soft correspondences via a probabilistic softmax over inner products p(v|u; x, x′, Φ).
• Train with triplets (x, x′, g) from synthetic or true deformations and optimize either a log-likelihood loss Llog or a distance-based loss Ldist (γ-robust).
• Use two CNN architectures (SIMPLE and DILATIONS) to control receptive field and context for learning dense embeddings.

Experimental results

Research questions

• RQ1 Can a dense, object-centric coordinate frame be learned unsupervised from unlabelled images?
• RQ2 Is this dense embedding invariant to intra-object deformations and cross-instance variations within a category (e.g., faces)?
• RQ3 How does incorporating distinctiveness with equivariance affect learning compared to using equivariance alone?
• RQ4 Can the method scale from synthetic/toy objects to real-world objects like human faces and cats?
• RQ5 What is the effect of label-space dimensionality on capturing object geometry (e.g., 2D circle vs 3D sphere dependencies)?

Key findings

• A dense, object-centric coordinate frame Z can be learned unsupervised, yielding locally smooth label mappings that are equivariant to deformations.
• The combination of equivariance with distinctiveness prevents trivial constant solutions and enables meaningful embeddings.
• The distance-based loss (Ldist) with a 3D label space (L=3) produces more geometrically consistent embeddings than the log-likelihood loss (Llog) in several setups.
• Capsule center localization achieves low error: Llog = 0.97%, Ldist with γ=1 = 1.13%, Ldist with γ=0.5 = 1.14% (as percent of image width).
• On arms, textured spheres, faces, and cats, the method yields consistent object frames across deformations, with better performance for the DILATIONS architecture and distance-based loss in challenging cases (faces, AFLW, MAFL datasets).
• Nearest-neighbor/landmark regression on AFLW achieves competitive unsupervised-to-supervised performance, e.g., DILATIONS + Ldist, γ=0.5 reaching 8.80% error vs. supervised baselines.