[Paper Review] Unsupervised part representation by Flow Capsules

Sara Sabour, Andrea Tagliasacchi+3 more|arXiv (Cornell University)|Nov 27, 2020

Image Processing and 3D Reconstruction40 references18 citations

TL;DR

This paper proposes FlowCapsules, an unsupervised method that learns primary capsule representations for object parts using motion as a self-supervisory signal. By leveraging optical flow between video frames and modeling depth ordering, FlowCapsules discovers disentangled shape masks, infers occluded regions, and outperforms prior methods in unsupervised part segmentation and image classification on cluttered, textured scenes.

ABSTRACT

Capsule networks aim to parse images into a hierarchy of objects, parts and relations. While promising, they remain limited by an inability to learn effective low level part descriptions. To address this issue we propose a way to learn primary capsule encoders that detect atomic parts from a single image. During training we exploit motion as a powerful perceptual cue for part definition, with an expressive decoder for part generation within a layered image model with occlusion. Experiments demonstrate robust part discovery in the presence of multiple objects, cluttered backgrounds, and occlusion. The part decoder infers the underlying shape masks, effectively filling in occluded regions of the detected shapes. We evaluate FlowCapsules on unsupervised part segmentation and unsupervised image classification.

Motivation & Objective

• To address the limitation of capsule networks in learning effective low-level part descriptors without supervision.
• To enable unsupervised discovery of atomic visual parts from single images using motion as a perceptual cue.
• To improve part segmentation and shape completion in the presence of occlusion, clutter, and texture variation.
• To disentangle shape, pose, and depth ordering in a capsule-based generative model.
• To outperform existing unsupervised part discovery methods on real-world and synthetic datasets.

Proposed method

• A Siamese network is trained to estimate optical flow between consecutive video frames, using capsule poses as a supervisory signal.
• The capsule encoder maps a single image to a set of primary capsules, each with a canonical shape mask, 3D pose (translation, rotation), and depth scalar.
• A differentiable decoder generates image frames from capsule parameters, modeling occlusion via depth-ordered layering.
• The model uses a flow reconstruction loss that enforces consistency between predicted capsule motions and observed optical flow.
• Depth ordering is explicitly modeled in the decoder to handle occlusion and improve shape completion.
• The framework is trained end-to-end in a self-supervised manner using only video frames, without ground-truth masks or labels.

Experimental results

Research questions

• RQ1 Can motion between video frames serve as an effective self-supervisory signal for discovering meaningful visual parts in unsupervised settings?
• RQ2 Can capsule-based representations with depth ordering improve shape completion and robustness to occlusion?
• RQ3 How does FlowCapsules compare to image reconstruction-based methods in part segmentation and classification under clutter and texture?
• RQ4 To what extent does the model disentangle shape, pose, and depth in the absence of supervision?
• RQ5 Does the method generalize across diverse object categories and complex scenes?

Key findings

• FlowCapsules achieves 94% IoU on the Geo dataset and 88% on Geo+ with 16 capsules, outperforming PSD and SCAE in unsupervised part segmentation.
• The model maintains strong performance even with reduced encoding length (e.g., 91% IoU at |s_k|=3), indicating robustness to compact shape representations.
• Omitting depth ordering causes a significant drop in performance (to 54% IoU), demonstrating its critical role in occlusion handling.
• Deeper decoders (6 layers) improve mask smoothness and IoU, especially for circular shapes, showing architectural benefits.
• FlowCapsules achieves 85.3% unsupervised image classification accuracy on CIFAR-100 with k-means clustering, outperforming SCAE.
• The model successfully infers shape masks that complete partially occluded regions, even when parts are not fully visible in any single frame.