[Paper Review] Learning a Hierarchical Compositional Shape Vocabulary for Multi-class Object Representation

Sanja Fidler, Marko Boben+1 more|arXiv (Cornell University)|Aug 23, 2014

Advanced Image and Video Retrieval Techniques10 references20 citations

TL;DR

This paper proposes an unsupervised, bottom-up framework for learning a hierarchical compositional shape vocabulary from oriented contour fragments, recursively combining them into increasingly complex, class-specific shape compositions. The method achieves state-of-the-art detection performance with logarithmic growth in vocabulary size and inference complexity, enabling scalable multi-class object recognition with fast inference and short training times.

ABSTRACT

Hierarchies allow feature sharing between objects at multiple levels of representation, can code exponential variability in a very compact way and enable fast inference. This makes them potentially suitable for learning and recognizing a higher number of object classes. However, the success of the hierarchical approaches so far has been hindered by the use of hand-crafted features or predetermined grouping rules. This paper presents a novel framework for learning a hierarchical compositional shape vocabulary for representing multiple object classes. The approach takes simple contour fragments and learns their frequent spatial configurations. These are recursively combined into increasingly more complex and class-specific shape compositions, each exerting a high degree of shape variability. At the top-level of the vocabulary, the compositions are sufficiently large and complex to represent the whole shapes of the objects. We learn the vocabulary layer after layer, by gradually increasing the size of the window of analysis and reducing the spatial resolution at which the shape configurations are learned. The lower layers are learned jointly on images of all classes, whereas the higher layers of the vocabulary are learned incrementally, by presenting the algorithm with one object class after another. The experimental results show that the learned multi-class object representation scales favorably with the number of object classes and achieves a state-of-the-art detection performance at both, faster inference as well as shorter training times.

Motivation & Objective

• To develop a scalable, multi-class object representation that captures complex shape structures without manual labeling.
• To address the limitations of flat, bag-of-words models by introducing hierarchical, compositional shape modeling.
• To enable feature sharing across object classes at multiple levels of abstraction for improved generalization and efficiency.
• To learn the shape vocabulary in a bottom-up, statistical manner, minimizing human supervision and avoiding hand-crafted features or fixed grouping rules.

Proposed method

• The method learns simple oriented contour fragments as the base level and identifies their frequent spatial configurations.
• Compositions are recursively built by combining lower-level parts using spatial relations modeled as Gaussians, forming increasingly complex hierarchical structures.
• Lower layers are trained jointly on all object classes to capture generic shape structures, while higher layers are learned incrementally per class.
• The analysis window size increases and spatial resolution decreases with each layer, enabling multi-scale shape modeling.
• Each composition is a generative probabilistic model that captures distribution over parts from the previous layer, enabling deformation modeling.
• The framework uses a hierarchical, bottom-up learning process that scales efficiently with the number of object classes.

Experimental results

Research questions

• RQ1 Can a hierarchical, compositional shape vocabulary be learned in an unsupervised manner from simple contour fragments to represent multiple object classes?
• RQ2 How does hierarchical composition improve generalization and inference efficiency compared to flat representations in multi-class object detection?
• RQ3 To what extent can shared features across classes reduce vocabulary size and training time while maintaining high detection accuracy?
• RQ4 Can the method scale effectively with increasing numbers of object classes, maintaining fast inference and compact representation?

Key findings

• The method achieves state-of-the-art detection performance on multiple object classes, including bottle, giraffe, mug, and car variants.
• Inference time grows logarithmically with the number of classes, significantly outperforming flat approaches.
• The vocabulary size grows logarithmically in lower layers, enabling scalable representation even as the number of classes increases.
• The model achieves high detection accuracy: 97.5% detection rate at 0.4 FPPI for cars (front view), and 96.9% for cows.
• The framework demonstrates strong generalization, with 93.0% recall at EER for face detection and 85.0% for person detection.
• The approach enables fast training and inference, with no need for manual part labeling or predefined grouping rules.