[Paper Review] Scene Parsing with Multiscale Feature Learning, Purity Trees, and Optimal Covers

Clément Farabet, Camille Couprie+2 more|arXiv (Cornell University)|Feb 9, 2012

Advanced Image and Video Retrieval TechniquesComputer Science26 references135 citations

TL;DR

This paper proposes a fast, end-to-end scene parsing system that uses multiscale convolutional features, a segmentation tree built from pixel dissimilarities, and an optimal cover algorithm to select pure segments for labeling. It achieves state-of-the-art accuracy on Stanford Background (79.5% per-pixel), SIFT Flow (78.5%), and Barcelona (67.8%) datasets while processing a 320×240 image in under one second.

ABSTRACT

Scene parsing, or semantic segmentation, consists in labeling each pixel in an image with the category of the object it belongs to. It is a challenging task that involves the simultaneous detection, segmentation and recognition of all the objects in the image. The scene parsing method proposed here starts by computing a tree of segments from a graph of pixel dissimilarities. Simultaneously, a set of dense feature vectors is computed which encodes regions of multiple sizes centered on each pixel. The feature extractor is a multiscale convolutional network trained from raw pixels. The feature vectors associated with the segments covered by each node in the tree are aggregated and fed to a classifier which produces an estimate of the distribution of object categories contained in the segment. A subset of tree nodes that cover the image are then selected so as to maximize the average "purity" of the class distributions, hence maximizing the overall likelihood that each segment will contain a single object. The convolutional network feature extractor is trained end-to-end from raw pixels, alleviating the need for engineered features. After training, the system is parameter free. The system yields record accuracies on the Stanford Background Dataset (8 classes), the Sift Flow Dataset (33 classes) and the Barcelona Dataset (170 classes) while being an order of magnitude faster than competing approaches, producing a 320 \ imes 240 image labeling in less than 1 second.

Motivation & Objective

• Address the challenge of simultaneous detection, segmentation, and recognition in semantic segmentation by leveraging multi-scale contextual features.
• Eliminate the need for hand-crafted features by training a convolutional network end-to-end from raw pixels.
• Improve segmentation consistency by selecting a subset of tree nodes that maximizes segment purity using an optimal cover algorithm.
• Achieve high accuracy and speed by combining hierarchical segmentation with efficient feature aggregation and classification.

Proposed method

• Construct a multi-scale contrast-normalized Laplacian pyramid from raw input images to capture diverse spatial contexts.
• Apply a two-stage convolutional network to each pyramid scale, producing dense, multi-scale feature maps that are upsampled and concatenated for each pixel.
• Build a segmentation tree via minimum spanning tree of a pixel dissimilarity graph, where edges represent color-based dissimilarity between neighboring pixels.
• Aggregate feature vectors within each tree node using a 5×5 spatial grid, applying component-wise max pooling to create scale-invariant segment representations.
• Train a classifier on aggregated feature grids to estimate the class distribution (and thus entropy-based impurity) for each segment.
• Select an optimal cover of tree nodes—using a greedy algorithm that minimizes average segment impurity (entropy)—to produce a globally consistent, pure segmentation.

Experimental results

Research questions

• RQ1 Can a multiscale convolutional network trained end-to-end from raw pixels produce effective features for scene parsing without engineered handcrafted features?
• RQ2 Can a segmentation tree derived from pixel dissimilarities encode meaningful image segments that support accurate semantic labeling?
• RQ3 Does an optimal cover based on minimizing segment impurity (entropy) yield better segmentation consistency than traditional inference methods like graph cuts?
• RQ4 Can the combination of multiscale features, tree-based segmentation, and purity-driven cover selection achieve state-of-the-art accuracy while maintaining sub-second inference speed?

Key findings

• The system achieved 79.5% per-pixel accuracy and 74.3% average per-class accuracy on the Stanford Background dataset, outperforming prior methods.
• On the SIFT Flow dataset, the method reached 78.5% per-pixel accuracy with balanced class sampling, significantly improving small-object recognition.
• For the Barcelona dataset with 170 classes, the system achieved 67.8% per-pixel accuracy when using balanced sampling, demonstrating robustness to high-class complexity.
• Inference on a 320×240 image took less than 1 second on a standard CPU, making it an order of magnitude faster than competing approaches.
• The system is parameter-free after training, requiring no threshold tuning or hyperparameter adjustment during inference.
• Frequency-balanced sampling improved recognition of rare classes but reduced overall pixel accuracy, highlighting a trade-off between global and per-class performance.