[Paper Review] Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

Philipp Krähenbühl, Vladlen Koltun|arXiv (Cornell University)|Oct 20, 2012

Sparse and Compressive Sensing Techniques23 references2,974 citations

TL;DR

The paper develops a highly efficient mean-field inference method for fully connected CRFs with Gaussian edge potentials, enabling pixel-level dense connectivity and significant accuracy gains in image segmentation while achieving fast runtimes.

ABSTRACT

Most state-of-the-art techniques for multi-class image segmentation and labeling use conditional random fields defined over pixels or image regions. While region-level models often feature dense pairwise connectivity, pixel-level models are considerably larger and have only permitted sparse graph structures. In this paper, we consider fully connected CRF models defined on the complete set of pixels in an image. The resulting graphs have billions of edges, making traditional inference algorithms impractical. Our main contribution is a highly efficient approximate inference algorithm for fully connected CRF models in which the pairwise edge potentials are defined by a linear combination of Gaussian kernels. Our experiments demonstrate that dense connectivity at the pixel level substantially improves segmentation and labeling accuracy.

Motivation & Objective

• Motivate dense (fully connected) CRFs for pixel-level image segmentation to improve boundary accuracy and labeling consistency.
• Develop a scalable approximate inference algorithm enabling MAP inference in fully connected CRFs.
• Learn model parameters (unary, pairwise kernels, and compatibility) effectively from data.

Proposed method

• Use a mean-field approximation to approximate the CRF distribution as a product of independent marginals.
• Formulate pairwise potentials as a linear combination of Gaussian kernels in an arbitrary feature space.
• Perform message passing as Gaussian filtering in feature space, reducing complexity from quadratic to linear in the number of variables.
• Employ high-dimensional filtering with the permutohedral lattice (after whitening the feature space) to implement efficient convolutions.
• Learn kernel weights and compatibility functions via piecewise training and gradient-based optimization (L-BFGS for compatibility, grid search for kernel widths).
• Use a Potts or learned symmetric compatibility function to model label interactions.

Experimental results

Research questions

• RQ1 Does a fully connected CRF with Gaussian edge potentials improve pixel-level segmentation accuracy compared to sparser or region-based models?
• RQ2 Can mean-field inference with high-dimensional Gaussian filtering provide practical runtimes for dense pixel graphs without sacrificing accuracy?
• RQ3 How do long-range connections, kernel widths, and label compatibilities affect segmentation quality and boundary precision?

Key findings

• Dense pixel-level connectivity in CRFs yields improved segmentation accuracy over grid and higher-order methods on MSRC-21 and PASCAL VOC 2010 datasets.
• The proposed mean-field inference runs in about 0.2 seconds on a single CPU core, versus hours for alternative inference methods like MCMC or graph cuts.
• Long-range connections (larger spatial and color ranges) increase accuracy up to a point, with more than 50% of pairwise energy assigned to edges at distances ≥35 pixels in some settings; however, excessive long-range propagation can mislead labeling.
• On MSRC-21, the fully connected CRF with learned label compatibility achieved 88.2% global and 84.7% average accuracy (vs 84.0%/76.6% with unary alone).
• On PASCAL VOC 2010, the fully connected model with Potts potentials achieved 29.1% average accuracy, improving to 30.2% with learned label compatibility (outperforming the grid CRF’s 28.3%).