[Paper Review] Superpixel Convolutional Networks using Bilateral Inceptions

Raghudeep Gadde, Varun Jampani+3 more|arXiv (Cornell University)|Nov 20, 2015

Advanced Neural Network Applications29 references4 citations

TL;DR

This paper proposes Bilateral Inception (BI) modules—learned, edge-aware filters that propagate features between superpixels at multiple scales in CNNs for semantic segmentation. By inserting BI modules between 1×1 convolution layers, the method improves segmentation accuracy over baseline CNNs and DenseCRFs while maintaining full-resolution outputs and faster inference.

ABSTRACT

In this paper we propose a CNN architecture for semantic image segmentation. We introduce a new 'bilateral inception' module that can be inserted in existing CNN architectures and performs bilateral filtering, at multiple feature-scales, between superpixels in an image. The feature spaces for bilateral filtering and other parameters of the module are learned end-to-end using standard backpropagation techniques. The bilateral inception module addresses two issues that arise with general CNN segmentation architectures. First, this module propagates information between (super) pixels while respecting image edges, thus using the structured information of the problem for improved results. Second, the layer recovers a full resolution segmentation result from the lower resolution solution of a CNN. In the experiments, we modify several existing CNN architectures by inserting our inception module between the last CNN (1x1 convolution) layers. Empirical results on three different datasets show reliable improvements not only in comparison to the baseline networks, but also in comparison to several dense-pixel prediction techniques such as CRFs, while being competitive in time.

Motivation & Objective

• Address the limitations of standard CNNs in capturing structured output relationships in semantic segmentation.
• Overcome the need for post-processing upsampling and CRF refinement by integrating edge-aware information propagation directly into the network architecture.
• Leverage superpixels as a structured, image-adaptive layout to improve feature propagation while preserving spatial and photometric consistency.
• Enable end-to-end learning of bilateral filtering parameters and feature spaces within a CNN framework.
• Achieve full-resolution segmentation outputs without additional upscaling or CRF post-processing steps.

Proposed method

• Introduce a novel 'bilateral inception' (BI) module that applies bilateral filtering across superpixels at multiple feature scales.
• Use superpixels as the spatial layout for feature propagation, replacing standard grid-based convolutions in the final layers.
• Learn the feature spaces and kernel parameters of the bilateral filters end-to-end via backpropagation.
• Insert BI modules between the final 1×1 convolution (FC) layers of existing CNN architectures such as DeepLab and AlexNet.
• Perform edge-aware information propagation using Gaussian kernels that consider both spatial proximity and color similarity.
• Recover full-resolution segmentation masks directly from low-resolution CNN features without interpolation or CRF refinement.

Experimental results

Research questions

• RQ1 Can learned bilateral filtering between superpixels improve semantic segmentation performance compared to standard CNNs and CRF post-processing?
• RQ2 Does using superpixels as a structured, image-adaptive layout enhance feature propagation while preserving object boundaries?
• RQ3 Can end-to-end learning of bilateral filter parameters in a CNN framework outperform fixed or hand-designed CRF inference strategies?
• RQ4 To what extent does the BI module reduce the need for separate upsampling or CRF refinement steps in semantic segmentation?
• RQ5 How does the BI module compare in accuracy and speed to DenseCRF and other dense prediction techniques on benchmark datasets?

Key findings

• The BI module achieves a 66.9% IoU on the Cityscapes validation set when added to the DeepLab model, outperforming the base model (65.7%) and the DeepLab + DenseCRF baseline (66.6%).
• On the PASCAL VOC 2012 dataset, the BI module improved the baseline DeepLab model from 62.2% to 63.1% IoU on half-resolution inputs and from 65.7% to 66.9% on full-resolution inputs.
• The BI module achieved better performance than DenseCRF on all three datasets tested, including Cityscapes and PASCAL VOC, while being significantly faster (e.g., 6.1s vs. 6.9s runtime on Cityscapes).
• The method produces sharper segmentation boundaries due to edge-aware propagation, avoiding the blurring common in bilinear interpolation and some CRF methods.
• The majority of runtime is attributed to superpixel computation (5.2s on Cityscapes), but the BI inference itself is efficient and scalable with the number of superpixels.
• The BI module is modular and can be inserted into existing CNN architectures without architectural overhaul, demonstrating broad applicability and ease of integration.