[Paper Review] Convolutional Oriented Boundaries: From Image Segmentation to High-Level Tasks

Kevis-Kokitsi Maninis, Jordi Pont-Tuset+2 more|arXiv (Cornell University)|Jan 17, 2017

Advanced Neural Network Applications48 references4 citations

TL;DR

This paper proposes Convolutional Oriented Boundaries (COB), a computationally efficient method that generates multiscale, oriented contours and hierarchical region segmentations from a single forward pass of a pre-trained CNN. COB achieves state-of-the-art performance in contour detection, semantic segmentation, and object proposal generation across multiple benchmarks, including PASCAL and MS-COCO, by jointly predicting contour strength and orientation using a novel sparse boundary representation.

ABSTRACT

We present Convolutional Oriented Boundaries (COB), which produces multiscale oriented contours and region hierarchies starting from generic image classification Convolutional Neural Networks (CNNs). COB is computationally efficient, because it requires a single CNN forward pass for multi-scale contour detection and it uses a novel sparse boundary representation for hierarchical segmentation; it gives a significant leap in performance over the state-of-the-art, and it generalizes very well to unseen categories and datasets. Particularly, we show that learning to estimate not only contour strength but also orientation provides more accurate results. We perform extensive experiments for low-level applications on BSDS, PASCAL Context, PASCAL Segmentation, and NYUD to evaluate boundary detection performance, showing that COB provides state-of-the-art contours and region hierarchies in all datasets. We also evaluate COB on high-level tasks when coupled with multiple pipelines for object proposals, semantic contours, semantic segmentation, and object detection on MS-COCO, SBD, and PASCAL; showing that COB also improves the results for all tasks.

Motivation & Objective

• To develop a generic, efficient framework that transforms pre-trained CNNs into high-quality contour and region detectors for both low- and high-level vision tasks.
• To overcome limitations of existing boundary detection methods by jointly predicting contour strength and orientation, improving accuracy and generalization.
• To eliminate the need for computationally expensive globalization steps in contour grouping by leveraging high-level features from deep CNNs.
• To demonstrate the modularity and transferability of the approach by applying it to diverse datasets and tasks, including semantic segmentation and object detection.
• To provide a publicly available, reusable framework that enables future improvements in perceptual grouping through advances in base CNN architectures.

Proposed method

• COB uses a single forward pass through a pre-trained classification CNN (e.g., ResNet) to extract multiscale feature maps for contour prediction.
• It introduces side branches that predict both contour strength and orientation at multiple scales, enabling multiscale, oriented contour detection.
• A novel sparse boundary representation is proposed to efficiently encode contour signals and enable fast hierarchical region grouping via Ultrametric Contour Maps (UCMs).
• The method fuses multiscale UCMs into a single hierarchical segmentation structure using a combinatorial grouping algorithm.
• COB’s output is compatible with downstream pipelines such as object proposal generation, semantic segmentation, and object detection, where it improves performance without retraining.
• The framework is designed to be modular, allowing seamless integration with new base CNN architectures as they emerge.

Experimental results

Research questions

• RQ1 Can joint prediction of contour strength and orientation significantly improve boundary detection accuracy compared to methods that only predict strength?
• RQ2 Does a single CNN forward pass suffice to generate high-quality, multiscale, oriented contours and hierarchical regions without additional post-processing?
• RQ3 Can COB eliminate the need for global contour regularization (e.g., spectral partitioning) when high-level features are available?
• RQ4 To what extent does COB generalize to unseen categories and datasets, especially in high-level vision tasks like object detection and semantic segmentation?
• RQ5 How does COB’s sparse boundary representation compare to dense representations in terms of efficiency and segmentation quality?

Key findings

• COB achieves state-of-the-art performance in boundary detection on BSDS500, PASCAL Context, PASCAL VOC, and NYUD, with a 0.8-second inference time per image.
• On PASCAL VOC 2012, COB object proposals improve Fast-RCNN object detection mAP by 1.5 percentage points (from 76.7% to 78.2%) without retraining.
• In semantic segmentation, COB superpixels improve boundary accuracy, especially in reducing false positives and refining edge regions, as shown in qualitative results on VOC and PASCAL datasets.
• COB generalizes effectively to MS-COCO, where it produces competitive object proposals without retraining, demonstrating strong zero-shot transferability.
• The method eliminates the need for spectral graph partitioning globalization, reducing computational cost while maintaining or improving performance.
• COB achieves a new state of the art in semantic boundary detection, with mAP of 76.0 on VOC 2007, outperforming prior methods.