[Paper Review] Bayesian Semantic Instance Segmentation in Open Set World

Trung Pham, Vijay Kumar B G+3 more|arXiv (Cornell University)|Jun 4, 2018

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a Bayesian semantic instance segmentation method for open-set environments that segments both known and unknown objects using only bounding box annotations for known classes. By formulating the problem in a Bayesian framework with a simulated annealing-optimized image partition sampler based on a boundary-driven region hierarchy, the method achieves competitive performance on known classes and outperforms unsupervised methods on unknown objects.

ABSTRACT

This paper addresses the semantic instance segmentation task in the open-set conditions, where input images can contain known and unknown object classes. The training process of existing semantic instance segmentation methods requires annotation masks for all object instances, which is expensive to acquire or even infeasible in some realistic scenarios, where the number of categories may increase boundlessly. In this paper, we present a novel open-set semantic instance segmentation approach capable of segmenting all known and unknown object classes in images, based on the output of an object detector trained on known object classes. We formulate the problem using a Bayesian framework, where the posterior distribution is approximated with a simulated annealing optimization equipped with an efficient image partition sampler. We show empirically that our method is competitive with state-of-the-art supervised methods on known classes, but also performs well on unknown classes when compared with unsupervised methods.

Motivation & Objective

• Address the challenge of semantic instance segmentation in open-set conditions where unknown object classes may appear.
• Reduce reliance on expensive instance mask annotations by enabling training with only bounding box supervision for known classes.
• Develop a global segmentation framework that groups perceptually coherent regions, assigning them to known detections or unknown object classes.
• Enable robust segmentation of both detected and miss-detected known objects, as well as unknown objects and stuff classes (e.g., sky, grass).

Proposed method

• Formulate instance segmentation as a Bayesian inference problem, where the posterior distribution over image partitions is approximated via simulated annealing.
• Use a boundary-driven region hierarchy generated from deep neural network-based edge detection to efficiently sample image partitions.
• Integrate three competing likelihood models: boundary model (for unknown regions), geometric bounding box model (for known detections), and optional mask model (for known instances).
• Apply a prior model that penalizes excessive region count and promotes object compactness.
• Employ a simulated annealing optimization to explore the large partition space and converge to high-quality segmentations.
• Use a hierarchical region selection mechanism to naturally perform split, merge, or split-and-merge moves during sampling.

Experimental results

Research questions

• RQ1 Can a Bayesian instance segmentation framework achieve competitive performance on known object classes using only bounding box supervision?
• RQ2 Can the method effectively segment unknown object classes and stuff regions without any supervision on those classes?
• RQ3 How does the proposed image partition sampler compare in efficiency and quality to standard Gibbs sampling in high-dimensional partition spaces?
• RQ4 To what extent does the boundary-driven region hierarchy improve segmentation accuracy and robustness in open-set scenarios?

Key findings

• The proposed method achieves mIoU scores competitive with fully supervised Mask R-CNN on known object classes, despite training only with bounding box annotations.
• The method successfully segments unknown object classes and stuff regions (e.g., grass, sky) with high accuracy, outperforming existing unsupervised methods on these classes.
• The simulated annealing-based image partition sampler enables efficient exploration of the large partition space, achieving high-quality segmentations faster than standard Gibbs sampling.
• The boundary-driven region hierarchy significantly improves segmentation quality by providing structured, perceptually meaningful region proposals.
• The method generalizes well to unseen object categories, demonstrating robustness in open-set conditions where state-of-the-art supervised models fail.
• Post-processing of Mask R-CNN predictions via pixel voting confirms that the proposed method’s global, pixelwise labeling strategy is more coherent and accurate than instance-wise mask outputs.