[Paper Review] Learning to Fuse Things and Stuff

Jie Li, Allan Raventos+3 more|arXiv (Cornell University)|Dec 4, 2018

Domain Adaptation and Few-Shot Learning38 references99 citations

TL;DR

The paper presents TASCNet, an end-to-end network for panoptic segmentation that fuses things (instances) and stuff (semantic) predictions via a Things and Stuff Consistency (TASC) constraint, achieving competitive results with a unified model.

ABSTRACT

We propose an end-to-end learning approach for panoptic segmentation, a novel task unifying instance (things) and semantic (stuff) segmentation. Our model, TASCNet, uses feature maps from a shared backbone network to predict in a single feed-forward pass both things and stuff segmentations. We explicitly constrain these two output distributions through a global things and stuff binary mask to enforce cross-task consistency. Our proposed unified network is competitive with the state of the art on several benchmarks for panoptic segmentation as well as on the individual semantic and instance segmentation tasks.

Motivation & Objective

• Motivate a unified approach to panoptic segmentation that combines semantic and instance segmentation into a single model.
• Leverage shared features and cross-task constraints to improve both individual tasks and the joint panoptic output.
• Propose a differentiable mechanism (TASC) to keep things and stuff outputs aligned during training.
• Demonstrate efficiency and competitive performance on Cityscapes, Mapillary Vistas, and COCO datasets.

Proposed method

• Use a ResNet50 backbone with Feature Pyramid Network (FPN) to share features between Things Head and Stuff Head.
• Implement a Stuff Head that densely predicts all stuff classes plus a single 'things' class as needed.
• Implement a Things Head using Region-based CNN heads with an RoIAlign-based pooling strategy for instance segmentation.
• Introduce Things and Stuff Consistency (TASC) loss to minimize residuals between the binary masks of the two tasks in a differentiable way (via RoI-Flatten for Things and a thresholded Stuff mask).
• Apply Mask-Guided Fusion to produce a final panoptic output by combining instance predictions with semantic predictions under the guidance of the learned masks.

Experimental results

Research questions

• RQ1 Can a single, unified network with shared backbone improve panoptic segmentation by exploiting the relation between things and stuff?
• RQ2 Does enforcing cross-task consistency via a differentiable TASC constraint stabilize training and improve accuracy for both semantic and instance segmentation tasks?
• RQ3 How does a single-network panoptic model compare to separate, independently trained models with late fusion across standard benchmarks?
• RQ4 What is the impact of ontology choices (full vs collapsed) in the Stuff head on final panoptic performance?

Key findings

• TASCNet achieves competitive panoptic quality (PQ) on Cityscapes, Mapillary Vistas, and COCO with a ResNet-50 backbone.
• Adding the TASC loss (λ > 0) improves PQ and related metrics over the non-TASC baseline, with λ = 1 yielding strong results.
• A full ontology in the Stuff head (predicting both N stuff and M thing classes) improves performance compared to collapsing things into a single ‘thing’ class.
• Joint training with both heads (Things and Stuff) tends to converge to better minima than staged training.
• On Cityscapes, TASCNet variants reach PQs close to or matching state-of-the-art baselines with significantly fewer parameters.