[Paper Review] Cityscapes-Panoptic-Parts and PASCAL-Panoptic-Parts datasets for Scene Understanding

Panagiotis Meletis, Xiaoxiao Wen+3 more|arXiv (Cornell University)|Apr 16, 2020

Image Retrieval and Classification Techniques4 citations

TL;DR

This paper introduces Cityscapes-Panoptic-Parts and PASCAL-Panoptic-Parts, two novel datasets that extend panoptic segmentation with hierarchical part-level annotations for 23 and 193 part classes, respectively. The datasets use a compact, hierarchical labeling format compatible with existing panoptic formats, enabling multi-level scene understanding with minimal annotation overhead, and are publicly released with code for processing and visualization.

ABSTRACT

In this technical report, we present two novel datasets for image scene understanding. Both datasets have annotations compatible with panoptic segmentation and additionally they have part-level labels for selected semantic classes. This report describes the format of the two datasets, the annotation protocols, the merging strategies, and presents the datasets statistics. The datasets labels together with code for processing and visualization will be published at https://github.com/tue-mps/panoptic_parts.

Motivation & Objective

• To address the lack of consistent, multi-level annotations for scene understanding by creating datasets with unified semantic, instance, and part-level labels.
• To enable research in holistic (panoptic segmentation) and detailed (part-level segmentation, object detection) image understanding tasks on a single, conflict-free label hierarchy.
• To reduce annotation burden while maintaining high-quality, consistent part-level labels through a structured merging and labeling protocol.
• To provide a standardized, extensible hierarchical label format that supports semantic, instance, and part-level labeling in a single compact ID encoding.

Proposed method

• Extends the Cityscapes and PASCAL-Context datasets by adding manual part-level annotations for 23 part classes in Cityscapes and 193 in PASCAL, focusing on things classes (e.g., person, car).
• Develops a hierarchical label format using a 7-digit integer ID: first two digits for semantic class, next three for instance ID (zero-padded), and final two for part class ID (zero-padded), enabling compact, unambiguous encoding.
• Applies a merging strategy that preserves original Cityscapes panoptic labels while integrating new part-level annotations without conflicts, minimizing annotation time to an average of 7 minutes per image.
• Uses a void class convention at each level: id=0 for semantic level (PASCAL), id=000 for parts level per instance to denote unlabeled parts, reducing unnecessary void pixels.
• Implements a hierarchical label encoding formula: id = semantic_id * 10^5 + instance_id * 10^2 + part_id, ensuring backward compatibility and extensibility.
• Releases the datasets and associated code at github.com/tue-mps/panoptic_parts for reproducibility and community use.

Experimental results

Research questions

• RQ1 Can a unified, conflict-free dataset be created that supports panoptic segmentation and part-level segmentation simultaneously on the same image set?
• RQ2 How can part-level annotations be efficiently integrated into existing panoptic datasets without disrupting original labels or increasing annotation time?
• RQ3 What is the optimal hierarchical label format that supports semantic, instance, and part-level labeling in a compact, extensible, and backward-compatible way?
• RQ4 How does the inclusion of part-level annotations affect the distribution and granularity of object understanding in complex urban and everyday scenes?
• RQ5 What is the impact of part-level annotations on downstream tasks such as instance-wise object parsing and fine-grained scene understanding?

Key findings

• Cityscapes-Panoptic-Parts extends the original Cityscapes dataset with part-level annotations for 23 part classes across 8 things classes, with 2975 training and 500 validation images.
• PASCAL-Panoptic-Parts merges PASCAL-Parts and PASCAL-Context to create a unified dataset with 100 stuff classes, 20 things classes, and 193 part classes, using 4998 training and 5105 validation images.
• The average annotation time per image for part-level labeling was reduced to 7 minutes through a structured merging and labeling protocol.
• The hierarchical label format enables compact, unambiguous encoding of semantic, instance, and part-level labels in a single 7-digit integer ID, supporting future extensions.
• The datasets are released with full code for processing, visualization, and evaluation, ensuring reproducibility and community adoption.
• The part-level annotations are confined to things classes, though the framework is designed to support future extension to stuff classes.