[Paper Review] IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments

Girish Varma, Anbumani Subramanian+3 more|arXiv (Cornell University)|Nov 26, 2018

Advanced Neural Network Applications21 references4 citations

TL;DR

This paper introduces IDD, a large-scale, finely annotated dataset of 10,004 images from Indian roads to address the challenges of autonomous navigation in unstructured, diverse environments. It features 34 semantic classes, a novel four-level label hierarchy, and significantly higher within-class diversity than Cityscapes, leading to substantially lower performance for state-of-the-art semantic segmentation models—highlighting its role in advancing domain adaptation, few-shot learning, and behavior prediction in real-world driving scenarios.

ABSTRACT

While several datasets for autonomous navigation have become available in recent years, they tend to focus on structured driving environments. This usually corresponds to well-delineated infrastructure such as lanes, a small number of well-defined categories for traffic participants, low variation in object or background appearance and strict adherence to traffic rules. We propose IDD, a novel dataset for road scene understanding in unstructured environments where the above assumptions are largely not satisfied. It consists of 10,004 images, finely annotated with 34 classes collected from 182 drive sequences on Indian roads. The label set is expanded in comparison to popular benchmarks such as Cityscapes, to account for new classes. It also reflects label distributions of road scenes significantly different from existing datasets, with most classes displaying greater within-class diversity. Consistent with real driving behaviours, it also identifies new classes such as drivable areas besides the road. We propose a new four-level label hierarchy, which allows varying degrees of complexity and opens up possibilities for new training methods. Our empirical study provides an in-depth analysis of the label characteristics. State-of-the-art methods for semantic segmentation achieve much lower accuracies on our dataset, demonstrating its distinction compared to Cityscapes. Finally, we propose that our dataset is an ideal opportunity for new problems such as domain adaptation, few-shot learning and behaviour prediction in road scenes.

Motivation & Objective

• To address the limitations of existing datasets that focus on structured, rule-bound driving environments in Europe and North America.
• To provide a large-scale, high-quality dataset capturing the visual and behavioral diversity of real-world unstructured roads, particularly in regions like India with complex traffic and infrastructure.
• To enable new research in domain adaptation, few-shot learning, and behavior prediction by introducing a dataset with high within-class variation and rare or ambiguous categories.
• To develop a multi-level label hierarchy that supports varying degrees of segmentation complexity and reduces ambiguity in real-world scene understanding.
• To empirically demonstrate the challenge of generalization by showing that SOTA models trained on Cityscapes perform significantly worse on IDD, indicating the need for more robust and diverse training data.

Proposed method

• Collection of 182 drive sequences across diverse Indian urban and rural roads, capturing extreme variations in lighting, weather, traffic density, and road infrastructure.
• Fine instance-level annotation of 10,004 images with 34 semantic classes, including novel categories such as autorickshaws, billboards, curbs, and drivable/non-drivable areas.
• Design of a four-level hierarchical labeling system (7, 16, 26, 30 labels) to support multi-granularity learning and reduce ambiguity in complex scenes.
• Inclusion of semantic cues for drivable areas beyond traditional road/lane labels, reflecting real-world navigation decisions influenced by dynamic and static obstacles.
• Evaluation of state-of-the-art models (e.g., DeepLabV3, Mask R-CNN) on both semantic and instance segmentation benchmarks using mIoU and AP metrics.
• Conducting ablation studies and confusion matrix analysis to identify failure modes and class-level challenges, such as high confusion between motorcycles/bicycles and billboards/traffic signs.

Experimental results

Research questions

• RQ1 How does the performance of state-of-the-art semantic segmentation models trained on Cityscapes generalize to a more diverse, unstructured driving environment like that in India?
• RQ2 To what extent does the high within-class variation in object appearance and background complexity in IDD challenge current semantic segmentation models compared to structured benchmarks?
• RQ3 How effective is the proposed four-level label hierarchy in reducing ambiguity and enabling multi-granularity learning in complex road scenes?
• RQ4 What are the key failure modes and confusions in model predictions on IDD, and how do they relate to class-level characteristics such as pixel count and visual similarity?
• RQ5 In what ways can IDD serve as a benchmark for emerging learning paradigms such as few-shot learning and domain adaptation in autonomous navigation?

Key findings

• State-of-the-art semantic segmentation models fine-tuned on IDD achieve significantly lower mIoU scores (e.g., 74.3% at level 4) compared to their performance on Cityscapes, indicating a substantial domain shift and higher difficulty.
• The model trained on IDD shows superior generalization in unstructured settings, correctly identifying muddy or non-standard drivable areas that are often missed by Cityscapes-trained models.
• Low IoU values (<25%) for classes like bicycle, traffic light, and fence are primarily due to low pixel counts, highlighting data scarcity issues for rare categories.
• Significant confusion exists between visually similar classes such as motorcycle and bicycle, billboard and traffic sign, and vegetation with wall, pole, and fence, indicating challenges in fine-grained recognition.
• Instance segmentation models on IDD achieve AP scores of up to 0.376 (PANet) and 0.268 (Mask R-CNN), which are substantially lower than those on standard benchmarks, underscoring the dataset's difficulty.
• The dataset reveals that drivable area prediction is highly ambiguous and context-dependent, requiring semantic cues beyond geometric or lane-based definitions, which challenges traditional segmentation approaches.