[Paper Review] Multi-domain semantic segmentation with pyramidal fusion

Petra Bevandić, Marin Oršić+3 more|arXiv (Cornell University)|Sep 2, 2020

Domain Adaptation and Few-Shot Learning19 references4 citations

TL;DR

This paper proposes a multi-domain semantic segmentation model using pyramidal feature fusion in a ResNet-152 backbone to handle diverse datasets with varying class distributions. It introduces a novel log-sum-prob loss with boundary-aware focal weighting, achieving state-of-the-art performance on out-of-distribution benchmarks, including 62.5% mIoU on VIPER and 45.4% on WildDash 2, while maintaining robustness across domains with minimal domain-specific adaptation.

ABSTRACT

We present our submission to the semantic segmentation contest of the Robust Vision Challenge held at ECCV 2020. The contest requires submitting the same model to seven benchmarks from three different domains. Our approach is based on the SwiftNet architecture with pyramidal fusion. We address inconsistent taxonomies with a single-level 193-dimensional softmax output. We strive to train with large batches in order to stabilize optimization of a hard recognition problem, and to favour smooth evolution of batchnorm statistics. We achieve this by implementing a custom backward step through log-sum-prob loss, and by using small crops before freezing the population statistics. Our model ranks first on the RVC semantic segmentation challenge as well as on the WildDash 2 leaderboard. This suggests that pyramidal fusion is competitive not only for efficient inference with lightweight backbones, but also in large-scale setups for multi-domain application.

Motivation & Objective

• To develop a single deep learning model capable of generalizing across multiple semantic segmentation datasets with differing class labels and distributions.
• To address domain shift and class ambiguity across datasets such as ADE20K, Cityscapes, VIPER, and ScanNet.
• To improve performance on out-of-distribution (OOD) data, especially in rare or overlapping class scenarios.
• To design a loss function that enhances model robustness to boundary errors and class overlap.
• To enable effective feature fusion across multi-scale representations for improved segmentation accuracy.

Proposed method

• The model employs a SwiftNet ResNet-152 backbone with pyramidal feature fusion to aggregate multi-scale contextual representations.
• A universal label set is constructed by merging, abstracting, or splitting overlapping or subset classes across datasets to unify semantic space.
• The proposed log-sum-prob loss combines negative log-likelihood with exponential weighting based on prediction confidence and boundary awareness.
• The loss function uses a dynamic weighting scheme: $ L^{ij} = -\alpha^{ij} e^{\gamma(1-p^{ij})} \log p^{ij} $, where $ p^{ij} $ is the sum of probabilities over all universal classes for pixel $ (i,j) $.
• Class-level merging and splitting are applied to resolve conflicts: exact matches are merged, subsets are abstracted, and overlapping classes are decomposed into disjoint components.
• Training uses data augmentation (horizontal flipping, scale jittering, cropping), Adam optimization with cosine annealing, and gradient accumulation on 6× V100 GPUs.

Experimental results

Research questions

• RQ1 Can a single deep learning model achieve strong performance across multiple, diverse semantic segmentation datasets with inconsistent class labels?
• RQ2 How can class-level conflicts—such as subset or overlapping classes—be resolved to unify multi-domain semantic segmentation?
• RQ3 To what extent does a boundary-aware, log-sum-prob loss improve segmentation robustness on out-of-distribution data?
• RQ4 Does pyramidal feature fusion enhance multi-scale context aggregation and improve generalization in cross-domain settings?
• RQ5 How does the model perform on rare or ambiguous classes, particularly in OOD scenarios like WildDash 2 and WDv2?

Key findings

• The model achieves 62.5% mIoU on the VIPER dataset, significantly outperforming the previous state-of-the-art (40.7%) on the same benchmark.
• On WildDash 2, the model reaches 45.4% mIoU, surpassing the next best method (37.9%) by 7.5 percentage points.
• The model demonstrates strong OOD generalization, with only 0.04% of pixels misclassified on MVD (out of 45.5B pixels), indicating high robustness.
• On ScanNet, the model achieves 54.6% mIoU, a 6.1% improvement over the baseline MSeg1080_RVC.
• The model maintains high inference speed, achieving 45 FPS at 384×384 input resolution during early training stages.
• The loss function effectively reduces errors on boundary regions, as evidenced by a 32.5% IoU on negative pixels in WildDash 2, indicating improved handling of ambiguous or rare classes.