[Paper Review] WEDGE: Web-Image Assisted Domain Generalization for Semantic Segmentation

Namyup Kim, Taeyoung Son+4 more|arXiv (Cornell University)|Sep 29, 2021

Domain Adaptation and Few-Shot Learning43 references4 citations

TL;DR

WEDGE proposes a web-image assisted domain generalization framework for semantic segmentation that leverages diverse, application-relevant web-crawled images to improve model generalization across unseen domains. By injecting web image styles into synthetic training images on-the-fly and using pseudo-labeled web images in a two-stage training process, WEDGE achieves state-of-the-art performance, outperforming existing methods by up to 5.5% on average across multiple real-world test domains.

ABSTRACT

Domain generalization for semantic segmentation is highly demanded in real applications, where a trained model is expected to work well in previously unseen domains. One challenge lies in the lack of data which could cover the diverse distributions of the possible unseen domains for training. In this paper, we propose a WEb-image assisted Domain GEneralization (WEDGE) scheme, which is the first to exploit the diversity of web-crawled images for generalizable semantic segmentation. To explore and exploit the real-world data distributions, we collect web-crawled images which present large diversity in terms of weather conditions, sites, lighting, camera styles, etc. We also present a method which injects styles of the web-crawled images into training images on-the-fly during training, which enables the network to experience images of diverse styles with reliable labels for effective training. Moreover, we use the web-crawled images with their predicted pseudo labels for training to further enhance the capability of the network. Extensive experiments demonstrate that our method clearly outperforms existing domain generalization techniques.

Motivation & Objective

• To address the challenge of domain generalization in semantic segmentation where models must generalize to unseen real-world domains without access to their data during training.
• To overcome the limitations of existing domain generalization methods that rely on limited style sources like ImageNet, which are often irrelevant to target applications.
• To leverage the diversity of real-world web-crawled images—covering weather, lighting, camera styles, and geolocation—for more effective and realistic domain generalization.
• To develop an efficient, on-the-fly style injection mechanism that transfers visual styles from web images to synthetic training data without requiring image-to-image translation.
• To enhance model generalization by using web-crawled images with predicted pseudo-labels in a two-stage training pipeline.

Proposed method

• Web-crawled images are collected using application-specific keywords (e.g., 'driving + road') to ensure relevance to target domains.
• A style injection module transfers feature-level styles from web images to synthetic training images in real time during training, using feature statistics from early residual blocks (1st and 2nd for ResNet, 2nd and 3rd for VGG).
• The training process is divided into two stages: in Stage 1, the model is trained with style injection using synthetic images; in Stage 2, web-crawled images with predicted pseudo-labels are added as additional training data.
• Pseudo-labels for web-crawled images are generated using the trained model from Stage 1, and a thresholding hyperparameter τ controls label confidence.
• The method avoids costly image-to-image translation by operating directly on feature maps, enabling efficient use of large-scale web image collections.
• Ablation studies confirm that injecting style at deeper layers (e.g., 4th residual block) degrades performance, justifying the use of early feature maps for style transfer.

Figure 1: Overall framework of WEDGE. (1) Crawling real and task-relevant images from the Web automatically. (2) Learning semantic segmentation while transferring feature statistics of web images to features of synthetic training images in the source domain. (3) Further training the model using both

Experimental results

Research questions

• RQ1 Can web-crawled images with diverse real-world characteristics improve domain generalization in semantic segmentation beyond synthetic data?
• RQ2 Does on-the-fly feature-level style injection from web images lead to better generalization than image-to-image translation or random style augmentation?
• RQ3 Can pseudo-labeled web images, even when semantically irrelevant, effectively improve model robustness when used with a two-stage training strategy?
• RQ4 How does the choice of web image keywords (e.g., 'driving + fog' vs. 'driving + road') affect model generalization performance?
• RQ5 What is the optimal set of feature layers for style injection to preserve semantic consistency while enhancing domain generalization?

Key findings

• WEDGE achieves state-of-the-art performance on three real-world test datasets (Cityscapes, BDD100K, Mapillary), with a 47.30% average mIoU on the Cityscapes test set when trained on GTA5 using ResNet101.
• The method outperforms prior domain generalization approaches by up to 5.5% average mIoU gain, demonstrating significant improvement in cross-domain generalization.
• Using the general keyword 'driving + road' for web image retrieval yields better performance (43.73% average mIoU) than domain-specific keywords like 'driving + fog' (43.39%)
• The two-stage training process with pseudo-labeling improves performance by 2.5–3.5% mIoU over the first stage alone, confirming the effectiveness of additional web image data.
• Ablation studies show that injecting style at deeper layers (e.g., 4th residual block) harms performance, indicating that early features are more suitable for style transfer.
• Thresholding pseudo-labels with τ = 0.05 yields the best balance between label quality and training stability, with performance dropping at lower thresholds (e.g., τ = 0.01).

Figure 2: Qualitative examples of web images and their pseudo labels generated by the segmentation model with ResNet101 backbone trained by the first stage of WEDGE. (a) Input images. (b) Pseudo segmentation labels. These images demonstrate large diversity of the web images, which is vital for achie