[Paper Review] What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation

Benedikt Blumenstiel, Johannes Jakubik+2 more|arXiv (Cornell University)|Jun 27, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper introduces the MESS benchmark for multi-domain zero-shot semantic segmentation, evaluating eight state-of-the-art models across 22 diverse datasets from medicine, earth monitoring, agriculture, and engineering. It reveals that model performance is significantly influenced by class semantic similarity and sensor type, with text-to-mask models outperforming vision-language models combined with SAM when using oracle bounding boxes.

ABSTRACT

While semantic segmentation has seen tremendous improvements in the past, there are still significant labeling efforts necessary and the problem of limited generalization to classes that have not been present during training. To address this problem, zero-shot semantic segmentation makes use of large self-supervised vision-language models, allowing zero-shot transfer to unseen classes. In this work, we build a benchmark for Multi-domain Evaluation of Semantic Segmentation (MESS), which allows a holistic analysis of performance across a wide range of domain-specific datasets such as medicine, engineering, earth monitoring, biology, and agriculture. To do this, we reviewed 120 datasets, developed a taxonomy, and classified the datasets according to the developed taxonomy. We select a representative subset consisting of 22 datasets and propose it as the MESS benchmark. We evaluate eight recently published models on the proposed MESS benchmark and analyze characteristics for the performance of zero-shot transfer models. The toolkit is available at https://github.com/blumenstiel/MESS.

Motivation & Objective

• To address the lack of holistic evaluation across diverse domains in zero-shot semantic segmentation, particularly for out-of-domain generalization.
• To identify key task-level characteristics—such as class semantics and sensor type—that influence zero-shot model performance.
• To provide a standardized benchmark (MESS) for evaluating zero-shot semantic segmentation models beyond in-domain settings.
• To compare text-to-mask, point-to-mask, and box-to-mask approaches, especially in combination with SAM and grounding models.
• To guide future model design by analyzing how vocabulary choice and prompt engineering affect segmentation accuracy.

Proposed method

• The authors curated and classified 120 datasets into a taxonomy based on domain, sensor type, and annotation style, selecting 22 representative datasets for the MESS benchmark.
• They evaluated eight recent zero-shot semantic segmentation models, including CAT-Seg-L, SAN-L, OVSeg-L, and Grounded-SAM, across the MESS benchmark.
• The evaluation focuses on open-vocabulary semantic segmentation (OVSS), comparing text-to-mask models with vision-language models like CLIP and SAM-based approaches.
• The study includes both quantitative metrics (e.g., mIoU) and qualitative analysis on predictions across domains such as medical imaging, satellite, and engineering.
• Oracle versions of SAM were used to isolate the impact of localization accuracy, comparing ground-truth bounding boxes with predicted ones from Grounding DINO.
• The authors analyze the effect of textual prompt design, comparing generic vs. domain-specific class labels on segmentation performance.

Figure 2: mIoU results for large models on a log scale. The datasets are grouped by their domain and sorted by supervised performance.

Experimental results

Research questions

• RQ1 How do zero-shot semantic segmentation models perform across diverse, real-world domains such as medicine, earth observation, and agriculture?
• RQ2 What task-level characteristics—such as class semantic similarity or sensor modality—most significantly affect model performance?
• RQ3 How does the choice of textual prompt (e.g., generic vs. domain-specific terminology) influence segmentation accuracy?
• RQ4 How do text-to-mask models compare to vision-language models combined with SAM in terms of zero-shot generalization?
• RQ5 To what extent does accurate object localization (via bounding boxes) improve zero-shot segmentation performance?

Key findings

• Text-to-mask models like CAT-Seg-L outperform Grounded-SAM with predicted bounding boxes, even though the latter achieves strong performance with oracle bounding boxes.
• The oracle SAM implementation, using perfect localization, outperforms many supervised semantic segmentation models, highlighting the importance of localization accuracy.
• Models trained on natural visual spectra struggle significantly with thermal or non-visible sensor data, indicating a major challenge for cross-sensor generalization.
• Semantic similarity between classes (e.g., bird species or medical instruments) negatively impacts performance, with models often confusing similar classes.
• Using generic class labels (e.g., 'tool' instead of 'surgical instrument') leads to significantly better segmentation performance than domain-specific terminology.
• The MESS benchmark reveals that current zero-shot models generalize poorly to domains like medical imaging and engineering, despite strong performance on in-domain datasets.

Figure 3: mIoU results of large models in relative comparison to the supervised mIoU on a log scale. 100 is equal to the supervised mIoU.