[Paper Review] Rethinking Evaluation Metrics of Open-Vocabulary Segmentaion

Hao Zhou, Tiancheng Shen+5 more|arXiv (Cornell University)|Nov 6, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes Open mIoU, Open AP, and Open PQ—novel evaluation metrics for open-vocabulary image segmentation that replace binary true positives with similarity-based soft scores using semantic similarity. By leveraging WordNet path similarity, the metrics better align with human judgment, especially in zero-shot and cross-dataset settings, and demonstrate superior consistency in evaluating model generalization beyond closed-set assumptions.

ABSTRACT

In this paper, we highlight a problem of evaluation metrics adopted in the open-vocabulary segmentation. That is, the evaluation process still heavily relies on closed-set metrics on zero-shot or cross-dataset pipelines without considering the similarity between predicted and ground truth categories. To tackle this issue, we first survey eleven similarity measurements between two categorical words using WordNet linguistics statistics, text embedding, and language models by comprehensive quantitative analysis and user study. Built upon those explored measurements, we designed novel evaluation metrics, namely Open mIoU, Open AP, and Open PQ, tailored for three open-vocabulary segmentation tasks. We benchmarked the proposed evaluation metrics on 12 open-vocabulary methods of three segmentation tasks. Even though the relative subjectivity of similarity distance, we demonstrate that our metrics can still well evaluate the open ability of the existing open-vocabulary segmentation methods. We hope that our work can bring with the community new thinking about how to evaluate the open ability of models. The evaluation code is released in github.

Motivation & Objective

• Identify the critical flaw in existing evaluation metrics for open-vocabulary segmentation: reliance on closed-set metrics that ignore semantic similarity between predicted and ground truth categories.
• Investigate and compare 11 word similarity measurement methods—WordNet, text embeddings (GloVe, FastText, Word2Vec), and language models (CLIP, BERT, GPT, T5)—for their suitability in open-vocabulary evaluation.
• Design new evaluation metrics—Open mIoU, Open AP, and Open PQ—tailored for open-vocabulary semantic, instance, and panoptic segmentation tasks.
• Benchmark 12 state-of-the-art open-vocabulary segmentation models across three tasks to validate the proposed metrics’ effectiveness and alignment with human perception.
• Promote a paradigm shift in evaluating model generalization by incorporating semantic similarity into evaluation pipelines, moving beyond strict category matching.

Proposed method

• Use a class-agnostic matching strategy that identifies the highest IoU between predicted and ground truth masks, regardless of category label, to avoid strict category-based matching.
• Replace binary true positive (TP), false positive (FP), and false negative (FN) scores with soft scores based on semantic similarity between predicted and ground truth categories.
• Employ WordNet path similarity as the preferred similarity metric due to its moderate values, ability to distinguish polysemy, data independence, and alignment with human cognition.
• Construct Open mIoU for semantic segmentation, Open AP for instance segmentation, and Open PQ for panoptic segmentation, each integrating IoU and similarity-based scoring.
• Apply the metrics across three tasks: in-dataset and cross-dataset settings on COCO and ADE20K, with ablation studies on different similarity methods (Path, GloVe, T5).
• Conduct a user study with 1,000 participants to validate human preference for evaluation results using the Path similarity method over other similarity measures.

Experimental results

Research questions

• RQ1 How do existing closed-set evaluation metrics fail to capture the true open-world performance of segmentation models when predictions are semantically similar but not identical to ground truth?
• RQ2 Which of the 11 word similarity measurement methods—WordNet, text embeddings, or pre-trained language models—best balances interpretability, robustness, and alignment with human perception for open-vocabulary evaluation?
• RQ3 To what extent do the proposed open metrics (Open mIoU, Open AP, Open PQ) improve evaluation consistency and human alignment compared to vanilla metrics in zero-shot and cross-dataset settings?
• RQ4 How sensitive are the new metrics to segmentation quality and classification accuracy, especially in cases of incorrect but semantically similar label predictions?
• RQ5 Can the proposed metrics be generalized across different segmentation tasks (semantic, instance, panoptic) and detection tasks, and how stable are their improvements across different datasets and IoU thresholds?

Key findings

• The Path similarity method from WordNet achieved the highest human satisfaction score (mean 6.12 out of 10) in a user study with 1,000 participants, outperforming GloVe (3.05) and T5 (4.29) in perceived alignment with human judgment.
• Open AP consistently outperformed vanilla AP in both in-dataset and cross-dataset settings for instance segmentation, with larger improvements observed on novel classes, indicating sensitivity to semantic similarity.
• Open RQ showed significantly larger improvements on the target dataset (ADE20K) than on the source dataset (COCO), validating its sensitivity to semantic distance and ability to capture recognition quality in open-world settings.
• Open PQ demonstrated strong sensitivity to segmentation quality, correctly assigning higher scores to mask pairs with incorrect labels when the semantic similarity was high, indicating robustness to misclassification in open-vocabulary contexts.
• Ablation studies confirmed that Open PQ using the Path method yielded moderate and reliable results, while GloVe and T5 overestimated recognition quality, suggesting potential bias in embedding-based similarity measures.
• The proposed metrics, especially Open AP and Open PQ, showed stable improvements across different IoU thresholds and tasks, including object detection, indicating broad applicability and robustness.