[Paper Review] Structure-measure: A New Way to Evaluate Foreground Maps

Deng-Ping Fan, Ming‐Ming Cheng+3 more|arXiv (Cornell University)|Aug 2, 2017

Visual Attention and Saliency DetectionComputer Science42 references109 citations

TL;DR

The paper introduces Structure-measure, a two-part structural similarity score for non-binary foreground maps that captures region-wise and object-wise structure, and demonstrates its superiority over AP, AUC, and Fbw across multiple datasets and meta-evaluations.

ABSTRACT

Foreground map evaluation is crucial for gauging the progress of object segmentation algorithms, in particular in the filed of salient object detection where the purpose is to accurately detect and segment the most salient object in a scene. Several widely-used measures such as Area Under the Curve (AUC), Average Precision (AP) and the recently proposed Fbw have been utilized to evaluate the similarity between a non-binary saliency map (SM) and a ground-truth (GT) map. These measures are based on pixel-wise errors and often ignore the structural similarities. Behavioral vision studies, however, have shown that the human visual system is highly sensitive to structures in scenes. Here, we propose a novel, efficient, and easy to calculate measure known an structural similarity measure (Structure-measure) to evaluate non-binary foreground maps. Our new measure simultaneously evaluates region-aware and object-aware structural similarity between a SM and a GT map. We demonstrate superiority of our measure over existing ones using 5 meta-measures on 5 benchmark datasets.

Motivation & Objective

• Motivate more reliable evaluation of foreground maps by accounting for object structure beyond pixel-wise errors.
• Propose a two-component structural similarity measure that captures region-level and object-level information.
• Show that the proposed measure aligns better with application rankings and human judgments across datasets.

Proposed method

• Define region-aware structural similarity S_r by partitioning SM and GT into K blocks around the GT centroid and weighting each block by GT coverage.
• Define region-level similarity for each block using a SSIM-inspired formulation and aggregate as S_r = sum_k w_k * ssim(k).
• Define object-aware similarity by separately evaluating foreground and background distributions; use mean values and coefficient of variation to form O_FG and O_BG (Equations 5–8).
• Compute final object-aware similarity S_o = mu * O_FG + (1 - mu) * O_BG, where mu is the GT foreground area ratio.
• Combine region- and object-aware measures into the final Structure-measure S = alpha * S_o + (1 - alpha) * S_r with alpha = 0.5 in implementation.

Figure 8: Structural changes examples. The first row are GT maps. The second row are its morphologically changed version.

Experimental results

Research questions

• RQ1 Can a structure-aware evaluation metric better reflect human judgments and application goals than traditional pixel-wise metrics?
• RQ2 Do region-aware and object-aware structural components capture complementary information about foreground maps?
• RQ3 Is the proposed measure robust across datasets and saliency models?

Key findings

• Structure-measure better ranks saliency maps with respect to GTs and applications compared to AP, AUC, and Fbw across multiple datasets.
• On Meta-measure 1, Structure-measure achieves best ranking consistency with an application (SalCut).
• On Meta-measure 2, it largely surpasses state-of-the-art metrics in preferring outputs from strong models over generic baselines.
• On Meta-measure 3, it reduces GT-switch ranking errors by substantial percentages across datasets.
• On Meta-measure 4, it shows robustness to annotation boundary changes, often outperforming Fbw.
• User studies indicate higher human preference for maps ranked by Structure-measure (63.69% on a 100-pair study).