[Paper Review] A Meta-Theory of Boundary Detection Benchmarks

Xiaodi Hou, Alan Yuille+1 more|arXiv (Cornell University)|Feb 25, 2013

Image and Signal Denoising Methods2 references9 citations

TL;DR

This paper proposes a psychophysical framework to assess the reliability of human-labeled boundary detection benchmarks by estimating perceptual strength of boundaries through pairwise comparison experiments. Using a two-alternative forced choice paradigm and a probabilistic graphical model, it identifies unreliable 'orphan' labels and reduces benchmark risk, showing that filtering low-strength boundaries improves evaluation fidelity with a 30.88% reduction in high-risk labels.

ABSTRACT

Human labeled datasets, along with their corresponding evaluation algorithms, play an important role in boundary detection. We here present a psychophysical experiment that addresses the reliability of such benchmarks. To find better remedies to evaluate the performance of any boundary detection algorithm, we propose a computational framework to remove inappropriate human labels and estimate the intrinsic properties of boundaries.

Motivation & Objective

• To investigate the reliability of human-labeled boundary detection benchmarks, such as BSDS300, which are widely used to evaluate algorithmic performance.
• To quantify the risk in existing benchmarks arising from inconsistent or incorrect human labeling, particularly false alarms and missed boundaries.
• To develop a computational framework that estimates the perceptual strength of boundary segments using human comparison data.
• To reduce benchmark risk by identifying and filtering low-confidence or isolated (orphan) labels that contribute to evaluation bias.
• To validate the framework using a psychophysical experiment and a probabilistic inference model, demonstrating improved consistency in benchmark evaluation.

Proposed method

• Employ a two-alternative forced choice (2AFC) psychophysical experiment where subjects compare the perceptual strength of boundary segments to establish a strict total ordering.
• Map each boundary segment to a perceptual strength value $ x \in [0,1] $, where higher values indicate stronger perceptual salience.
• Use a probabilistic graphical model where label responses depend on perceptual strength $ x_i $ and labeler-specific response profiles $ \mu^l $, modeled via a sigmoid function with hidden parameters $ \theta^l $.
• Apply an EM algorithm to estimate $ x_i $, $ \mu^l $, and $ \theta^l $, with $ P(y_i^l=1 \mid \mu^l, x_i) $ computed via convolution with a Gaussian kernel $ \phi_\sigma $, $ \sigma = 0.15 $.
• Define benchmark risk as $ R(\mathcal{S}, \mathcal{A}) = P(x_i < x_j \mid s_i \in \mathcal{S}, s_j \in \mathcal{A} \setminus \mathcal{S}) $, measuring the probability that a strong algorithmic boundary is misclassified as a false alarm.
• Evaluate risk across thresholded subsets $ \bar{\mathcal{S}}_{\tau_i} $ using majority voting from 5 subjects over 500 trials, with thresholds $ \tau_1 = 0.2 $ to $ \tau_4 = 1 $.

Experimental results

Research questions

• RQ1 How reliable are human-labeled boundary detection benchmarks like BSDS300, given known label variability and inconsistency?
• RQ2 What is the perceptual strength of individual boundary segments, and how can it be estimated from pairwise human comparisons?
• RQ3 To what extent do 'orphan' labels—those labeled by only one subject—contribute to benchmark risk?
• RQ4 How does filtering boundaries by perceptual strength threshold affect the risk of misclassifying algorithmic detections as false positives?
• RQ5 Can a probabilistic inference model improve the consistency and reliability of benchmark evaluation?

Key findings

• The initial perceptual strength distribution of boundaries in BSDS300 is spiky and inconsistent, reflecting labeler-specific biases.
• After EM inference, the perceptual strength distribution is smoothed and many low-strength boundaries are suppressed, especially orphan labels.
• 30.88% of all boundaries in BSDS300 are orphan labels, which are the primary source of benchmark risk.
• Risk decreases monotonically with increasing perceptual strength threshold, with the lowest risk at $ \tau = 1 $, indicating that only the most salient boundaries are safe to use as ground truth.
• The majority-voting risk estimate (right figure, Fig. 3) shows a clear trend: higher thresholds yield significantly lower risk, validating the risk-utility trade-off model.
• The framework successfully identifies and downweights unreliable labels, demonstrating that current benchmarks contain systematic errors that compromise algorithm evaluation.