[Paper Review] One Metric to Measure them All: Localisation Recall Precision (LRP) for Evaluating Visual Detection Tasks

Kemal Öksüz, Barış Can Çam+2 more|arXiv (Cornell University)|Nov 21, 2020

Advanced Neural Network Applications4 citations

TL;DR

This paper introduces Localisation Recall Precision (LRP) Error as a unified, complete, and interpretable metric for evaluating visual detection tasks, addressing key limitations of Average Precision (AP) and Panoptic Quality (PQ). LRP Error quantifies detection performance by combining localization accuracy, recall, and precision into a single error score, with Optimal LRP (oLRP) identifying the best confidence threshold per detector, outperforming AP in discriminative power across seven detection tasks and ten datasets.

ABSTRACT

Despite being widely used as a performance measure for visual detection tasks, Average Precision (AP) is limited in (i) reflecting localisation quality, (ii) interpretability and (iii) robustness to the design choices regarding its computation, and its applicability to outputs without confidence scores. Panoptic Quality (PQ), a measure proposed for evaluating panoptic segmentation (Kirillov et al., 2019), does not suffer from these limitations but is limited to panoptic segmentation. In this paper, we propose Localisation Recall Precision (LRP) Error as the average matching error of a visual detector computed based on both its localisation and classification qualities for a given confidence score threshold. LRP Error, initially proposed only for object detection by Oksuz et al. (2018), does not suffer from the aforementioned limitations and is applicable to all visual detection tasks. We also introduce Optimal LRP (oLRP) Error as the minimum LRP Error obtained over confidence scores to evaluate visual detectors and obtain optimal thresholds for deployment. We provide a detailed comparative analysis of LRP Error with AP and PQ, and use nearly 100 state-of-the-art visual detectors from seven visual detection tasks (i.e. object detection, keypoint detection, instance segmentation, panoptic segmentation, visual relationship detection, zero-shot detection and generalised zero-shot detection) using ten datasets to empirically show that LRP Error provides richer and more discriminative information than its counterparts. Code available at: https://github.com/kemaloksuz/LRP-Error

Motivation & Objective

• To address the limitations of Average Precision (AP) in reflecting localization quality, interpretability, and robustness to thresholding in visual detection tasks.
• To propose a unified evaluation metric applicable to all visual detection tasks—object detection, keypoint detection, instance segmentation, panoptic segmentation, visual relationship detection, and zero-shot detection—without requiring confidence scores.
• To introduce Optimal LRP (oLRP) Error as a threshold-optimized performance measure that identifies the best confidence threshold for deployment.
• To empirically demonstrate that LRP Error provides richer, more discriminative performance insights than AP and PQ across diverse detection tasks and datasets.

Proposed method

• LRP Error is defined as the average matching error of a detector, computed from localization error, false positive rate, and false negative rate, using a confidence score threshold.
• The metric combines localization quality (IoU-based), recall (TP/(TP+FN)), and precision (TP/(TP+FP)) into a single error score, with lower values indicating better performance.
• Optimal LRP (oLRP) Error is computed as the minimum LRP Error across all confidence score thresholds, identifying the best-performing threshold for deployment.
• The method is applied to 98 state-of-the-art detectors across 10 datasets spanning 7 visual detection tasks, including object detection, instance segmentation, panoptic segmentation, and zero-shot detection.
• The approach uses IoU-based matching for bounding boxes and masks, and handles detectors without confidence scores by using a threshold-agnostic formulation.
• The framework is validated through comparative analysis with AP and PQ, including ablation on threshold sensitivity and class-specific thresholding.

Figure 1 : Three different object detection results (for an image from COCO [ 1 ] ) with very different PR curves but the same AP. First Row : Blue, white and orange colors denote ground-truth, TPs and FPs respectively. Numbers are confidence scores, $s$ , of the detections. Second row : PR curves f

Experimental results

Research questions

• RQ1 Can a single metric effectively evaluate localization, classification, and detection quality across diverse visual detection tasks?
• RQ2 How does LRP Error compare to AP and PQ in terms of completeness, interpretability, and robustness to threshold selection?
• RQ3 Does oLRP Error consistently identify optimal confidence thresholds across different detectors and datasets?
• RQ4 To what extent does LRP Error reveal performance differences that AP fails to capture, especially in localization quality?
• RQ5 Can class-specific LRP-Optimal thresholds improve detector performance in long-tailed and zero-shot detection scenarios?

Key findings

• LRP Error outperforms AP in discriminative power: in a case study with three detection results on COCO, AP assigned the same value to all three despite significant differences in localization quality, while oLRP Error correctly penalized the worst-localized detections.
• oLRP Error identified optimal confidence thresholds that significantly improved performance: in a video object detection use-case, class-specific oLRP thresholds improved AP50 by up to 9 points and oLRP Error by 4 points compared to fixed thresholds.
• The LRP-Optimal threshold distribution varies widely across classes: 70% of rare classes in LVIS had an oLRP threshold of 0.0, indicating that fixed thresholds are suboptimal for long-tailed datasets.
• For zero-shot and generalized zero-shot detection, oLRP thresholds were consistently lower for unseen classes, confirming the need for class-specific thresholding.
• On average, oLRP Error provided more informative performance insights than AP: detectors with poor localization or high FP rates were clearly penalized, unlike in AP, which masked such issues.
• The method is robust across tasks: LRP Error was successfully applied to seven visual detection tasks, including panoptic segmentation, with consistent and meaningful performance ranking.

(a) Object Detection