[Paper Review] The Probabilistic Object Detection Challenge

John Skinner, David Hall+3 more|arXiv (Cornell University)|Mar 19, 2019

Adversarial Robustness in Machine Learning7 references4 citations

TL;DR

This paper introduces the ACRV Robotic Vision Challenge 1: Probabilistic Object Detection, a new benchmark that requires object detectors to output spatial uncertainty using Gaussian distributions and evaluates them via the Probability-based Detection Quality (PDQ) metric. The challenge uses simulated video sequences with diverse lighting, viewpoints, and camera heights, emphasizing uncertainty calibration for real-world robotic deployment.

ABSTRACT

We introduce a new challenge for computer and robotic vision, the first ACRV Robotic Vision Challenge, Probabilistic Object Detection. Probabilistic object detection is a new variation on traditional object detection tasks, requiring estimates of spatial and semantic uncertainty. We extend the traditional bounding box format of object detection to express spatial uncertainty using gaussian distributions for the box corners. The challenge introduces a new test dataset of video sequences, which are designed to more closely resemble the kind of data available to a robotic system. We evaluate probabilistic detections using a new probability-based detection quality (PDQ) measure. The goal in creating this challenge is to draw the computer and robotic vision communities together, toward applying object detection solutions for practical robotics applications.

Motivation & Objective

• Address the lack of benchmarks that evaluate uncertainty in object detection for robotic applications.
• Create a realistic test dataset that mimics robot-sensed video with temporal correlation, varying lighting, and camera heights.
• Promote development of object detectors that express well-calibrated spatial and semantic uncertainty.
• Encourage generalization over dataset-specific fitting by withholding training data and using COCO-like classes.
• Provide a standardized evaluation framework using the PDQ metric to reward accurate uncertainty estimation.

Proposed method

• Generate synthetic video sequences using 3D simulation environments with day/night lighting and three camera heights.
• Define probabilistic object detection as requiring Gaussian distributions to represent spatial uncertainty around bounding boxes.
• Use the Probability-based Detection Quality (PDQ) metric, which combines spatial quality (via foreground and background losses) and label quality (predicted probability of true class).
• Calculate spatial quality using log-probability losses: $ L_{FG} $ for pixels inside the ground truth box and $ L_{BG} $ for pixels outside but within detection.
• Compute pairwise PDQ as the geometric mean: $ \operatorname{pPDQ}(\mathcal{G},\mathcal{D}) = \sqrt{Q_S(\mathcal{G},\mathcal{D}) \cdot Q_L(\mathcal{G},\mathcal{D})} $.
• Perform optimal assignment between ground truth and detection pairs to compute final average PDQ, enabling detailed feedback on false positives, false negatives, and detection quality.

Experimental results

Research questions

• RQ1 How can object detection be extended to include spatial and semantic uncertainty estimation in a way that supports robotic decision-making?
• RQ2 To what extent can existing detectors be adapted to output probabilistic spatial uncertainty using Gaussian distributions?
• RQ3 How does the PDQ metric effectively evaluate and rank detectors based on uncertainty calibration?
• RQ4 What are the trade-offs between detection recall, precision, and uncertainty calibration in real-world robotic perception?
• RQ5 Can a simulated video dataset with temporal correlation and diverse viewpoints better reflect robot-sensed data than static image benchmarks?

Key findings

• Participant 1 achieved the highest overall PDQ score, indicating the best balance of spatial and semantic uncertainty calibration.
• Participant 4 had the highest average pPDQ among detected objects but suffered from high false negative rates, missing many ground truths.
• Participant 2 detected the most objects with high precision but had lower average quality, resulting in a lower overall PDQ than Participant 1.
• The PDQ metric successfully differentiated systems based on uncertainty calibration, rewarding those that expressed accurate confidence levels.
• The simulated dataset with 18 test sequences (3 environments × 2 lighting × 3 camera heights) and 123,704 images in the test_dev set provides a rich, diverse benchmark for temporal and spatial uncertainty evaluation.
• The validation set, containing 4 sequences from a different environment than the test set, confirmed that models generalizing well across environments achieved better scores.