[Paper Review] SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition

Kaiyu Yang, Olga Russakovsky+1 more|arXiv (Cornell University)|Aug 7, 2019

Multimodal Machine Learning Applications40 references4 citations

TL;DR

SpatialSense introduces a novel, adversarially crowdsourced benchmark for spatial relation recognition that reduces bias from simple visual and language priors by focusing on challenging, long-tail relations. The dataset contains 17,498 spatial relations across 11,569 images with balanced positive and negative samples, revealing that state-of-the-art models perform poorly, indicating reliance on superficial cues rather than true spatial reasoning.

ABSTRACT

Understanding the spatial relations between objects in images is a surprisingly challenging task. A chair may be "behind" a person even if it appears to the left of the person in the image (depending on which way the person is facing). Two students that appear close to each other in the image may not in fact be "next to" each other if there is a third student between them. We introduce SpatialSense, a dataset specializing in spatial relation recognition which captures a broad spectrum of such challenges, allowing for proper benchmarking of computer vision techniques. SpatialSense is constructed through adversarial crowdsourcing, in which human annotators are tasked with finding spatial relations that are difficult to predict using simple cues such as 2D spatial configuration or language priors. Adversarial crowdsourcing significantly reduces dataset bias and samples more interesting relations in the long tail compared to existing datasets. On SpatialSense, state-of-the-art recognition models perform comparably to simple baselines, suggesting that they rely on straightforward cues instead of fully reasoning about this complex task. The SpatialSense benchmark provides a path forward to advancing the spatial reasoning capabilities of computer vision systems. The dataset and code are available at https://github.com/princeton-vl/SpatialSense.

Motivation & Objective

• To address the lack of a rigorous benchmark for spatial relation recognition that avoids language and visual priors.
• To reduce dataset bias by using adversarial crowdsourcing, where annotators are tasked with finding ambiguous or misleading examples.
• To create a balanced, long-tailed dataset with equal representation of positive and negative spatial relations to better evaluate reasoning capabilities.
• To decouple object detection from spatial relation recognition by providing object names and bounding boxes as input.
• To provide a benchmark that exposes the limitations of current models in understanding complex spatial configurations beyond 2D layout and linguistic patterns.

Proposed method

• Adversarial crowdsourcing: human annotators are explicitly tasked with identifying spatial relations that are difficult to predict using 2D layout or language priors.
• Each relation is labeled as positive or negative, ensuring a 50/50 class distribution to prevent model bias toward majority classes.
• The task is formulated as binary classification: given two objects with names and bounding boxes, predict whether a specific spatial relation holds.
• A predefined set of 9 spatial predicates (e.g., 'on', 'under', 'in front of') is used to standardize annotations and reduce ambiguity.
• Object localization and names are provided as input, decoupling spatial relation recognition from object detection.
• Models are evaluated on their ability to generalize across rare object classes, with 2,139 of 3,679 object classes appearing only once.

Experimental results

Research questions

• RQ1 Can adversarial crowdsourcing effectively reduce dataset bias in spatial relation recognition datasets?
• RQ2 To what extent do state-of-the-art models rely on simple visual or linguistic cues rather than true spatial reasoning?
• RQ3 How does the long-tail distribution of object classes affect model generalization in spatial relation recognition?
• RQ4 Does a balanced positive-negative split in the dataset lead to more reliable evaluation of model performance?
• RQ5 Can a benchmark with carefully curated adversarial examples better expose the limitations of current vision models in spatial understanding?

Key findings

• State-of-the-art models perform comparably to simple baselines on SpatialSense, indicating they rely on superficial cues rather than deep spatial reasoning.
• The dataset contains 17,498 relations across 11,569 images, with 3,679 unique object classes and 2,139 appearing only once, creating a strong long-tail distribution.
• Adversarial crowdsourcing successfully reduces language bias—unlike datasets such as Visual Genome, which are dominated by 'on' relations (89.37% for table-related relations).
• The benchmark reveals that models fail on subtle spatial reasoning tasks, such as determining if an object is in front of another despite being visually to the right, due to 3D configuration.
• The binary classification formulation with balanced positive and negative samples leads to more reliable evaluation than Recall@K metrics used in prior work.
• Models trained on SpatialSense show significant performance drops compared to prior benchmarks, highlighting the dataset's challenge and validity.