[Paper Review] TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priors

Gabriel Sarch, Zhaoyuan Fang+5 more|arXiv (Cornell University)|Jul 21, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

TIDEE proposes a visuo-semantic commonsense reasoning framework for robotic room tidying that uses a graph convolutional network (rGCN) to infer plausible receptacle contexts and a language-conditioned out-of-place detector to prioritize misplaced objects. By integrating 3D semantic mapping, FMM-based navigation, and learnable language priors, TIDEE achieves 8.9% FixedStrict and 2.6% Success on the 2021 Rearrangement Challenge, significantly outperforming baselines and enabling dynamic adaptation of placement priors via natural language instructions.

ABSTRACT

We introduce TIDEE, an embodied agent that tidies up a disordered scene based on learned commonsense object placement and room arrangement priors. TIDEE explores a home environment, detects objects that are out of their natural place, infers plausible object contexts for them, localizes such contexts in the current scene, and repositions the objects. Commonsense priors are encoded in three modules: i) visuo-semantic detectors that detect out-of-place objects, ii) an associative neural graph memory of objects and spatial relations that proposes plausible semantic receptacles and surfaces for object repositions, and iii) a visual search network that guides the agent's exploration for efficiently localizing the receptacle-of-interest in the current scene to reposition the object. We test TIDEE on tidying up disorganized scenes in the AI2THOR simulation environment. TIDEE carries out the task directly from pixel and raw depth input without ever having observed the same room beforehand, relying only on priors learned from a separate set of training houses. Human evaluations on the resulting room reorganizations show TIDEE outperforms ablative versions of the model that do not use one or more of the commonsense priors. On a related room rearrangement benchmark that allows the agent to view the goal state prior to rearrangement, a simplified version of our model significantly outperforms a top-performing method by a large margin. Code and data are available at the project website: https://tidee-agent.github.io/.

Motivation & Objective

• To address the challenge of robotic room tidying in novel, unstructured environments where objects are misplaced and receptacle contexts are ambiguous.
• To improve object placement accuracy by leveraging visuo-semantic commonsense priors derived from scene context and object relationships.
• To enable dynamic adaptation of placement priors through fine-tuning with natural language instructions, allowing agents to learn new contextual rules post-training.
• To develop a robust, end-to-end system combining 3D semantic mapping, FMM-based navigation, and a dual-stage detection pipeline for out-of-place object localization and placement.
• To evaluate performance on standardized benchmarks such as the 2021 and 2022 Rearrangement Challenges, demonstrating generalization to unseen environments.

Proposed method

• TIDEE constructs 2D and 3D spatial maps from RGB-D observations, maintaining a 3D semantic occupancy map with 116 object categories and a dynamic object memory of 3D centroids.
• It uses a dDETR detector for 2D object detection and projects bounding boxes into 3D using depth data and Gaussian filtering, avoiding reliance on less reliable 3D detectors.
• A graph convolutional network (rGCN) infers plausible receptacle contexts by modeling relationships between objects and receptacles in the 3D scene.
• The system employs Fast Marching Method (FMM) for path planning and navigation, with exploration guided by sampling unexplored, traversible regions in the 2D occupancy map.
• A language-conditioned out-of-place detector (BERT-OOP) is fine-tuned using relation-label pairings (e.g., 'alarm clock supported-by desk' → 'out-of-place') to inject new commonsense priors via natural language.
• When placement fails, TIDEE attempts to open receptacles or searches for general receptacles using farthest point sampling from a set of candidate locations.

Figure 1: Example images of out-of-place objects.

Experimental results

Research questions

• RQ1 Can visuo-semantic commonsense priors improve robotic object placement in novel, unstructured indoor environments?
• RQ2 How effectively can a learned object-receptacle relationship model (rGCN) generalize to unseen rooms and object configurations?
• RQ3 To what extent can language instructions dynamically alter the agent’s perception of what constitutes an 'out-of-place' object?
• RQ4 What is the performance of TIDEE on standardized benchmarks like the 2021 and 2022 Rearrangement Challenges compared to existing methods?
• RQ5 How does the integration of 3D semantic mapping and FMM-based navigation impact exploration and task success?

Key findings

• TIDEE achieved 8.9% FixedStrict and 2.6% Success on the 2021 Rearrangement Challenge, significantly outperforming the baseline Weihs et al. [RoomR] with 1.4% FixedStrict and 0.3% Success.
• The addition of noisy pose or estimated depth degraded performance slightly, with 6.6% FixedStrict and 1.4% Success, indicating robustness to sensor noise.
• Language instruction fine-tuning successfully increased the out-of-place confidence for target relations: e.g., 'alarm clock supported-by desk' rose from 0.10 to 0.70, and 'newspaper supported-by sofa' from 0.43 to 0.98.
• The Visual Search Network produced high-probability predictions at plausible object locations, though occasional false positives occurred (e.g., 'bed' near a dresser), likely due to recall-focused training.
• TIDEE’s ability to detect and place misplaced objects improved significantly when using rGCN-inferred receptacle priors, reducing misplacement rates to 95% on the 2021 benchmark.
• The system demonstrated effective generalization to novel rooms by combining semantic mapping, context-aware navigation, and adaptive reasoning via language-conditioned priors.

Figure 2: Example images shown to Amazon Mechanical Turk evaluators.