[Paper Review] Approaches and Challenges in Robotic Perception for Table-top Rearrangement and Planning

Aditya Agarwal, Bipasha Sen+4 more|arXiv (Cornell University)|May 9, 2022

Robotics and Sensor-Based Localization4 citations

TL;DR

This paper presents a comprehensive analysis of perception challenges in table-top robotic rearrangement, focusing on 3D scene registration, unknown object detection, and grasp point selection. It proposes solutions like object-centric grasp proposal transfer and on-the-fly grasp verification, demonstrating improved robustness in cluttered environments through insights from the ICRA 2022 Open Cloud Robot Challenge, where the authors placed third.

ABSTRACT

Table-top Rearrangement and Planning is a challenging problem that relies heavily on an excellent perception stack. The perception stack involves observing and registering the 3D scene on the table, detecting what objects are on the table, and how to manipulate them. Consequently, it greatly influences the system's task-planning and motion-planning stacks that follow. We present a comprehensive overview and discuss the different challenges associated with the perception module. This work is a result of our extensive involvement in the ICRA 2022 Open Cloud Robot Table Organization Challenge, in which we stood third in the final rankings.

Motivation & Objective

• To address the critical challenges in the perception stack of table-top robotic rearrangement systems.
• To improve robustness in unknown object detection and 6D pose estimation for diverse, real-world household objects.
• To develop reliable grasp assignment and verification mechanisms in cluttered, dynamic scenes.
• To enhance generalization and scalability of perception pipelines beyond known-object assumptions.
• To guide future research toward structure-aware grasp ranking and scene-aware manipulation planning.

Proposed method

• Using multi-view 3D scene reconstruction with ICP-based registration to create a dense global point cloud of the table-top environment.
• Applying object-centric grasp proposal networks (e.g., Contact-GraspNet) on isolated object point clouds before transferring them to the scene using detected object poses.
• Implementing a grasp assignment logic that uses distance thresholds relative to object centroids, while avoiding collisions with neighboring objects.
• Introducing on-the-fly grasp verification via real-time scene segmentation and projection matching to confirm correct object pickup.
• Leveraging scene context and object geometry to prioritize grasp poses that are structurally sound and collision-free.
• Proposing a reinforcement learning-based approach for grasp ranking that accounts for object shape, orientation, and scene constraints.

Experimental results

Research questions

• RQ1 How can grasp proposals be reliably assigned to individual objects in cluttered, multi-object scenes?
• RQ2 What are the limitations of existing grasp proposal networks when applied to irregular or large-scale scene point clouds?
• RQ3 How can grasp verification be performed in real time to ensure correct object pickup despite proximity to similar objects?
• RQ4 To what extent can grasp selection be improved by incorporating object structure and scene context?
• RQ5 What are the key challenges in extending perception pipelines to unknown-category objects in real-world settings?

Key findings

• Object-centric grasp proposal transfer significantly improves grasp assignment accuracy by ensuring proposals are correctly mapped to their respective objects.
• Naive grasp assignment based on proximity to centroid often fails due to interference from neighboring objects, highlighting the need for context-aware selection.
• On-the-fly grasp verification via segmentation and projection matching effectively reduces incorrect object pickups in cluttered environments.
• Existing grasp proposal networks like Contact-GraspNet do not inherently consider object structure, leading to suboptimal grasp poses for tools like pliers or bottles.
• The lack of generalization to unknown-category objects remains a major bottleneck, especially for deformable or irregularly shaped items.
• Grasp ranking based on structural and contextual awareness is a promising but underexplored direction requiring further research.