[Paper Review] Visual Foresight Tree for Object Retrieval from Clutter with Nonprehensile Rearrangement

Baichuan Huang, Shuai D. Han+2 more|arXiv (Cornell University)|May 6, 2021

Robotics and Sensor-Based Localization52 references4 citations

TL;DR

This paper proposes Visual Foresight Tree (VFT), a model-based method that uses a deep neural network to predict object repositioning after robotic pushes, enabling efficient tree search over visual foresight to plan nonprehensile push sequences for object retrieval in clutter. The approach achieves higher success rates and fewer actions than model-free and myopic model-based methods in simulation and real-world experiments.

ABSTRACT

This paper considers the problem of retrieving an object from a set of tightly packed objects by using a combination of robotic pushing and grasping actions. Object retrieval from confined spaces that contain clutter is an important skill for robots in order to effectively operate in households and everyday environments. The proposed solution, Visual Foresight Tree (VFT), cleverly rearranges the clutter surrounding the target object so that it can be grasped easily. Rearrangement with nested non-prehensile actions is challenging as it requires predicting complex object interactions in a combinatorially large configuration space of multiple objects. We first show that a deep neural network can be trained to accurately predict the poses of the packed objects when the robot pushes one of them. The predictive network provides visual foresight and is used in a tree search as a state transition function in the space of scene images. The tree search returns a sequence of consecutive push actions that result in the best arrangement of the clutter for grasping the target object. Experiments in simulation and using a real robot and objects show that the proposed approach outperforms model-free techniques as well as model-based myopic methods both in terms of success rates and the number of executed actions, on several challenging tasks. A video introducing VFT, with robot experiments, is accessible at this https URL. Full source code will also be made available upon publication of this manuscript.

Motivation & Objective

• To address the challenge of retrieving a target object from tightly packed clutter in household environments.
• To overcome the difficulty of planning nonprehensile rearrangements in high-dimensional, combinatorially complex configuration spaces.
• To improve success rates and reduce action counts in object retrieval by leveraging visual foresight for long-horizon planning.
• To develop a method that generalizes across diverse cluttered scenes using learned visual prediction and search-based planning.

Proposed method

• A deep neural network is trained to predict the new poses of objects in a cluttered scene after a single push action.
• The trained network serves as a visual foresight model, providing state transition predictions from scene images to future scene images.
• A tree search algorithm uses the visual foresight model to simulate sequences of push actions and evaluate their outcomes.
• The search explores the space of image-based scene states to find a sequence of pushes that optimally repositions the target object for grasping.
• The method integrates learned dynamics with symbolic tree search, enabling long-horizon planning without requiring explicit world models.
• The approach is evaluated in both simulation and on a real robot, demonstrating robustness to real-world dynamics and visual noise.

Experimental results

Research questions

• RQ1 Can a deep neural network accurately predict the outcome of a single push in a cluttered object arrangement?
• RQ2 How does visual foresight-based tree search compare to model-free reinforcement learning in terms of success rate and sample efficiency for cluttered object retrieval?
• RQ3 Can a model-based planner that uses visual foresight outperform myopic model-based methods in complex, high-dimensional rearrangement tasks?
• RQ4 To what extent does the method generalize across diverse cluttered scenes and object configurations?

Key findings

• The visual foresight network achieves high accuracy in predicting object repositioning after push actions in cluttered scenes.
• The VFT method significantly outperforms model-free reinforcement learning baselines in both success rate and number of actions required.
• The approach achieves higher success rates than myopic model-based methods, particularly in complex, tightly packed configurations.
• The method generalizes well to real-world scenarios, demonstrating robust performance on a physical robot with minimal domain randomization.