[Paper Review] TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specification

Mingdong Wu, Fangwei Zhong+2 more|arXiv (Cornell University)|Sep 2, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes TarGF, a score-based method that learns a target gradient field from example layouts to guide object rearrangement without explicit goal specification. By estimating the gradient of the log-density of the target distribution, TarGF provides object-wise guidance for both model-based planning and reinforcement learning, significantly outperforming SOTA methods in terminal state quality, efficiency, and scalability across ball and room rearrangement tasks.

ABSTRACT

Object Rearrangement is to move objects from an initial state to a goal state. Here, we focus on a more practical setting in object rearrangement, i.e., rearranging objects from shuffled layouts to a normative target distribution without explicit goal specification. However, it remains challenging for AI agents, as it is hard to describe the target distribution (goal specification) for reward engineering or collect expert trajectories as demonstrations. Hence, it is infeasible to directly employ reinforcement learning or imitation learning algorithms to address the task. This paper aims to search for a policy only with a set of examples from a target distribution instead of a handcrafted reward function. We employ the score-matching objective to train a Target Gradient Field (TarGF), indicating a direction on each object to increase the likelihood of the target distribution. For object rearrangement, the TarGF can be used in two ways: 1) For model-based planning, we can cast the target gradient into a reference control and output actions with a distributed path planner; 2) For model-free reinforcement learning, the TarGF is not only used for estimating the likelihood-change as a reward but also provides suggested actions in residual policy learning. Experimental results in ball and room rearrangement demonstrate that our method significantly outperforms the state-of-the-art methods in the quality of the terminal state, the efficiency of the control process, and scalability.

Motivation & Objective

• To address object rearrangement in environments without explicit goal specification, where defining a target state or reward function is difficult.
• To enable AI agents to learn effective rearrangement policies using only a set of example target layouts, avoiding costly expert demonstrations.
• To develop a method that provides both motion guidance and reward signals for control without relying on handcrafted rewards or goal-conditioned supervision.
• To improve the quality, efficiency, and scalability of object rearrangement in complex, real-world-like environments.

Proposed method

• Train a target score network via denoising score matching to estimate the gradient of the log-density of the target distribution, forming the Target Gradient Field (TarGF).
• Use TarGF to generate object-wise pseudo-velocities that point toward higher-density regions of the target distribution, guiding object movement.
• Integrate TarGF into a model-based path planner (e.g., ORCA) to generate collision-free, efficient trajectories for object rearrangement.
• Derive a likelihood-change-based reward signal from TarGF for use in reinforcement learning, enabling reward shaping without expert trajectories.
• Combine TarGF with a residual policy network in a model-free RL framework (e.g., SAC) to improve sample efficiency and policy performance.
• Extend the method to image-based input by rendering state examples into images for training the score network, enabling visual reward learning.

Experimental results

Research questions

• RQ1 Can a score-based model learn a meaningful target gradient field from only a set of example layouts, without explicit goal specification?
• RQ2 How effectively can the learned target gradient field guide both planning and reinforcement learning in object rearrangement?
• RQ3 Can TarGF improve the quality and efficiency of terminal configurations compared to SOTA methods in complex, high-dimensional environments?
• RQ4 Does TarGF generalize across different dynamics, such as velocity-based and force-based control?
• RQ5 Can the method maintain diversity in final configurations while achieving high-quality states, avoiding mode collapse?

Key findings

• TarGF-based methods significantly outperform SOTA baselines in terminal state quality, as measured by lower average entropy of latent distributions (0.0037 vs. 0.0066 for ground truth) and better coverage of all target modes.
• In ball rearrangement, TarGF-guided planning reduced path length and collision count compared to ORCA and SQIL baselines, achieving faster convergence.
• The method achieved comparable performance to SAC-based baselines even when using image-based reward learning, though with a slight drop in efficiency due to higher input dimensionality.
• In force-based dynamics, the TarGF-based PID controller achieved performance close to SAC, demonstrating robustness to control dynamics and error.
• The bi-level policy using TarGF for object selection (via gradient magnitude) outperformed a goal-based baseline in both qualitative and quantitative metrics.
• The average latent distribution of rearrangement results closely matched the ground truth mode centers, indicating effective coverage of diverse target configurations.