[Paper Review] Acquiring Target Stacking Skills by Goal-Parameterized Deep Reinforcement Learning

Wenbin Li, Jeannette Bohg+1 more|arXiv (Cornell University)|Nov 1, 2017

Reinforcement Learning in Robotics22 references3 citations

TL;DR

This paper proposes goal-parameterized deep reinforcement learning (GDQN) to enable agents to learn block stacking skills in a physics-simulated environment by directly incorporating goal states into the policy network. The method achieves superior generalization across diverse target structures compared to standard DQN, with success rates improving from 0.00 to 0.17 on 4-block tasks and overlap ratios reaching 0.79 with reward shaping.

ABSTRACT

Understanding physical phenomena is a key component of human intelligence and enables physical interaction with previously unseen environments. In this paper, we study how an artificial agent can autonomously acquire this intuition through interaction with the environment. We created a synthetic block stacking environment with physics simulation in which the agent can learn a policy end-to-end through trial and error. Thereby, we bypass to explicitly model physical knowledge within the policy. We are specifically interested in tasks that require the agent to reach a given goal state that may be different for every new trial. To this end, we propose a deep reinforcement learning framework that learns policies which are parametrized by a goal. We validated the model on a toy example navigating in a grid world with different target positions and in a block stacking task with different target structures of the final tower. In contrast to prior work, our policies show better generalization across different goals.

Motivation & Objective

• To enable artificial agents to autonomously learn physical interaction skills through trial-and-error without explicitly modeling physics.
• To address the challenge of generalizing policies across varying goal states in manipulation tasks, such as stacking blocks into different target configurations.
• To develop a deep reinforcement learning framework that integrates goal information directly into the policy network for improved generalization.
• To validate the framework on both a grid-world navigation task and a complex block-stacking task with diverse target structures.

Proposed method

• The paper introduces a goal-parameterized deep Q-network (GDQN) that takes both the current state and a goal embedding as input to produce action policies.
• The goal is encoded as a vector and concatenated with the observation state to guide the policy toward specific target configurations.
• The framework uses deep reinforcement learning with experience replay and target networks, similar to DQN, but extends it to handle goal-conditioned learning.
• Reward shaping is applied using overlap ratio and distance transform to improve learning efficiency and final performance.
• The model is trained end-to-end in a simulated block-stacking environment with physics-based dynamics.
• The training process includes epsilon-greedy exploration with annealing and a replay buffer of size 200K steps.

Experimental results

Research questions

• RQ1 Can a single deep reinforcement learning policy generalize across multiple distinct goal states in a block-stacking task?
• RQ2 How does integrating goal information into the policy network affect generalization performance compared to standard DQN?
• RQ3 To what extent does reward shaping with overlap ratio and distance transform improve policy performance and sample efficiency?
• RQ4 Does the proposed goal-parameterized framework generalize effectively to more complex target structures with increasing numbers of blocks?

Key findings

• In the 5×5 gridworld navigation task, GDQN achieved a success rate of 0.97 compared to DQN’s 0.67, demonstrating strong generalization across random start and goal positions.
• For 2-block stacking, GDQN achieved a success rate of 0.72 and overlap ratio of 0.82, significantly outperforming DQN’s 0.70 success rate and 0.70 overlap ratio.
• In 4-block stacking, DQN failed completely with a success rate of 0.00 and overlap ratio of 0.03, while GDQN achieved a success rate of 0.17 and overlap ratio of 0.41.
• With reward shaping using distance transform, GDQN achieved a success rate of 0.56 and overlap ratio of 0.79 on 4-block tasks, showing substantial improvement over unshaped GDQN.
• The performance gap between DQN and GDQN widened with increasing task complexity, indicating GDQN’s superior generalization capability on harder goals.
• The results confirm that goal-parameterized learning enables a single policy to handle diverse target structures effectively, even when the number of blocks increases.