[Paper Review] Disentangled Planning and Control in Vision Based Robotics via Reward Machines

Alberto Rivas, Jacob Varley+3 more|arXiv (Cornell University)|Dec 28, 2020

Computability, Logic, AI Algorithms34 references4 citations

TL;DR

This paper proposes Deep Q-Learning with a Reward Machine (DQRM) to improve sample efficiency and policy quality in vision-based robotic manipulation tasks. By decoupling planning and control through abstract state representations and dense reward shaping via a reward machine, DQRM achieves superior performance—learning high-quality policies in under 200 batch training steps, outperforming standard DQN and hybrid baselines.

ABSTRACT

In this work we augment a Deep Q-Learning agent with a Reward Machine (DQRM) to increase speed of learning vision-based policies for robot tasks, and overcome some of the limitations of DQN that prevent it from converging to good-quality policies. A reward machine (RM) is a finite state machine that decomposes a task into a discrete planning graph and equips the agent with a reward function to guide it toward task completion. The reward machine can be used for both reward shaping, and informing the policy what abstract state it is currently at. An abstract state is a high level simplification of the current state, defined in terms of task relevant features. These two supervisory signals of reward shaping and knowledge of current abstract state coming from the reward machine complement each other and can both be used to improve policy performance as demonstrated on several vision based robotic pick and place tasks. Particularly for vision based robotics applications, it is often easier to build a reward machine than to try and get a policy to learn the task without this structure.

Motivation & Objective

• To address the sample inefficiency and poor convergence of standard Deep Q-Networks (DQN) in vision-based robotic tasks.
• To investigate whether decoupling latent state feature learning from policy learning improves training stability and performance.
• To evaluate the effectiveness of combining abstract state supervision and dense reward shaping in deep reinforcement learning.
• To demonstrate that reward machines can be constructed from expert demonstrations to bootstrap policy learning.
• To show that abstract state representations, even when redundant in observations, are crucial for DQN to learn effective policies.

Proposed method

• Introduces a Reward Machine (RM), a finite state machine that models task structure through abstract states and transition-based dense rewards.
• Uses hand-crafted feature detectors to extract task-relevant state features (e.g., block positions, gripper state) to define abstract states.
• Constructs the RM from expert demonstrations, ensuring Q-values remain centered around zero for stable deep learning.
• Augments DQN with a one-hot encoded abstract state input and a dense reward function derived from the RM to guide exploration.
• Employs a ResNet-based encoder to classify raw observations into abstract states, enabling end-to-end training with abstract state supervision.
• Combines demonstrations with off-policy replay buffers, allowing DQRM to leverage expert data for faster convergence.

Experimental results

Research questions

• RQ1 Can abstract state representations significantly improve the sample efficiency and final policy quality of DQN in vision-based robotic tasks?
• RQ2 Does combining abstract state supervision with dense reward shaping lead to better performance than using either signal alone?
• RQ3 Can a reward machine be effectively constructed from a small number of expert demonstrations to guide DQN training?
• RQ4 Is the performance gain from abstract states dependent on task horizon length (e.g., short vs. long-horizon tasks)?
• RQ5 To what extent does the DQRM framework reduce the number of training steps required to achieve high policy success rates?

Key findings

• Standard DQN failed to learn good-quality policies in simple vision-based pick-and-place tasks, even when abstract state information was present in observations.
• DQRM achieved a success rate comparable to DQN trained with 500 demonstrations using only 100 demonstrations, demonstrating strong data efficiency.
• DQRM reached acceptable success rates in under 200 batch training steps, while DQN with reward shaping alone required 1,500 steps to converge.
• The combination of abstract states and dense reward shaping in DQRM led to more stable and higher-quality policies than either signal used in isolation.
• Abstract state supervision was particularly effective in long-horizon tasks (e.g., kitting task with six steps), where DQN(AS) alone failed to improve performance.
• The ResNet-based classifier successfully mapped raw observations to abstract states, confirming that abstract state recognition is feasible and easy in practice.