[Paper Review] Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control

Fangyi Zhang, Jürgen Leitner+3 more|arXiv (Cornell University)|Nov 12, 2015

Advanced Vision and ImagingComputer Science8 references213 citations

TL;DR

This paper proposes a vision-based deep reinforcement learning system using Deep Q Networks (DQN) to train a robotic manipulator to perform target reaching using only raw pixel inputs, without prior knowledge of joint states or configuration. The key finding is that while the agent succeeds in simulation and when using synthetic images in the real world, it fails with real camera images due to domain shift, highlighting the need for domain adaptation or robustness in visual DRL for real-world robotics.

ABSTRACT

This paper introduces a machine learning based system for controlling a robotic manipulator with visual perception only. The capability to autonomously learn robot controllers solely from raw-pixel images and without any prior knowledge of configuration is shown for the first time. We build upon the success of recent deep reinforcement learning and develop a system for learning target reaching with a three-joint robot manipulator using external visual observation. A Deep Q Network (DQN) was demonstrated to perform target reaching after training in simulation. Transferring the network to real hardware and real observation in a naive approach failed, but experiments show that the network works when replacing camera images with synthetic images.

Motivation & Objective

• To enable a robotic manipulator to autonomously learn target-reaching skills using only visual observation and no prior knowledge of joint states or configuration.
• To investigate the feasibility of applying Deep Q Networks (DQN) to vision-based robotic manipulation in both simulation and real-world settings.
• To identify and analyze the failure of DQN agents trained in simulation when deployed in real-world environments with real camera inputs.
• To evaluate whether synthetic images generated from real-time joint states can bridge the sim-to-real gap in vision-based DRL.
• To explore challenges and future directions for robust vision-based deep reinforcement learning in real-world robotic manipulation.

Proposed method

• A 2D robotic arm simulator was developed to train a DQN agent for target-reaching using only raw pixel observations from a monocular camera.
• The DQN agent was trained in simulation using a reward function based on distance to the target, with exploration via epsilon-greedy policy.
• ROS-based interfaces were implemented to enable real-time communication between the Baxter robot and the DQN agent using real-time joint angle feedback.
• Synthetic images were generated in real time from the robot’s actual joint angles (S1, E1, W1) to match simulation input distributions and reduce domain shift.
• The same trained DQN agent was evaluated in real-world experiments using both real camera images and synthetic images to isolate the impact of visual domain shift.
• Image differences between simulation and real-world settings were analyzed as a primary cause of failure, with potential sources including camera pose variations, color distortions, and shape inconsistencies.

Experimental results

Research questions

• RQ1 Can a DQN agent learn target-reaching solely from raw visual observations without any prior knowledge of robot configuration or joint states?
• RQ2 Why does a DQN agent trained in simulation fail when deployed in the real world using real camera images?
• RQ3 Can synthetic images generated from real-time joint states bridge the sim-to-real gap in vision-based DRL for robotic manipulation?
• RQ4 What are the main visual domain shift factors (e.g., image distortions, camera pose differences) that cause failure in real-world deployment?
• RQ5 How can future DRL architectures be made robust to visual domain shift in real-world robotic control?

Key findings

• The DQN agent successfully learned to perform target reaching in simulation with a consistent success rate when trained using raw pixel inputs and a distance-based reward function.
• When deployed in the real world using real camera images, the same agent achieved a success rate of 0%, indicating complete failure to generalize from simulation.
• In contrast, when using synthetic images generated from real-time joint angles, the agent achieved a success rate consistent with simulation, confirming that the failure was due to visual domain shift.
• The primary cause of failure was identified as differences in input images between simulation and real-world scenarios, including camera pose variations, color distortions, and shape inconsistencies.
• The study demonstrates that naive sim-to-real transfer fails for vision-based DRL in robotic manipulation, even with identical network architectures and training procedures.
• Future work must focus on either reducing domain shift (e.g., via better simulation fidelity or image preprocessing) or improving model robustness to visual domain gaps.