[Paper Review] Towards Monocular Vision based Obstacle Avoidance through Deep Reinforcement Learning

Linhai Xie, Sen Wang+2 more|arXiv (Cornell University)|Jun 29, 2017

Robotic Path Planning Algorithms24 references136 citations

TL;DR

The paper proposes a dueling architecture based deep double-Q network (D3QN) for monocular vision obstacle avoidance, trained in simulation and transferable to real robots despite noisy depth predictions.

ABSTRACT

Obstacle avoidance is a fundamental requirement for autonomous robots which operate in, and interact with, the real world. When perception is limited to monocular vision avoiding collision becomes significantly more challenging due to the lack of 3D information. Conventional path planners for obstacle avoidance require tuning a number of parameters and do not have the ability to directly benefit from large datasets and continuous use. In this paper, a dueling architecture based deep double-Q network (D3QN) is proposed for obstacle avoidance, using only monocular RGB vision. Based on the dueling and double-Q mechanisms, D3QN can efficiently learn how to avoid obstacles in a simulator even with very noisy depth information predicted from RGB image. Extensive experiments show that D3QN enables twofold acceleration on learning compared with a normal deep Q network and the models trained solely in virtual environments can be directly transferred to real robots, generalizing well to various new environments with previously unseen dynamic objects.

Motivation & Objective

• Address obstacle avoidance for robots using only monocular RGB vision.
• Develop a learning-based method that can leverage simulated data and transfer to real-world scenarios.
• Improve training efficiency and robustness through dueling and double Q-network architectures.

Proposed method

• Use a two-phase network that predicts depth from RGB images and then applies a D3QN to output actions.
• Adopt a dueling network to separately estimate state value and action advantages.
• Incorporate a double Q-learning framework to stabilize training and reduce overestimation.
• Train in Gazebo with two environments (simple and complex) and apply random noise/blur to depth to enhance sim-to-real transfer.
• Discretize actions into independent linear and angular velocities and define a reward r = v * cos(omega) * delta_t with collision penalty.

Experimental results

Research questions

• RQ1 Can monocular RGB input be effectively used for obstacle avoidance via deep reinforcement learning?
• RQ2 Does the D3QN architecture improve learning speed and performance compared to DQN and DDQN in this domain?
• RQ3 To what extent can policies trained in simulation transfer to real robots under noisy depth predictions?
• RQ4 How robust is the learned policy to unseen real-world environments and dynamic obstacles?

Key findings

• D3QN achieves about twofold faster learning than a standard DQN in obstacle avoidance tasks.
• The dueling and double-Q mechanisms improve training efficiency and policy performance over baseline DQN and DDQN.
• Policies trained in simulation with depth prediction noise generalize to real-world scenarios and unseen dynamic objects.
• Real-world tests show reasonable action predictions from static RGB images across diverse indoor environments.
• The approach enables monocular vision based obstacle avoidance with direct sim-to-real transfer, validated on a Turtlebot.