[Paper Review] Obstacle Avoidance and Navigation Utilizing Reinforcement Learning with Reward Shaping

Daniel Zhang, Colleen P. Bailey|arXiv (Cornell University)|Mar 28, 2020

Reinforcement Learning in Robotics13 references4 citations

TL;DR

This paper proposes enhanced Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization (PPO) algorithms with a Lyapunov-based reward shaping technique for obstacle avoidance and navigation in unmanned ground vehicles (UGVs). The method accelerates convergence and improves performance by shaping the reward using a stability-driven term, demonstrating superior average rewards and faster convergence in Gazebo simulations with Turtlebot 3 Burger.

ABSTRACT

In this paper, we investigate the obstacle avoidance and navigation problem in the robotic control area. For solving such a problem, we propose revised Deep Deterministic Policy Gradient (DDPG) and Proximal Policy Optimization algorithms with an improved reward shaping technique. We compare the performances between the original DDPG and PPO with the revised version of both on simulations with a real mobile robot and demonstrate that the proposed algorithms achieve better results.

Motivation & Objective

• Address the challenge of obstacle avoidance and navigation in complex, real-world environments for unmanned ground vehicles (UGVs).
• Overcome the limitations of traditional SLAM-based approaches that struggle with generalization across diverse environments.
• Improve sample efficiency and convergence speed in reinforcement learning (RL) for continuous control tasks in mobile robotics.
• Evaluate the effectiveness of a Lyapunov-stability-inspired reward shaping technique in enhancing DDPG and PPO performance on real robotic platforms.

Proposed method

• Implement a reward shaping technique based on Lyapunov stability theory, defined as $ R^{lyap}(s_{t+1},a_{t+1}) = R(s_t,a_t) + \eta(\gamma R(s_{t+1},a_{t+1}) - R(s_t,a_t)) $, to guide policy learning.
• Integrate the reward shaping into DDPG by modifying the temporal difference (TD) error using the shaped reward in the critic update.
• Apply the shaped reward to the advantage function estimator in PPO, adjusting $ \hat{A}_t $ to incorporate the Lyapunov-based reward signal.
• Use a Gazebo-ROS-Turtlebot 3 Burger® simulation platform to train and evaluate both DDPG and PPO with and without reward shaping.
• Employ an actor-critic framework with deep neural networks for both algorithms, using Adam optimizer with learning rate 0.0003 and batch size 32.
• Represent the state as a 26-dimensional vector including 24 LiDAR readings, with actions in continuous space controlling linear and angular velocity.

Experimental results

Research questions

• RQ1 Does the proposed Lyapunov-based reward shaping technique improve the convergence speed and final performance of DDPG and PPO in UGV navigation tasks?
• RQ2 How do DDPG and PPO compare in terms of stability, convergence rate, and average cumulative reward when applied to obstacle avoidance with and without reward shaping?
• RQ3 Can the integration of reward shaping lead to a more robust and efficient policy in complex, real-world-like robotic environments?
• RQ4 To what extent does the reward shaping technique reduce training instability and improve sample efficiency in continuous control RL for mobile robots?

Key findings

• PPO outperforms DDPG in both convergence speed and final performance, achieving a higher average cumulative reward of -323.59 compared to DDPG's -477.06 with reward shaping.
• The inclusion of reward shaping significantly improves performance: DDPG’s average reward improved from -710.56 (without shaping) to -477.06 (with shaping), and PPO’s improved from -679.43 to -323.59.
• The minimum reward for PPO with shaping reached -155.06, indicating fewer episodes with severe penalties, suggesting more consistent policy learning.
• DDPG with shaping achieved a maximum reward of -177.65, a notable improvement over the baseline maximum of -360.87, indicating better policy robustness.
• The reward shaping technique led to faster convergence, as evidenced by earlier stabilization of average reward curves in both DDPG and PPO.
• The proposed method maintains optimality and convergence guarantees due to the theoretical foundation in Lyapunov stability, ensuring unbiased optimal policies.