[Paper Review] Bipedal Walking Robot using Deep Deterministic Policy Gradient

Arun Kumar, Navneet Paul+1 more|arXiv (Cornell University)|Jul 16, 2018

Robotic Locomotion and Control15 references20 citations

TL;DR

This paper presents a planar bipedal walking robot trained via Deep Deterministic Policy Gradient (DDPG) reinforcement learning in the Gazebo simulator, achieving stable forward walking at 0.83 m/s with a gait pattern closely resembling human locomotion. The robot learns through trial-and-error without prior knowledge of dynamics, using a reward function that encourages speed and stability.

ABSTRACT

Machine learning algorithms have found several applications in the field of robotics and control systems. The control systems community has started to show interest towards several machine learning algorithms from the sub-domains such as supervised learning, imitation learning and reinforcement learning to achieve autonomous control and intelligent decision making. Amongst many complex control problems, stable bipedal walking has been the most challenging problem. In this paper, we present an architecture to design and simulate a planar bipedal walking robot(BWR) using a realistic robotics simulator, Gazebo. The robot demonstrates successful walking behaviour by learning through several of its trial and errors, without any prior knowledge of itself or the world dynamics. The autonomous walking of the BWR is achieved using reinforcement learning algorithm called Deep Deterministic Policy Gradient(DDPG). DDPG is one of the algorithms for learning controls in continuous action spaces. After training the model in simulation, it was observed that, with a proper shaped reward function, the robot achieved faster walking or even rendered a running gait with an average speed of 0.83 m/s. The gait pattern of the bipedal walker was compared with the actual human walking pattern. The results show that the bipedal walking pattern had similar characteristics to that of a human walking pattern. The video presenting our experiment is available at https://goo.gl/NHXKqR.

Motivation & Objective

• To develop a framework for applying reinforcement learning in the Gazebo robotics simulator for bipedal walking.
• To implement and evaluate Deep Deterministic Policy Gradient (DDPG) for continuous control in a planar bipedal walker.
• To compare the learned gait pattern of the robot with actual human walking data from optical motion capture.
• To demonstrate model-free learning of stable, high-speed walking without prior knowledge of robot or environment dynamics.

Proposed method

• The bipedal walker was modeled in SolidWorks and converted to URDF format for integration into the Gazebo simulator.
• The robot’s state space, consisting of 12 variables (hip/knee angles and velocities, linear velocities, ground contact), was used as input to the DDPG agent.
• The action space was defined with 4 continuous control outputs for hip and knee joint torques.
• A custom reward function was designed to encourage forward motion, stability, and energy efficiency, with higher rewards for faster, smoother gaits.
• The DDPG algorithm was trained on an NVIDIA GTX 1050 Ti GPU over approximately 41 hours using a replay buffer and target networks.
• ROS served as the communication interface between the learning agent and the Gazebo simulation at a 50 Hz update rate.

Experimental results

Research questions

• RQ1 Can DDPG successfully learn stable, continuous bipedal walking in a physics-based simulation without prior knowledge of dynamics?
• RQ2 How does the gait pattern of a DDPG-trained bipedal robot compare to that of a human in terms of joint angle trajectories and phase relationships?
• RQ3 What reward shaping is most effective in promoting high-speed, stable walking in a simulated bipedal robot?
• RQ4 To what extent can model-free reinforcement learning replicate human-like locomotion in a planar bipedal system?

Key findings

• The DDPG-trained bipedal walker achieved stable forward walking for 10 meters without falling after 41 hours of training.
• The robot reached an average walking speed of 0.83 m/s, demonstrating efficient and dynamic locomotion.
• The hip joint angle trajectories of the robot were approximately 180 degrees out of phase, closely matching the human gait pattern observed in motion capture.
• The knee joint angle frequency was approximately double that of the hip, mirroring the human walking gait, as confirmed by optical motion capture data.
• The robot’s gait pattern exhibited similar characteristics to human walking, including phase-locked joint movements and smooth transitions.
• The simulation results show that DDPG can effectively learn complex, nonlinear control policies in continuous action spaces for bipedal locomotion.