[Paper Review] Sim-to-Real Strategy for Spatially Aware Robot Navigation in Uneven Outdoor Environments

Kasun Weerakoon, Adarsh Jagan Sathyamoorthy+1 more|arXiv (Cornell University)|May 18, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a sim-to-real strategy for stable robot navigation in uneven outdoor terrains by leveraging intermediate attention features from a pre-trained DRL policy to generate a navigation cost map. The method combines DWA with adaptive velocity constraints based on IMU and elevation gradients, reducing vibration by 13.09% and average speed by 19.33% while improving success rate by 5% over end-to-end DRL in real-world outdoor environments.

ABSTRACT

Deep Reinforcement Learning (DRL) is hugely successful due to the availability of realistic simulated environments. However, performance degradation during simulation to real-world transfer still remains a challenging problem for the policies trained in simulated environments. To close this sim-to-real gap, we present a novel hybrid architecture that utilizes an intermediate output from a fully trained attention DRL policy as a navigation cost map for outdoor navigation. Our attention DRL network incorporates a robot-centric elevation map, IMU data, the robot's pose, previous actions, and goal information as inputs to compute a navigation cost-map that highlights non-traversable regions. We compute least-cost waypoints on the cost map and utilize the Dynamic Window Approach (DWA) with velocity constraints on high cost regions to follow the waypoints in highly uneven outdoor environments. Our formulation generates dynamically feasible velocities along stable, traversable regions to reach the robot's goals. We observe an increase of 5% in terms of success rate, 13.09% of the decrease in average robot vibration, and a 19.33% reduction in average velocity compared to end-to-end DRL method and state-of-the-art methods in complex outdoor environments. We evaluate the benefits of our method using a Clearpath Husky robot in both simulated and real-world uneven environments.

Motivation & Objective

• To address the sim-to-real gap in deep reinforcement learning for robot navigation in complex, uneven outdoor environments.
• To improve robot stability and reduce vibration during navigation on rough, sloped terrains such as hills and rocks.
• To enable consistent performance across both simulated and real-world outdoor environments by decoupling perception from control.
• To develop a hybrid architecture that uses intermediate DRL features for terrain-aware cost map generation and velocity-constrained trajectory planning.
• To minimize reliance on heuristic or human-annotated terrain features by learning from raw sensor inputs (point clouds, IMU, pose, goal)

Proposed method

• A novel attention-based DRL network processes robot-centric elevation maps, IMU data, robot pose, previous actions, and goal information to generate a navigation cost map highlighting non-traversable regions.
• The cost map is used to compute least-cost waypoints, which guide the robot along stable, traversable paths with minimal elevation gradients.
• Dynamic Window Approach (DWA) is enhanced with velocity space constraints that penalize high velocities on steep or rough terrain based on real-time vibration and orientation feedback.
• IMU and elevation gradient-based rewards are used during DRL training to identify unstable regions and encourage safe navigation behavior.
• The system decouples perception (via DRL) from control (via DWA), enabling transferability from simulation to real-world deployment.
• The method uses 3D point clouds processed into 2D robot-centric elevation maps as input, enabling spatial awareness in unstructured outdoor environments.

Experimental results

Research questions

• RQ1 Can intermediate features from a pre-trained DRL policy improve sim-to-real transfer in outdoor robot navigation?
• RQ2 How can velocity constraints be dynamically adapted to reduce robot vibration and prevent flip-overs on uneven terrains?
• RQ3 Does a hybrid architecture combining DRL-based perception with DWA-based control outperform end-to-end DRL in real-world outdoor navigation?
• RQ4 To what extent does terrain-aware velocity control improve stability and success rate in complex outdoor environments?
• RQ5 Can the method maintain comparable performance in both simulation and real-world scenarios, especially on steep or rough surfaces?

Key findings

• The proposed method achieved a 5% higher success rate compared to end-to-end DRL in real-world outdoor scenarios, with 89% success in Scenario 2 and 78% in Scenario 3.
• Average robot vibration was reduced by 13.09% compared to end-to-end DRL, with the lowest average vibration (0.079) among all methods in real-world Scenario 3.
• Average navigation speed was reduced by 19.33% compared to end-to-end DRL, achieving 0.357 m/s in real-world Scenario 3, indicating safer, more controlled motion.
• The method maintained high performance in both simulation and real-world environments, unlike end-to-end DRL, which showed significant degradation in real-world transfer.
• Trajectories generated by the method were more stable and less prone to oscillation or instability, particularly on steep slopes, due to adaptive velocity constraints.
• The approach produced longer but safer trajectories than DWA and TERP, prioritizing stability and reduced vibration over path length.