[Paper Review] Verifying Learning-Based Robotic Navigation Systems

Guy Amir, Davide Corsi+5 more|arXiv (Cornell University)|May 26, 2022

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper presents a verification-driven approach to selecting optimal deep reinforcement learning (DRL) policies for robotic navigation, using off-the-shelf DNN verification tools to detect unsafe behaviors like collisions and infinite loops, and suboptimal behaviors such as overly conservative paths. The method enables effective model selection in real-world robotic systems, outperforming gradient-based attacks and demonstrating practical safety improvements on a physical robot.

ABSTRACT

Deep reinforcement learning (DRL) has become a dominant deep-learning paradigm for tasks where complex policies are learned within reactive systems. Unfortunately, these policies are known to be susceptible to bugs. Despite significant progress in DNN verification, there has been little work demonstrating the use of modern verification tools on real-world, DRL-controlled systems. In this case study, we attempt to begin bridging this gap, and focus on the important task of mapless robotic navigation -- a classic robotics problem, in which a robot, usually controlled by a DRL agent, needs to efficiently and safely navigate through an unknown arena towards a target. We demonstrate how modern verification engines can be used for effective model selection, i.e., selecting the best available policy for the robot in question from a pool of candidate policies. Specifically, we use verification to detect and rule out policies that may demonstrate suboptimal behavior, such as collisions and infinite loops. We also apply verification to identify models with overly conservative behavior, thus allowing users to choose superior policies, which might be better at finding shorter paths to a target. To validate our work, we conducted extensive experiments on an actual robot, and confirmed that the suboptimal policies detected by our method were indeed flawed. We also demonstrate the superiority of our verification-driven approach over state-of-the-art, gradient attacks. Our work is the first to establish the usefulness of DNN verification in identifying and filtering out suboptimal DRL policies in real-world robots, and we believe that the methods presented here are applicable to a wide range of systems that incorporate deep-learning-based agents.

Motivation & Objective

• To bridge the gap between formal DNN verification and real-world DRL-controlled robotic systems.
• To address the lack of formal safety guarantees in learning-based robotic navigation, especially for multistep, reactive behaviors.
• To enable effective model selection by identifying and filtering out suboptimal or unsafe DRL policies.
• To provide insights into training stability and policy improvement across multiple training iterations.
• To demonstrate the practical applicability of formal verification in real robotic platforms, beyond simulation.

Proposed method

• Leveraging off-the-shelf DNN verification engines to analyze DRL policies for safety and liveness properties.
• Formally verifying properties such as collision avoidance, path length minimization, and absence of infinite loops across multiple policy invocations.
• Applying verification to detect both unsafe behaviors (e.g., collisions) and suboptimal behaviors (e.g., overly conservative paths).
• Using verification results to perform model selection from a pool of candidate DRL policies.
• Comparing verification-based filtering with gradient-based attacks to evaluate robustness and detection capability.
• Validating findings on a physical robot to confirm that verified suboptimal policies indeed exhibit flawed behavior in practice.

Experimental results

Research questions

• RQ1 Can formal DNN verification techniques detect unsafe and suboptimal behaviors in real-world DRL-based robotic navigation systems?
• RQ2 How does verification-based model selection compare to gradient-based attack methods in identifying flawed policies?
• RQ3 To what extent can formal verification provide insights into the stability and improvement of DRL training processes across iterations?
• RQ4 Can verification tools effectively identify policies that cause infinite loops or collisions in multistep navigation tasks?
• RQ5 How scalable and practical is the integration of formal verification into the real-world deployment pipeline of DRL agents?

Key findings

• Policies flagged as suboptimal by the verification framework were confirmed to exhibit collisions and infinite loops during real-world testing on a physical robot.
• The verification-based approach successfully identified and filtered out policies with overly conservative behavior, enabling selection of shorter, more efficient navigation paths.
• The method outperformed state-of-the-art gradient-based attacks in detecting problematic policies, particularly in identifying non-obvious liveness and safety violations.
• Verification revealed meaningful differences in policy behavior across training iterations, offering insights into training stability and convergence.
• The approach demonstrated practical feasibility and reliability in real-world deployment, with results confirmed via physical robot experiments and a public video demonstration.