[Paper Review] Demo Abstract: Real-Time Out-of-Distribution Detection on a Mobile Robot

Michael Yuhas, Arvind Easwaran|arXiv (Cornell University)|Nov 15, 2022

Autonomous Vehicle Technology and Safety4 citations

TL;DR

This paper presents a real-time out-of-distribution (OOD) detection system for mobile robots using a YOLOv7 tiny object detector and a variational autoencoder-based OOD detector, deployed on a Jetson Nano-equipped Duckiebot. The system successfully identifies OOD samples (e.g., simulated snowfall) in under 800 ms, triggers emergency braking without false positives, and maintains real-time performance for lane following and object detection under shared computational load.

ABSTRACT

In a cyber-physical system such as an autonomous vehicle (AV), machine learning (ML) models can be used to navigate and identify objects that may interfere with the vehicle's operation. However, ML models are unlikely to make accurate decisions when presented with data outside their training distribution. Out-of-distribution (OOD) detection can act as a safety monitor for ML models by identifying such samples at run time. However, in safety critical systems like AVs, OOD detection needs to satisfy real-time constraints in addition to functional requirements. In this demonstration, we use a mobile robot as a surrogate for an AV and use an OOD detector to identify potentially hazardous samples. The robot navigates a miniature town using image data and a YOLO object detection network. We show that our OOD detector is capable of identifying OOD images in real-time on an embedded platform concurrently performing object detection and lane following. We also show that it can be used to successfully stop the vehicle in the presence of unknown, novel samples.

Motivation & Objective

• To address the safety risk posed by machine learning models failing on out-of-distribution (OOD) data in autonomous systems like mobile robots.
• To develop an OOD detection system that operates in real time on embedded hardware without disrupting critical control tasks.
• To validate that OOD detection can trigger emergency braking before collisions in response to novel, OOD conditions such as simulated snowfall.
• To evaluate the impact of concurrent task execution on response times for OOD detection, object detection, and lane following.
• To demonstrate the feasibility of using a lightweight, independently deployed OOD detector as a real-time safety monitor on a physical robot platform.

Proposed method

• A Duckiebot equipped with a Jetson Nano 2GB and PREEMPT_RT kernel was used as a surrogate for an autonomous vehicle.
• The system runs three concurrent tasks: lane following via traditional computer vision, object detection using a quantized YOLOv7 tiny model, and OOD detection using a variational autoencoder on optical flow features.
• Optical flow matrices were generated from sequential images using the Farnebäck algorithm and used as input to the OOD detector.
• A genetic algorithm optimized preprocessing parameters (input size, number of flows, interpolation method) to maximize F1 score and minimize inference time.
• The final OOD detector used 60x80 input size, 10 optical flow frames, and bilinear interpolation, achieving a mean inference time of 53.3 ms.
• All models were quantized to 8-bit using dynamic quantization and executed via the QNNPACK backend on AArch64.

Experimental results

Research questions

• RQ1 Can an OOD detector be deployed on an embedded platform and meet real-time deadlines while sharing resources with other safety-critical tasks?
• RQ2 Does the OOD detector successfully trigger emergency braking in response to novel, OOD conditions such as simulated snowfall?
• RQ3 To what extent does concurrent execution of OOD detection, object detection, and lane following degrade response times, particularly for the object detector?
• RQ4 Can the OOD detector maintain low false-positive rates while detecting previously unseen environmental disturbances?
• RQ5 How does the choice of input representation (e.g., optical flow) and model configuration affect the trade-off between detection accuracy and inference speed?

Key findings

• The OOD detector achieved a mean inference time of 53.3 ms and a maximum response time of less than 800 ms, satisfying real-time constraints.
• The OOD detector successfully triggered emergency braking when confetti was used to simulate snowfall, demonstrating effective OOD sample detection.
• In the absence of OOD conditions, the system did not generate false positives, maintaining safe navigation along the track.
• The lane follower maintained its real-time deadlines even when all three tasks were running concurrently.
• The object detection task experienced significant performance degradation (increased response time) due to high system utilization and memory swapping under concurrent workloads.
• The genetic algorithm identified a configuration (60x80 input size, 10 flows, bilinear interpolation) that achieved a high F1 score of 0.97 while maintaining low inference time.