[Paper Review] DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing

Mengshi Zhang, Yuqun Zhang+3 more|arXiv (Cornell University)|Feb 7, 2018

Software Testing and Debugging TechniquesComputer Science12 references63 citations

TL;DR

DeepRoad uses GAN-based image-to-image translation to synthesize authentic weather-condition driving scenes and applies metamorphic testing to detect inconsistencies in DNN-based autonomous driving systems under extreme weather (snow/rain).

ABSTRACT

While Deep Neural Networks (DNNs) have established the fundamentals of DNN-based autonomous driving systems, they may exhibit erroneous behaviors and cause fatal accidents. To resolve the safety issues of autonomous driving systems, a recent set of testing techniques have been designed to automatically generate test cases, e.g., new input images transformed from the original ones. Unfortunately, many such generated input images often render inferior authenticity, lacking accurate semantic information of the driving scenes and hence compromising the resulting efficacy and reliability. In this paper, we propose DeepRoad, an unsupervised framework to automatically generate large amounts of accurate driving scenes to test the consistency of DNN-based autonomous driving systems across different scenes. In particular, DeepRoad delivers driving scenes with various weather conditions (including those with rather extreme conditions) by applying the Generative Adversarial Networks (GANs) along with the corresponding real-world weather scenes. Moreover, we have implemented DeepRoad to test three well-recognized DNN-based autonomous driving systems. Experimental results demonstrate that DeepRoad can detect thousands of behavioral inconsistencies in these systems.

Motivation & Objective

• Motivate robust testing of DNN-based autonomous driving systems beyond simple image filters.
• Introduce an unsupervised GAN-based framework to synthesize authentic driving scenes under extreme weather.
• Define metamorphic relations to test consistency of driving decisions across weather-transformed scenes.
• Evaluate the framework on real-world autonomous driving models to reveal robustness gaps.

Proposed method

• Use UNIT (GAN+VAE) to perform unsupervised image-to-image translation between scene domains (fine vs snowy/rainy).
• Train UNIT with unpaired real driving images from two domains to learn a shared latent space.
• Apply trained UNIT to transform each original driving scene into weather-transformed versions tau(i).
• Feed original and transformed image pairs into autonomous driving DNNs and compare steering outputs to detect inconsistencies.
• Quantify inconsistency via a thresholded difference in predicted steering angles across i and tau(i).
• Evaluate on three Udacity driving models (Autumn, Chauffeur, Rwightman) using Udacity and YouTube-sourced weather data.

Experimental results

Research questions

• RQ1 Can GAN-based metamorphic transformations produce authentic-looking driving scenes across weather conditions?
• RQ2 Do DNN-based autonomous driving systems exhibit inconsistent steering predictions when driving scenes are transformed to snowy or rainy conditions?
• RQ3 Which models demonstrate greater robustness or vulnerability under metamorphic weather transformations?
• RQ4 How do varying error bounds affect the detected inconsistencies across models?

Key findings

• DeepRoad identifies thousands of inconsistent steering predictions across three Udacity driving models under snowy and rainy conditions.
• Autumn shows the highest inconsistency in both weather scenarios, while Rwightman is the most stable across conditions.
• Under rainy conditions with a 10° error bound, detected inconsistencies are 5279 (Autumn), 710 (Chauffeur), and 656 (Rwightman).
• Under snowy conditions with a 10° error bound, detected inconsistencies are 11635 (Autumn), 4839 (Chauffeur), and 334 (Rwightman).
• Increasing the error bound generally reduces the number of detected inconsistencies for each model, indicating robustness differences.
• GAN-generated scenes qualitatively resemble real weather scenes and preserve major semantic content such as road structure and objects.