[Paper Review] Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts?

Angelos Filos, Panagiotis Tigkas+4 more|arXiv (Cornell University)|Jun 26, 2020

Mining Techniques and Economics53 citations

TL;DR

The paper introduces robust imitative planning (RIP) and adaptive RIP (AdaRIP) which use epistemic uncertainty to detect distribution shifts in autonomous driving, recover from them, and optionally query an expert for online adaptation; it also presents CARNOVEL, a novel out-of-distribution driving benchmark.

ABSTRACT

Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment, typically leading to arbitrary deductions and poorly-informed decisions. In principle, detection of and adaptation to OOD scenes can mitigate their adverse effects. In this paper, we highlight the limitations of current approaches to novel driving scenes and propose an epistemic uncertainty-aware planning method, called \emph{robust imitative planning} (RIP). Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. If the model's uncertainty is too great to suggest a safe course of action, the model can instead query the expert driver for feedback, enabling sample-efficient online adaptation, a variant of our method we term \emph{adaptive robust imitative planning} (AdaRIP). Our methods outperform current state-of-the-art approaches in the nuScenes \emph{prediction} challenge, but since no benchmark evaluating OOD detection and adaption currently exists to assess \emph{control}, we introduce an autonomous car novel-scene benchmark, exttt{CARNOVEL}, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.

Motivation & Objective

• Motivate the need for detecting and handling out-of-training-distribution (OOD) scenarios in autonomous driving.
• Develop an epistemic uncertainty-aware planning method to detect and recover from distribution shifts.
• Enable online, sample-efficient adaptation by querying an expert when uncertainty is high (AdaRIP).
• Benchmark robustness to novel scenes with a new autonomous car novel-scene benchmark (CARNOVEL).
• Evaluate against state-of-the-art baselines on real (nuScenes) and simulated data.

Proposed method

• Train an autoregressive Bayesian imitative model q(y|x;θ) to imitate expert trajectories using maximum likelihood with an ensemble of θ indicators to approximate p(θ|D).
• Detect distribution shifts via the epistemic uncertainty measure u(y)=Varθ|D[log q(y|x;θ)].
• Plan under epistemic uncertainty by RIP, optimizing y to maximize the imitation prior plus a goal likelihood under the posterior over θ (aggregation operator ⊕).
• Use two aggregation operators: RIP-WCM (worst-case model minimax) and RIP-MA (model averaging via Bayesian posterior), with a mention of alternatives like CVaR.
• AdaRIP augments RIP with online expert feedback to adapt when uncertainty is high, enabling safe online improvement.
• Demonstrate superiority through new benchmarks and real-data experiments.

Experimental results

Research questions

• RQ1 Q1. Can autonomous driving methods detect distribution shifts without explicit OOD cues?
• RQ2 Q2. How robust are these methods in recovering from distribution shifts?
• RQ3 Q3. Can RIP’s epistemic uncertainty quantify novel or out-of-distribution scenes effectively?
• RQ4 Q4. Does the explicit recovery mechanism (AdaRIP) improve performance under distribution shifts?
• RQ5 Q5. How do RIP variants compare to state-of-the-art baselines on real and simulated data?

Key findings

• RIP and AdaRIP outperform state-of-the-art baselines on nuScenes prediction tasks and in CARNOVEL under distribution shifts.
• Epistemic uncertainty enables detection of distribution shifts via model disagreement across an ensemble.
• RIP-WCM and RIP-MA provide robust planning under uncertainty, with RIP-MA leveraging model averaging and RIP-WCM using worst-case consideration.
• AdaRIP yields online sample-efficient adaptation by querying the expert when uncertainty is high.
• CARNOVEL demonstrates robustness across tasks like abnormal turns, hills, and roundabouts in simulated driving scenarios.
• RIP-based methods show favorable performance gains over DIM and other baselines in both real and simulated datasets.