[Paper Review] Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

Mikael Henaff, Alfredo Canziani+1 more|arXiv (Cornell University)|Jan 8, 2019

Gaussian Processes and Bayesian Inference40 references78 citations

TL;DR

The paper learns driving policies from purely observational data by training a stochastic forward model and a policy that optimizes a policy cost plus an uncertainty regularizer, enabling multi-step backpropagation through the learned dynamics without environment interaction.

ABSTRACT

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

Motivation & Objective

• Motivate learning policies from observational driving data where environment interaction is costly or dangerous.
• Propose a two-stage approach: learn an action-conditional forward model from data, then train a policy with backpropagation through the unrolled model.
• Introduce an uncertainty cost based on model prediction uncertainty to discourage states far from training data.
• Demonstrate that uncertainty-regularized model-based planning improves policy quality in dense-traffic driving.
• Release dataset and environment to facilitate further research.

Proposed method

• Train an action-conditional forward model f_theta(s_1:t, a_t, z_t) using a VAE-style latent z_t and a posterior q_phi with z-dropout to keep responsiveness to actions.
• Unroll the forward model over horizon T and backpropagate a differentiable loss combining a policy cost C and an uncertainty cost U, where U is the trace of the covariance of forward predictions under multiple dropout masks.
• Estimate uncertainty U via dropout-based approximations, computing Var across K forward passes and taking the trace: U(s_hat_{t+1}) = tr(Cov[{f_theta_k(s_1:t,a_t,z_t)}_{k=1}^K]).
• Optionally relate the forward model to Bayesian neural networks and describe how the posterior over latent variables and dropout weights approximates the true posterior.
• Define two variants: MPUR (Model-Predictive Policy with Uncertainty Regularization) and MPER (Model-Predictive Policy with Expert Regularization).
• Apply the learned dynamics to train a stochastic policy pi_psi by gradient-based optimization over rolled-out trajectories with gradients flowing through the unrolled model.

Experimental results

Research questions

• RQ1 Can a policy be learned from purely observational driving data without environment interaction by penalizing divergence from training data trajectories?
• RQ2 Does incorporating an uncertainty regularizer in the forward-model-based policy learning improve performance in dense-traffic driving?
• RQ3 How does a modified posterior for latent variables (z-dropout) affect responsiveness to actions and policy performance in stochastic dynamics models?
• RQ4 What is the impact of longer rollout horizons on matching the induced state distribution to the training manifold?

Key findings

• MPUR and MPER policies substantially outperform baselines learned from observational data, including imitation-learning and SVG/VG variants.
• Including the uncertainty cost is essential; removing it (VG) yields high uncertainty and poorer real-environment performance.
• Longer rollout horizons markedly improve policy performance across methods, with stochastic models and z-dropout providing the best gains.
• The modified posterior with z-dropout improves responsiveness to actions and boosts policy success when using stochastic dynamics.
• The MPUR approach can achieve near-human performance on dense traffic driving tasks within the observational-data regime.
• Policy and environment results are supported by quantitative metrics and qualitative trajectory analysis.