[Paper Review] Learning a Driving Simulator

Eder Santana, George Hotz|arXiv (Cornell University)|Aug 3, 2016

Generative Adversarial Networks and Image SynthesisComputer Science15 references183 citations

TL;DR

The paper learns to simulate driving scenes by embedding road frames into a Gaussian latent space with a variational autoencoder guided by GAN-based costs, then models transitions with an action-conditioned RNN to predict future frames without end-to-end pixel-space optimization.

ABSTRACT

Comma.ai's approach to Artificial Intelligence for self-driving cars is based on an agent that learns to clone driver behaviors and plans maneuvers by simulating future events in the road. This paper illustrates one of our research approaches for driving simulation. One where we learn to simulate. Here we investigate variational autoencoders with classical and learned cost functions using generative adversarial networks for embedding road frames. Afterwards, we learn a transition model in the embedded space using action conditioned Recurrent Neural Networks. We show that our approach can keep predicting realistic looking video for several frames despite the transition model being optimized without a cost function in the pixel space.

Motivation & Objective

• Motivate vision-based approaches for self-driving cars that rely on learned video prediction rather than hand-crafted simulators.
• Develop a compact latent representation of road video frames for tractable dynamics prediction.
• Demonstrate that a learned transition model can predict realistic future frames in the latent space while remaining within high-density regions.
• Release driving dataset and code to foster further research in learned driving simulation.

Proposed method

• Embed road frames into a Gaussian latent space using a VAE/GAN hybrid autoencoder to produce a compact representation z_t of size 2048.
• Train a transition model as an action-conditioned RNN to predict z_{t+1} from z_t and controls S_t, A_t.
• Decode predicted latent codes with a GAN-based generator to render future frames.
• Use teacher forcing for the first portion of sequences and hallucination (feedback) for subsequent frames to train the transition model.
• Optimize the autoencoder with a combined loss including KL divergence, discriminator-based feature matching, and GAN loss as in Larsen et al. 2015.

Experimental results

Research questions

• RQ1 Can a learned latent space capture the essential structure of real driving scenes for video prediction?
• RQ2 Is a transition model in latent space capable of generating realistic future frames conditioned on vehicle controls?
• RQ3 Does combining VAE and GAN objectives yield visually plausible frames compared to pixel-space MSE training?
• RQ4 How far can a learned driving simulator predict realistic road scenes into the future?
• RQ5 What datasets and code are released to enable replication and further research on learned driving simulators?

Key findings

• Autoencoder with GAN-based cost produces more realistic road textures than MSE-based reconstruction.
• The transition model maintains road structure and can generate plausible sequences up to around 100 frames, including events like passing lanes and changing lead vehicles.
• The transition model sometimes struggles with curves, tending to straighten lanes in curved scenarios, suggesting room for more advanced sequence models and context fusion.
• Sampling with different seeds yields varied driving events, indicating the model learns meaningful dynamics in the latent space.
• A compact latent representation (2048-dim) is achieved, enabling scalable prediction in latent space while staying within high-density regions of the latent prior.