[Paper Review] End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception

Zhengyuan Yang, Yixuan Zhang+3 more|arXiv (Cornell University)|Jan 20, 2018

Autonomous Vehicle Technology and Safety17 references40 citations

TL;DR

The paper proposes a multi-modal multi-task network that jointly predicts steering angle and speed (including discrete speed commands and continuous speed values) from front-view images and prior speeds, evaluated on Udacity and a new SAIC dataset, with data-synthesis methods to mitigate error accumulation in road tests.

ABSTRACT

Convolutional Neural Networks (CNN) have been successfully applied to autonomous driving tasks, many in an end-to-end manner. Previous end-to-end steering control methods take an image or an image sequence as the input and directly predict the steering angle with CNN. Although single task learning on steering angles has reported good performances, the steering angle alone is not sufficient for vehicle control. In this work, we propose a multi-task learning framework to predict the steering angle and speed control simultaneously in an end-to-end manner. Since it is nontrivial to predict accurate speed values with only visual inputs, we first propose a network to predict discrete speed commands and steering angles with image sequences. Moreover, we propose a multi-modal multi-task network to predict speed values and steering angles by taking previous feedback speeds and visual recordings as inputs. Experiments are conducted on the public Udacity dataset and a newly collected SAIC dataset. Results show that the proposed model predicts steering angles and speed values accurately. Furthermore, we improve the failure data synthesis methods to solve the problem of error accumulation in real road tests.

Motivation & Objective

• Motivate end-to-end vehicle control beyond steering angle by jointly predicting speed and steering.
• Develop a multi-task framework that can predict discrete speed commands initially and refine to speed values.
• Incorporate prior feedback speeds as an extra modality to improve speed and steering predictions.
• Collect and evaluate on Udacity and a newly collected SAIC dataset (day and night driving).
• Address failure data synthesis to mitigate error accumulation in real-world road tests.

Proposed method

• Propose a base CNN model for end-to-end steering angle prediction with larger kernels and 1:1 input aspect ratio.
• Introduce a discrete speed command network that predicts accelerating, decelerating, or maintaining speed from image sequences using an LSTM for temporal context.
• Develop a multi-modal multi-task network that adds a speed encoder (for prior speeds) and fuses visual features with speed features to predict both steering angle and speed.
• Use mean absolute error (MAE) for steering angle, cross-entropy for discrete speed commands, and a weighted joint loss over tasks.
• Apply data augmentation including HSV color space transformations, small rotations, horizontal flips, and side-camera data synthesis to generate failure cases.
• Evaluate on Udacity and SAIC datasets, comparing with PilotNet and other baselines, and perform road tests with extended data synthesis to address error accumulation.

Experimental results

Research questions

• RQ1 Can end-to-end learning predict both steering angle and speed (discrete commands and/or continuous values) from visual input?
• RQ2 Does incorporating previous feedback speeds as an additional modality improve prediction accuracy for steering and speed?
• RQ3 How does the proposed multi-modal multi-task network compare to single-task or purely vision-based steering models on standard driving datasets?
• RQ4 What data augmentation and failure-case synthesis strategies improve real-world robustness and reduce error accumulation in road tests?
• RQ5 How does the newly collected SAIC dataset (day and night) affect model performance and generalization?

Key findings

• Steering angle MAE: Multi-modal Multi-task Network achieves 1.26 degrees on Udacity after improvements over base and discrete speed models.
• Base Steering Model achieves 2.84 degrees MAE, outperforming PilotNet and a simpler Cg Network in the evaluation setup.
• Discrete Speed Command Network reaches 1.85 degrees MAE for steering when predicting discrete commands, indicating benefits from joint training.
• Speed value prediction MAE: 0.19 m/s on Udacity and 0.45 m/s on SAIC with the multi-modal multi-task network.
• Speed values and steering angle predictions improve when using previous speeds as an extra input modality.
• SAIC dataset includes day and night driving; two hours of data after filtering are used for training/validation/testing.
• Road tests demonstrate that data augmentation and synthetic failure cases help mitigate error accumulation and enable autonomous driving under test conditions.