[Paper Review] MultiNet: Multi-Modal Multi-Task Learning for Autonomous Driving

Sauhaarda Chowdhuri, Tushar Pankaj+1 more|arXiv (Cornell University)|Sep 16, 2017

Autonomous Vehicle Technology and Safety27 references4 citations

TL;DR

This paper proposes MultiNet, a multi-modal multi-task deep learning framework that integrates behavioral mode as a conditional input to a shared convolutional network, enabling a single model to learn distinct driving behaviors like direct and furtive driving. The method outperforms separate multi-task networks, achieving 8.31% higher autonomy in direct mode and significantly lower validation loss, while using fewer parameters.

ABSTRACT

Autonomous driving requires operation in different behavioral modes ranging from lane following and intersection crossing to turning and stopping. However, most existing deep learning approaches to autonomous driving do not consider the behavioral mode in the training strategy. This paper describes a technique for learning multiple distinct behavioral modes in a single deep neural network through the use of multi-modal multi-task learning. We study the effectiveness of this approach, denoted MultiNet, using self-driving model cars for driving in unstructured environments such as sidewalks and unpaved roads. Using labeled data from over one hundred hours of driving our fleet of 1/10th scale model cars, we trained different neural networks to predict the steering angle and driving speed of the vehicle in different behavioral modes. We show that in each case, MultiNet networks outperform networks trained on individual modes while using a fraction of the total number of parameters.

Motivation & Objective

• Address the lack of datasets and models for unstructured autonomous driving, such as sidewalks and unpaved roads.
• Overcome limitations of single-task learning and conventional multi-task learning in handling diverse driving behaviors.
• Develop a unified deep learning architecture that shares parameters across multiple behavioral modes while maintaining mode-specific performance.
• Create a scalable and efficient solution for autonomous driving in complex, real-world environments using a single neural network.
• Demonstrate that incorporating behavioral mode as a conditional input improves generalization and reduces model complexity compared to training separate networks.

Proposed method

• Introduce a novel multi-modal multi-task learning framework where behavioral mode is concatenated with image features to condition the network's output.
• Design a deep convolutional neural network, Z2Color, for end-to-end steering and speed prediction using stereo images and behavioral mode as inputs.
• Use a soft parameter sharing strategy across tasks (steering and speed prediction) while conditioning on behavioral mode to enable parameter efficiency.
• Implement a modified DAgger algorithm using live corrective data from model car experiments to improve robustness and generalization.
• Train the MultiNet model on a custom dataset of 100+ hours of 1/10th scale model car driving in unstructured environments, including sidewalks and trails.
• Evaluate performance using both validation loss on a held-out dataset and live on-road experiments with autonomy metrics and qualitative analysis of behavior.

Experimental results

Research questions

• RQ1 Can a single deep neural network effectively learn multiple distinct driving behaviors—such as direct and furtive driving—by conditioning on behavioral mode?
• RQ2 How does MultiNet’s performance compare to conventional multi-task learning and single-task networks in terms of accuracy, parameter efficiency, and generalization?
• RQ3 To what extent does incorporating behavioral mode as a conditional input improve model performance on unstructured driving tasks?
• RQ4 Does the validation loss metric correlate with real-world autonomy performance, especially in complex behavioral modes?
• RQ5 Can live corrective data from model car experiments improve the robustness of the MultiNet architecture in real-world deployment?

Key findings

• MultiNet achieved 92.68% autonomy in direct mode, outperforming the MTL baseline by 8.31 percentage points.
• In furtive mode, MultiNet achieved 88.23% autonomy compared to 87.55% for MTL, with a 12.58% reduction in validation loss, indicating superior learning of complex behaviors.
• The validation loss difference between MultiNet and MTL was 8.16% in direct mode and 12.58% in furtive mode, showing that the loss metric effectively captures performance gains beyond simple path-following.
• MultiNet demonstrated more pronounced furtive behaviors—such as speed modulation near obstacles and boundary following—compared to MTL networks, which were less adaptive.
• The MSE loss metric was found to be a valid proxy for real-world performance in direct mode but more sensitive than autonomy in furtive mode, indicating it captures subtle behavioral learning.
• MultiNet used a fraction of the parameters compared to training separate networks per mode, confirming its parameter efficiency and scalability.