[Paper Review] Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Alex Kendall, Yarin Gal+1 more|arXiv (Cornell University)|May 19, 2017

Domain Adaptation and Few-Shot Learning44 references500 citations

TL;DR

The paper introduces a principled multi-task loss that uses homoscedastic (task) uncertainty to automatically weight losses for semantic segmentation, instance segmentation, and depth regression from a single monocular image, outperforming single-task models.

ABSTRACT

Numerous deep learning applications benefit from multi-task learning with multiple regression and classification objectives. In this paper we make the observation that the performance of such systems is strongly dependent on the relative weighting between each task's loss. Tuning these weights by hand is a difficult and expensive process, making multi-task learning prohibitive in practice. We propose a principled approach to multi-task deep learning which weighs multiple loss functions by considering the homoscedastic uncertainty of each task. This allows us to simultaneously learn various quantities with different units or scales in both classification and regression settings. We demonstrate our model learning per-pixel depth regression, semantic and instance segmentation from a monocular input image. Perhaps surprisingly, we show our model can learn multi-task weightings and outperform separate models trained individually on each task.

Motivation & Objective

• Motivate the need for effective loss weighting in multi-task learning for scene understanding.
• Propose a principled multi-task loss based on homoscedastic uncertainty to balance regression and classification losses.
• Design a unified encoder-decoder architecture that outputs semantic segmentation, instance segmentation, and depth simultaneously.
• Demonstrate that jointly learning geometry and semantics can improve performance over single-task models.

Proposed method

• Formulate a probabilistic multi-task loss by modeling each task with a Gaussian (regression) or Softmax likelihood (classification) and introducing a task-specific noise parameter (uncertainty).
• Derive a joint loss that weights task losses by inverses of their (learned) noise terms, with a regularising log-variance term to prevent degenerate solutions.
• Train a DeepLabV3-based encoder with task-specific decoders for semantic segmentation, instance segmentation, and depth regression.
• Use an instance centroid regression approach for instance segmentation where per-pixel votes point to instance centroids and clustering (OPTICS) yields final instances.
• Predict per-pixel inverse depth for depth regression and employ a cross-entropy loss for semantic segmentation, combining them through the learned uncertainty-based weights.

Experimental results

Research questions

• RQ1 Can homoscedastic (task) uncertainty be used to automatically balance losses across multiple tasks in a deep network?
• RQ2 Does a single multi-task model with uncertainty-weighted losses outperform independently trained single-task models for scene geometry and semantics?
• RQ3 How does joint learning of depth, semantic, and instance segmentation affect performance and generalisation on road scene understanding?
• RQ4 What is the architectural arrangement that best supports joint semantic, instance, and depth prediction from monocular input?

Key findings

• Uncertainty-based task weighting improves multi-task performance over naïve or uniformly weighted losses.
• The proposed approach can outperform separate single-task models and even approximate or exceed optimally tuned weights obtained by grid search.
• Joint learning yields competitive or superior results across semantic segmentation, instance segmentation, and depth estimation on Cityscapes subset and full dataset.
• The model achieves state-of-the-art-like results for all three tasks with a single network, demonstrating effective shared representation learning.
• The learned task weights evolve during training and tend to favour semantic segmentation, depth regression, and instance segmentation in specific ratios-by the end of training the reported ratios were 43:1:0.16 for semantic, depth, and instance respectively (final model).