[Paper Review] Auxiliary Tasks in Multi-task Learning

Lukas Liebel, Marco Körner|arXiv (Cornell University)|May 16, 2018

Advanced Neural Network ApplicationsComputer Science31 references178 citations

TL;DR

The paper proposes adding auxiliary tasks to multi-task CNNs to regularize learning and boost main task performance, demonstrated on vision-based road scene understanding with a synthetic multi-task dataset synMT.

ABSTRACT

Multi-task convolutional neural networks (CNNs) have shown impressive results for certain combinations of tasks, such as single-image depth estimation (SIDE) and semantic segmentation. This is achieved by pushing the network towards learning a robust representation that generalizes well to different atomic tasks. We extend this concept by adding auxiliary tasks, which are of minor relevance for the application, to the set of learned tasks. As a kind of additional regularization, they are expected to boost the performance of the ultimately desired main tasks. To study the proposed approach, we picked vision-based road scene understanding (RSU) as an exemplary application. Since multi-task learning requires specialized datasets, particularly when using extensive sets of tasks, we provide a multi-modal dataset for multi-task RSU, called synMT. More than 2.5 $\cdot$ 10^5 synthetic images, annotated with 21 different labels, were acquired from the video game Grand Theft Auto V (GTA V). Our proposed deep multi-task CNN architecture was trained on various combination of tasks using synMT. The experiments confirmed that auxiliary tasks can indeed boost network performance, both in terms of final results and training time.

Motivation & Objective

• Motivate multi-task learning to solve multiple vision tasks from a single image.
• Introduce auxiliary tasks that are easy to learn and provide regularization to improve main task performance.
• Develop a synthetic multi-task RSU dataset (synMT) with diverse labels for experimentation.
• Propose a multi-task loss with learnable task weights and a regularization term to stabilize training.

Proposed method

• Extend the multi-task loss with task weights c_tau and a regularizer R(c_tau) to prevent trivial solutions (L_T = sum_tau (1/(2 c_tau^2)) L_tau + log(1 + c_tau^2)).
• Adopt an encoder-decoder CNN architecture (based on DeepLab/ResNet) with task-specific decoders for SIDE and semantic segmentation plus auxiliary branches for time of day and weather.
• Use non-linear, scaled depth loss r(d) = 1 - log(d)/log(1000) to emphasize near-field depth accuracy.
• Model time as a cyclic regression loss (minimizing cyclic time distance) and weather as a classification loss; train with Adam and a small batch size on synMT.

Experimental results

Research questions

• RQ1 Can auxiliary tasks improve main task performance in multi-task CNNs for road scene understanding?
• RQ2 What auxiliary tasks are effective regularizers and how do they affect training convergence?
• RQ3 Does a synthetic multi-task RSU dataset provide insights transferable to real-world data?
• RQ4 What is the impact of including auxiliary tasks on depth, segmentation, time, and weather predictions?
• RQ5 How should task weights c_tau be managed during training to avoid trivial solutions?

Key findings

• Auxiliary tasks can boost main task performance and speed up convergence compared to standard multi-task setups.
• Depth estimation benefited most when including a weather auxiliary task (TIME and WEATHER), with notable improvements over some baselines.
• Semantic segmentation often did not improve with multi-task setups, indicating auxiliary task benefits are task-dependent.
• Learning with auxiliary tasks accelerates optimization, showing faster convergence across intermediate training states.
• A four-task setup (main tasks plus both auxiliary tasks) can learn a representation generalizing to four tasks, though careful selection of auxiliary tasks is important.