[Paper Review] MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning

Yuan Gao, Haoping Bai+4 more|arXiv (Cornell University)|Mar 31, 2020

Domain Adaptation and Few-Shot Learning72 references9 citations

TL;DR

MTL-NAS proposes a task-agnostic neural architecture search framework for general-purpose multi-task learning by decoupling task-specific backbones from a hierarchical, layerwise feature fusion mechanism. It introduces a unified search space over cross-task connections and a single-shot gradient-based search with minimum entropy regularization, enabling direct deployment of searched architectures without retraining, achieving state-of-the-art performance across diverse task sets and backbones.

ABSTRACT

We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL). Existing NAS methods typically define different search spaces according to different tasks. In order to adapt to different task combinations (i.e., task sets), we disentangle the GP-MTL networks into single-task backbones (optionally encode the task priors), and a hierarchical and layerwise features sharing/fusing scheme across them. This enables us to design a novel and general task-agnostic search space, which inserts cross-task edges (i.e., feature fusion connections) into fixed single-task network backbones. Moreover, we also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures and the final evaluation architecture. This is realized with a minimum entropy regularization on the architecture weights during the search phase, which makes the architecture weights converge to near-discrete values and therefore achieves a single model. As a result, our searched model can be directly used for evaluation without (re-)training from scratch. We perform extensive experiments using different single-task backbones on various task sets, demonstrating the promising performance obtained by exploiting the hierarchical and layerwise features, as well as the desirable generalizability to different i) task sets and ii) single-task backbones. The code of our paper is available at https://github.com/bhpfelix/MTLNAS.

Motivation & Objective

• To address the limitation of existing NAS methods that are task-specific and incompatible with general-purpose multi-task learning (GP-MTL).
• To decouple task priors from feature sharing/fusion mechanisms in GP-MTL to enable a unified, task-agnostic search space.
• To close the performance gap between search and evaluation phases in single-shot NAS by minimizing uncertainty in architecture weights.
• To enable direct deployment of searched architectures without retraining, improving efficiency and practicality.

Proposed method

• Disentangle GP-MTL into fixed, task-specific single-task backbones and a general, hierarchical, layerwise feature fusion scheme across them.
• Define a task-agnostic search space by inserting searchable cross-task edges (feature fusion connections) between fixed backbone layers.
• Propose a single-shot gradient-based search algorithm with minimum entropy regularization on architecture weights to encourage convergence to near-discrete values.
• Apply deterministic or stochastic continuous relaxation and discretization components within a unified framework to generalize DARTS and SNAS.
• Use entropy minimization to reduce uncertainty in architecture weights, ensuring stable and high-performance discrete architectures after search.
• Enable direct evaluation of the searched architecture without retraining from scratch, significantly reducing inference cost.

Experimental results

Research questions

• RQ1 Can a unified, task-agnostic search space be designed for multi-task learning that supports arbitrary combinations of tasks?
• RQ2 How can the performance gap between the search phase and final evaluation be closed in single-shot NAS?
• RQ3 Can entropy minimization effectively regularize architecture weights to achieve near-discrete solutions without retraining?
• RQ4 Does the proposed method generalize across different single-task backbones and task sets in GP-MTL?
• RQ5 Can the searched architecture be directly deployed for evaluation without retraining, maintaining high performance?

Key findings

• The proposed method achieves state-of-the-art performance on multiple task sets, including surface normal prediction and semantic segmentation, with mIoU of 88.2% and PAcc of 91.2% on the Cityscapes dataset.
• Minimum entropy regularization effectively regularizes architecture weights, causing most α values to converge to 0 or 1, as shown in the histogram of converged α values.
• The method closes the performance gap between search and evaluation: the searched architecture achieves 37.7% mean IoU and 67.9% mIoU on the Cityscapes dataset without retraining.
• Different combinations of deterministic and stochastic components (relaxation and discretization) yield negligible performance differences under entropy regularization, indicating robustness and stability.
• The learned architectures are heterogeneous and asymmetric, suggesting they are non-trivial to discover manually, validating the method’s capability to find complex, effective structures.
• The ablation study confirms that minimum entropy regularization significantly improves performance across all configurations of continuous relaxation and discretization, outperforming both DARTS and SNAS.