[Paper Review] Cross-Task Consistency Learning Framework for Multi-Task Learning

Akihiro Nakano, Shi Chen+1 more|arXiv (Cornell University)|Nov 28, 2021

Domain Adaptation and Few-Shot Learning38 references4 citations

TL;DR

This paper proposes a cross-task consistency learning framework for two-task multi-task learning (MTL) that improves performance by enforcing consistency between predictions across tasks using a novel cross-task consistency loss and alignment loss. The method, implemented in XTasC-Net, transfers predictions between task-specific networks, achieving state-of-the-art results on Cityscapes and NYU datasets with fewer parameters, especially excelling in depth estimation and semantic segmentation with improved mIoU and reduced depth error.

ABSTRACT

Multi-task learning (MTL) is an active field in deep learning in which we train a model to jointly learn multiple tasks by exploiting relationships between the tasks. It has been shown that MTL helps the model share the learned features between tasks and enhance predictions compared to when learning each task independently. We propose a new learning framework for 2-task MTL problem that uses the predictions of one task as inputs to another network to predict the other task. We define two new loss terms inspired by cycle-consistency loss and contrastive learning, alignment loss and cross-task consistency loss. Both losses are designed to enforce the model to align the predictions of multiple tasks so that the model predicts consistently. We theoretically prove that both losses help the model learn more efficiently and that cross-task consistency loss is better in terms of alignment with the straight-forward predictions. Experimental results also show that our proposed model achieves significant performance on the benchmark Cityscapes and NYU dataset.

Motivation & Objective

• To address the challenge of learning effective task relationships in multi-task learning (MTL) without relying on stereo or specialized data collection.
• To improve generalization and efficiency in MTL by enforcing consistency between predictions of two related tasks, such as semantic segmentation and depth estimation.
• To develop a self-supervised MTL framework applicable to both stereo and monocular datasets, such as Cityscapes (stereo) and NYU (infrared).
• To theoretically and empirically validate a new loss formulation—cross-task consistency loss—that enhances prediction alignment and model robustness.
• To demonstrate parameter efficiency and superior performance compared to existing MTL methods on benchmark datasets.

Proposed method

• The framework introduces two novel loss terms: alignment loss and cross-task consistency loss, inspired by cycle-consistency and contrastive learning.
• A two-branch architecture is used: one for direct predictions (from input image), and another for task-transferred predictions (using predictions from one task as input to the other).
• The cross-task consistency loss computes the loss between the transferred prediction of one task and the ground-truth of the other, enforcing consistency across tasks.
• The alignment loss ensures that predictions from both tasks are aligned in the latent space, promoting shared representation learning.
• The model, named XTasC-Net, uses encoder-decoder networks for both tasks and leverages cross-prediction transfer to exploit task relationships.
• Theoretical analysis proves that cross-task consistency loss is superior to alignment loss in approximating direct predictions and is upper-bounded by a small value.

Experimental results

Research questions

• RQ1 Can a self-supervised MTL framework improve performance on both semantic segmentation and depth estimation tasks without requiring stereo data?
• RQ2 How does enforcing cross-task consistency via transferred predictions affect model generalization and performance?
• RQ3 Is the proposed cross-task consistency loss theoretically and empirically superior to alignment loss in capturing task relationships?
• RQ4 Can the framework achieve state-of-the-art performance while using fewer parameters than existing MTL models?
• RQ5 Does the method generalize across different data collection modalities, such as stereo and infrared cameras?

Key findings

• On the Cityscapes dataset, XTasC-Net achieved a mIoU of 30.31 and relative depth error of 0.2235, outperforming all baselines including AdaMT-Net in mIoU and relative error.
• On the NYU dataset, XTasC-Net achieved a mIoU of 30.31 and absolute depth error of 0.5954, showing a 7.87-point improvement in mIoU over the base ST-Net and a 0.0288 reduction in relative error.
• The model achieved state-of-the-art performance on both Cityscapes and NYU datasets, with significant improvements in segmentation mIoU and depth estimation accuracy.
• The cross-task consistency loss was shown to be theoretically superior to alignment loss in approximating direct predictions and is bounded by a small value.
• The method is parameter-efficient and applicable to both stereo and monocular datasets, unlike prior self-supervised MTL methods requiring stereo pairs.
• The qualitative results on Cityscapes show that the model produces more contrastive depth predictions, likely due to information transfer from segmentation to depth prediction.