[Paper Review] Self-Supervised Generalisation with Meta Auxiliary Learning

Shikun Liu, Andrew J. Davison+1 more|arXiv (Cornell University)|Jan 25, 2019

Domain Adaptation and Few-Shot Learning32 references66 citations

TL;DR

MAXL automatically learns auxiliary labels to boost a primary task’s generalisation without needing extra data, via a meta-trained label-generator and a multi-tasklearner.

ABSTRACT

Learning with auxiliary tasks can improve the ability of a primary task to generalise. However, this comes at the cost of manually labelling auxiliary data. We propose a new method which automatically learns appropriate labels for an auxiliary task, such that any supervised learning task can be improved without requiring access to any further data. The approach is to train two neural networks: a label-generation network to predict the auxiliary labels, and a multi-task network to train the primary task alongside the auxiliary task. The loss for the label-generation network incorporates the loss of the multi-task network, and so this interaction between the two networks can be seen as a form of meta learning with a double gradient. We show that our proposed method, Meta AuXiliary Learning (MAXL), outperforms single-task learning on 7 image datasets, without requiring any additional data. We also show that MAXL outperforms several other baselines for generating auxiliary labels, and is even competitive when compared with human-defined auxiliary labels. The self-supervised nature of our method leads to a promising new direction towards automated generalisation. Source code can be found at https://github.com/lorenmt/maxl.

Motivation & Objective

• Motivate auxiliary learning to enhance generalisation without manual auxiliary labels.
• Propose a self-supervised framework to automatically generate auxiliary labels.
• Demonstrate that MAXL improves primary-task accuracy across multiple image datasets.

Proposed method

• Two-network MAXL architecture: a multi-task network for primary and auxiliary tasks, and a label-generation network for auxiliary labels.
• Hierarchical auxiliary label structure per primary class with masked SoftMax (Mask SoftMax) to enforce class-wise auxiliary mapping.
• Meta-learning gradient flow where the label-generator is trained via the primary-task performance (second-derivative Hessian trick).
• Entropy regularisation on auxiliary label distribution to avoid collapsing auxiliary labels.
• Focal loss used for both primary and auxiliary tasks to focus on hard examples.
• Training alternates between updating the multi-task network with generated auxiliary labels and updating the label-generator via the primary-task performance.

Experimental results

Research questions

• RQ1 Can a self-generated auxiliary label space improve primary-task generalisation without any extra data?
• RQ2 How effective are automatically generated auxiliary labels compared to random, unsupervised clustering, or human-defined auxiliaries?
• RQ3 Does incorporating a hierarchical auxiliary label structure help or hinder performance across datasets?
• RQ4 What are the dynamics of gradient similarity between auxiliary and primary losses when using MAXL?
• RQ5 Can MAXL approach or match performance of human-defined auxiliary labels without supervision?

Key findings

• MAXL outperforms single-task learning on seven image datasets using the same labelled data.
• MAXL surpasses baseline auxiliary-label generation methods (Random, K-Means) and is competitive with Human-defined auxiliary labels on CIFAR-100.
• Across CIFAR-100 with hierarchies, MAXL maintains high auxiliary-gradient usefulness (positive cosine similarity) throughout training, unlike fixed-label baselines.
• MAXL yields improved primary-task separability in t-SNE visualisations compared to Single Task, and approaches the separation achieved with human auxiliary labels.
• The method remains robust across a range of hierarchies (psi values) without needing dataset-specific tuning.