[Paper Review] Adapting Auxiliary Losses Using Gradient Similarity

Yunshu Du, Wojciech Marian Czarnecki+4 more|arXiv (Cornell University)|Dec 5, 2018

Adversarial Robustness in Machine Learning51 references95 citations

TL;DR

The paper proposes a gradient cosine similarity heuristic to decide when to use an auxiliary loss to help the main task, ensuring convergence and blocking negative transfer across supervised and reinforcement learning settings.

ABSTRACT

One approach to deal with the statistical inefficiency of neural networks is to rely on auxiliary losses that help to build useful representations. However, it is not always trivial to know if an auxiliary task will be helpful for the main task and when it could start hurting. We propose to use the cosine similarity between gradients of tasks as an adaptive weight to detect when an auxiliary loss is helpful to the main loss. We show that our approach is guaranteed to converge to critical points of the main task and demonstrate the practical usefulness of the proposed algorithm in a few domains: multi-task supervised learning on subsets of ImageNet, reinforcement learning on gridworld, and reinforcement learning on Atari games.

Motivation & Objective

• Motivate the use of auxiliary losses to improve data efficiency while avoiding negative transfer to the main task.
• Introduce a lightweight, gradient-based heuristic to dynamically weight auxiliary losses during training.
• Theoretically guarantee convergence of the main task under the proposed update rule.
• Empirically validate the approach across supervised learning and reinforcement learning domains.
• Discuss implications and limitations of using gradient similarity as a measure of task relatedness.

Proposed method

• Define a shared-parameter architecture with a main task network and an auxiliary task network.
• Propose a gradient-based update rule that adds the auxiliary gradient to the main gradient only when their cosine similarity is non-negative (or use a binary variant).
• Prove convergence to a local minimum of the main loss under small learning rates (Propositions 1 and 2).
• Show that the method blocks negative transfer and can enable positive transfer in practice (Algorithm 1 and Algorithm 2 variants).
• Apply the method to binary classification on ImageNet and rotated MNIST, RL gridworlds, and Atari to demonstrate adaptability across domains.
• Discuss that the resulting vector field need not be conservative, highlighting practical training implications.

Experimental results

Research questions

• RQ1 Can gradient cosine similarity reliably indicate when an auxiliary task will help or hurt the main task during training?
• RQ2 Does adapting the auxiliary weight based on gradient similarity guarantee convergence to the main task's critical points?
• RQ3 How does the proposed heuristic perform in supervised learning and reinforcement learning settings with changing task relatedness?
• RQ4 What are the practical implications and potential limitations of using gradient similarity as a transfer-measure across domains?

Key findings

• Gradient cosine similarity distinguishes related versus unrelated auxiliary tasks, being higher for near pairs and lower for far pairs in ImageNet experiments.
• The proposed cos-weighted method can speed initial learning and then match single-task performance when the auxiliary becomes unhelpful (near class pairs).
• The method blocks negative transfer by turning off the auxiliary when cosine similarity is negative and can recover main-task performance (far class pairs).
• In rotated MNIST, the method improves or matches single-task performance and mitigates negative interference from the auxiliary at higher rotations.
• In RL gridworlds and Atari, cosine-weighted distillation adapts auxiliary signals to improve learning speed and final performance, while avoiding forgetting main-task skills.
• Propositions show convergence guarantees for the main task under the proposed update rules, even with non-conservative auxiliary vector fields.