[Paper Review] Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learning

Baijiong Lin, Feiyang Ye+2 more|arXiv (Cornell University)|Nov 20, 2021

Domain Adaptation and Few-Shot Learning38 citations

TL;DR

Introduces Random Weighting (RW) methods for multi-task learning, showing that randomly sampled loss/gradient weights can converge and achieve competitive generalization versus state-of-the-art baselines.

ABSTRACT

Multi-Task Learning (MTL) has achieved success in various fields. However, how to balance different tasks to achieve good performance is a key problem. To achieve the task balancing, there are many works to carefully design dynamical loss/gradient weighting strategies but the basic random experiments are ignored to examine their effectiveness. In this paper, we propose the Random Weighting (RW) methods, including Random Loss Weighting (RLW) and Random Gradient Weighting (RGW), where an MTL model is trained with random loss/gradient weights sampled from a distribution. To show the effectiveness and necessity of RW methods, theoretically we analyze the convergence of RW and reveal that RW has a higher probability to escape local minima, resulting in better generalization ability. Empirically, we extensively evaluate the proposed RW methods to compare with twelve state-of-the-art methods on five image datasets and two multilingual problems from the XTREME benchmark to show RW methods can achieve comparable performance with state-of-the-art baselines. Therefore, we think that the RW methods are important baselines for MTL and should attract more attentions.

Motivation & Objective

• Motivate the need for simple baselines to test task balancing in MTL beyond Equal Weighting (EW).
• Propose Random Weighting (RW) methods—Random Loss Weighting (RLW) and Random Gradient Weighting (RGW)—as stochastic baselines for loss and gradient balancing.
• Provide theoretical analysis of convergence and generalization guarantees for RW methods.
• Empirically evaluate RW against twelve SOTA methods on CV and XTREME multilingual benchmarks to assess effectiveness and robustness.

Proposed method

• Define RW as sampling task weights from a distribution and normalizing to form a simplex, then updating parameters using the weighted aggregated loss or gradient.
• Propose RLW and RGW algorithms with softmax-based normalization of sampled weights from a standard normal distribution.
• Prove that RLW is a stochastic variant of EW and derive convergence guarantees under standard assumptions.
• Show that added randomness helps escape sharp local minima, improving generalization.
• Empirically compare RW against loss and gradient balancing baselines on five CV datasets and two XTREME multilingual tasks.
• Investigate combinations of RW with other balancing methods and architecture variants.

Experimental results

Research questions

• RQ1 Does random weighting in loss and gradient balancing converge and provide competitive performance compared to fixed equal weighting (EW)?
• RQ2 What are the convergence and generalization properties of RLW/RGW relative to EW under standard optimization assumptions?
• RQ3 How do RW methods perform across diverse MTL settings (CV and multilingual benchmarks) and architectures?
• RQ4 Can RW serve as a robust litmus test baseline for evaluating more sophisticated task-balancing strategies?

Key findings

• RW methods (RLW and RGW) consistently outperform EW across evaluated tasks.
• RLW achieves the highest reported improvement over EW on NYUv2 among the loss-balancing baselines.
• RGW and RLW attain competitive performance with state-of-the-art gradient/ loss-balancing methods across benchmarks.
• Theoretical results show RLW is a stochastic variant of EW with convergence guarantees and potential for better generalization due to escaping sharp local minima.
• RW methods demonstrate robustness to different weight distributions and can be efficiently integrated with various MTL architectures.
• RW methods yield notable improvements when combined with some gradient balancing methods, and can surpass certain baselines in multilingual tasks.