[Paper Review] Efficient and Scalable Multi-task Regression on Massive Number of Tasks

Xiao He, Francesco Alesiani+1 more|arXiv (Cornell University)|Nov 14, 2018

Machine Learning and ELM4 citations

TL;DR

This paper proposes CCMTL, a novel multi-task regression method that integrates convex clustering on a k-nearest neighbor graph of prediction models to enable efficient, scalable learning across massive numbers of tasks. It achieves linear scalability and global convergence, outperforming state-of-the-art methods with 30-second training on 23,812 tasks—dramatically faster than hours or days for prior approaches.

ABSTRACT

Many real-world large-scale regression problems can be formulated as Multi-task Learning (MTL) problems with a massive number of tasks, as in retail and transportation domains. However, existing MTL methods still fail to offer both the generalization performance and the scalability for such problems. Scaling up MTL methods to problems with a tremendous number of tasks is a big challenge. Here, we propose a novel algorithm, named Convex Clustering Multi-Task regression Learning (CCMTL), which integrates with convex clustering on the k-nearest neighbor graph of the prediction models. Further, CCMTL efficiently solves the underlying convex problem with a newly proposed optimization method. CCMTL is accurate, efficient to train, and empirically scales linearly in the number of tasks. On both synthetic and real-world datasets, the proposed CCMTL outperforms seven state-of-the-art (SoA) multi-task learning methods in terms of prediction accuracy as well as computational efficiency. On a real-world retail dataset with 23,812 tasks, CCMTL requires only around 30 seconds to train on a single thread, while the SoA methods need up to hours or even days.

Motivation & Objective

• Address the challenge of scaling multi-task learning to problems with hundreds of thousands of tasks, common in retail and transportation domains.
• Overcome the limitations of existing multi-task learning methods that scale super-linearly or assume all tasks are related.
• Develop a method that captures task relationships through clustering while maintaining computational efficiency and convergence guarantees.
• Enable practical deployment of multi-task regression in real-world large-scale applications where training time and accuracy are critical.
• Provide a scalable alternative to existing task clustering methods that are computationally prohibitive for massive task sets.

Proposed method

• Formulates multi-task regression as a regularized optimization problem that combines prediction error minimization with a convex clustering penalty on task weight vectors.
• Constructs a k-nearest neighbor (k-NN) graph over the prediction models to define task similarities and guide clustering.
• Uses the ℓ₂ norm on differences between task weights as the clustering regularizer, ensuring convexity and stable clustering behavior.
• Proposes a new optimization algorithm that converges to the global optimum, enabling reliable and efficient training.
• Integrates the k-NN graph structure into the convex clustering objective to adaptively group similar tasks without requiring pre-specified cluster counts.
• Employs a dual ascent method with convergence guarantees, ensuring robustness and scalability across large task sets.

Experimental results

Research questions

• RQ1 Can convex clustering on a k-NN graph of prediction models effectively group related tasks in large-scale multi-task regression?
• RQ2 Does the proposed CCMTL method achieve linear scalability in the number of tasks, unlike existing SOTA methods that scale super-linearly?
• RQ3 How does CCMTL compare in prediction accuracy and training efficiency to state-of-the-art multi-task learning methods on synthetic and real-world datasets?
• RQ4 Can CCMTL maintain high performance and low training time on real-world datasets with tens of thousands of tasks, such as retail sales forecasting?
• RQ5 Is the proposed optimization method guaranteed to converge to the global optimum, and how does it compare in speed to existing solvers?

Key findings

• CCMTL achieves state-of-the-art prediction accuracy on both synthetic and real-world datasets, outperforming seven SOTA multi-task learning methods.
• On a real-world retail dataset with 23,812 tasks, CCMTL trains in approximately 30 seconds on a single thread, while SOTA methods require hours or even days.
• CCMTL demonstrates linear scalability in training time with respect to the number of tasks, confirmed by experiments on datasets up to 160,000 tasks.
• Among all compared methods, only CCMTL and RMTL show linear runtime scaling; all others exhibit super-linear growth, with FuseMTL, MTFactor, and TriFactor failing to complete 40k-task runs within 24 hours.
• Regression analysis in the supplementary material confirms that CCMTL’s runtime scales linearly, while other methods scale quadratically.
• CCMTL’s use of ℓ₂ norm in the clustering regularizer leads to better performance than methods using ℓ₁ or squared ℓ₂ norms, such as FuseMTL and SRMTL.