[Paper Review] Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction

Zhiyong Yang, Qianqian Xu+2 more|arXiv (Cornell University)|Apr 29, 2020

Domain Adaptation and Few-Shot Learning75 references4 citations

TL;DR

This paper proposes Task-Feature Collaborative Learning (TFCL), a novel multi-task learning method that jointly models task and feature grouping via a block-diagonal structure regularizer to mitigate negative transfer. The method ensures global convergence and outperforms state-of-the-art approaches in personalized attribute prediction on real-world datasets, especially for hard tasks, by reducing performance variance and improving generalization.

ABSTRACT

As an effective learning paradigm against insufficient training samples, Multi-Task Learning (MTL) encourages knowledge sharing across multiple related tasks so as to improve the overall performance. In MTL, a major challenge springs from the phenomenon that sharing the knowledge with dissimilar and hard tasks, known as negative transfer, often results in a worsened performance. Though a substantial amount of studies have been carried out against the negative transfer, most of the existing methods only model the transfer relationship as task correlations, with the transfer across features and tasks left unconsidered. Different from the existing methods, our goal is to alleviate negative transfer collaboratively across features and tasks. To this end, we propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL). Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks and suppressing inter-group knowledge sharing. We then propose an optimization method for the model. Extensive theoretical analysis shows that our proposed method has the following benefits: (a) it enjoys the global convergence property and (b) it provides a block-diagonal structure recovery guarantee. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks. We further apply it to the personalized attribute prediction problem with fine-grained modeling of user behaviors. Finally, experimental results on both simulated dataset and real-world datasets demonstrate the effectiveness of our proposed method

Motivation & Objective

• To address negative transfer in multi-task learning by jointly modeling task and feature relationships.
• To develop a regularization framework that encourages collaborative grouping across tasks and features to suppress knowledge sharing between unrelated groups.
• To ensure global convergence and structural recovery guarantees in the optimization process.
• To extend the method to personalized attribute prediction with fine-grained user behavior modeling.
• To empirically validate the method on simulated and real-world datasets, demonstrating robustness on hard tasks.

Proposed method

• Proposes a base model with a heterogeneous block-diagonal structure regularizer to enforce collaborative grouping of tasks and features.
• Introduces an iterative optimization method that solves convex subproblems, with a novel closed-form solution for one subproblem.
• Establishes global convergence and block-diagonal structure recovery guarantees through theoretical analysis.
• Extends the model to allow overlapping features and differentiates hard tasks via hierarchical model decomposition.
• Uses a co-grouping regularizer that outperforms existing co-grouping terms, such as in CoCMTL.
• Applies the method to personalized attribute prediction by modeling user-specific behavior patterns with improved generalization.

Experimental results

Research questions

• RQ1 Can joint modeling of task and feature grouping effectively reduce negative transfer in multi-task learning?
• RQ2 Does the proposed block-diagonal regularizer ensure structural recovery and global convergence in optimization?
• RQ3 How does TFCL perform compared to state-of-the-art MTL methods on personalized attribute prediction tasks?
• RQ4 To what extent does TFCL improve performance on hard or outlier tasks, especially in the presence of data scarcity?
• RQ5 Can the method maintain robustness and generalization when applied to real-world datasets with complex user behavior patterns?

Key findings

• TFCL significantly outperforms all competitors on both the Shoes and Sun datasets, with average AUC improvements of 1.95, 2.95, 0.44, 2.37, and 1.29 for attributes CL, MO, OA, RU, and SO, respectively.
• The method achieves a more compact distribution of user-specific AUC scores, indicating reduced performance variance and better handling of hard tasks.
• The ablation study confirms that removing the grouping factor causes a more significant performance drop than replacing the loss function, highlighting the importance of structural grouping.
• The proposed regularizer outperforms the co-grouping regularizer used in CoCMTL across all attributes in both datasets.
• On the simulated dataset, TFCL successfully recovers the correct co-grouping structure, validating its structural learning capability.
• The optimization method demonstrates global convergence, and the solution exhibits a connection to optimal transport, offering new theoretical insight into negative transfer mitigation.