[Paper Review] Deep Multi-Task Learning via Generalized Tensor Trace Norm

Yi Zhang, Yu Zhang+1 more|arXiv (Cornell University)|Feb 12, 2020

Tensor decomposition and applications29 references4 citations

TL;DR

This paper proposes a Generalized Tensor Trace Norm (GTTN) for deep multi-task learning that captures all possible low-rank structures in a parameter tensor by combining matrix trace norms of all tensor flattenings. It learns optimal combination weights for these flattenings, enabling automatic importance estimation and improved performance over existing tensor trace norms on real-world datasets.

ABSTRACT

The trace norm is widely used in multi-task learning as it can discover low-rank structures among tasks in terms of model parameters. Nowadays, with the emerging of big datasets and the popularity of deep learning techniques, tensor trace norms have been used for deep multi-task models. However, existing tensor trace norms cannot discover all the low-rank structures and they require users to manually determine the importance of their components. To solve those two issues together, in this paper, we propose a Generalized Tensor Trace Norm (GTTN). The GTTN is defined as a convex combination of matrix trace norms of all possible tensor flattenings and hence it can discover all the possible low-rank structures. In the induced objective function, we will learn combination coefficients in the GTTN to automatically determine the importance. Experiments on real-world datasets demonstrate the effectiveness of the proposed GTTN.

Motivation & Objective

• To address the limitation of existing tensor trace norms in missing some low-rank structures due to using only a subset of possible tensor flattenings.
• To overcome the suboptimal performance caused by assuming equal importance for all tensor flattenings in existing methods.
• To develop a unified regularization framework that automatically learns the importance of different tensor flattenings through data-driven combination coefficients.
• To improve model interpretability by revealing which tensor axes or flattenings contribute most to the learned low-rank structure.
• To provide theoretical analysis of GTTN, including its dual norm and generalization bound, to support its practical use.

Proposed method

• Define GTTN as a convex combination of matrix trace norms over all possible tensor flattenings of a p-way parameter tensor.
• Treat the combination coefficients in GTTN as learnable parameters optimized via an objective function that balances regularization and task-specific loss.
• Use a tensor flattening strategy that considers all 2^p - 2 possible ways to reshape a p-way tensor into a matrix, focusing on those with distinct trace norms (2^{p-1} - 1 unique ones).
• Formulate the optimization problem with a regularized objective that includes GTTN and task-specific losses, enabling end-to-end training in deep learning models.
• Analyze the dual norm of GTTN and derive a generalization bound to support theoretical guarantees.
• Implement and evaluate the method on multiple real-world datasets, including ImageCLEF, Office-31, Office-Home, and Office-Caltech10, using features from different layers (pool5, fc7).

Experimental results

Research questions

• RQ1 Can a generalized tensor trace norm that includes all possible tensor flattenings capture more comprehensive low-rank structures than existing tensor trace norms?
• RQ2 Does learning the combination coefficients of tensor flattenings lead to better performance than assuming equal importance for all components?
• RQ3 Can the learned combination coefficients provide interpretable insights into which tensor axes or structural patterns are most relevant for the multi-task learning problem?
• RQ4 How does the computational complexity of GTTN scale with tensor order, and is it feasible for practical deep learning applications?
• RQ5 What is the generalization performance of GTTN compared to state-of-the-art multi-task learning methods on real-world benchmarks?

Key findings

• The GTTN outperforms state-of-the-art multi-task learning baselines on ImageCLEF, Office-31, Office-Home, and Office-Caltech10 datasets, demonstrating consistent performance gains.
• The learned combination coefficients (α) in GTTN vary across datasets and training proportions, indicating data-dependent importance of different tensor flattenings.
• On the ImageCLEF dataset with fc7 features and λ=0.25, the α values were 0.3718, 0.2154, and 0.4128 for θ=0.5, 0.6, and 0.7, respectively, showing dynamic adaptation to training data.
• For Office-31 with fc7 features and λ=0.65, the α values shifted to 0.3279, 0.2399, and 0.4322 under θ=0.7, reflecting changing structural importance with data availability.
• The GTTN achieved the best performance across all tested datasets and feature layers, confirming its ability to capture diverse low-rank structures.
• Theoretical analysis confirms that GTTN has a well-defined dual norm and a generalization bound, supporting its use in statistical learning.