[Paper Review] Excess risk bounds for multitask learning with trace norm regularization

Andreas Maurer, Massimiliano Pontil|arXiv (Cornell University)|Dec 6, 2012

Sparse and Compressive Sensing TechniquesEngineering18 references18 citations

TL;DR

This paper establishes excess risk bounds for multitask learning using trace norm regularization, demonstrating that generalization error depends on the number of tasks, samples per task, and data distribution—without dependence on input dimension. The key contribution is a theoretical analysis showing improved generalization via structured regularization in high- or infinite-dimensional spaces, including bounds on random positive semidefinite matrix sums with subexponential moments.

ABSTRACT

Trace norm regularization is a popular method of multitask learning. We give excess risk bounds with explicit dependence on the number of tasks, the number of examples per task and properties of the data distribution. The bounds are independent of the dimension of the input space, which may be infinite as in the case of reproducing kernel Hilbert spaces. A byproduct of the proof are bounds on the expected norm of sums of random positive semidefinite matrices with subexponential moments.

Motivation & Objective

• To provide theoretical excess risk bounds for multitask learning with trace norm regularization.
• To analyze how generalization error depends on the number of tasks, samples per task, and data distribution.
• To derive bounds independent of input space dimension, including infinite-dimensional cases like reproducing kernel Hilbert spaces.
• To establish novel concentration inequalities for sums of random positive semidefinite matrices with subexponential moments as a byproduct.

Proposed method

• The authors model multitask learning as a linear map from a Hilbert space to R^T, with predictors defined by weight vectors per task.
• They employ empirical risk minimization over a constraint set defined by the trace norm: ||W||_1 ≤ B√T, enforcing low-rank structure across tasks.
• A key technique involves Rademacher complexity analysis to bound the expected risk deviation, using symmetrization and matrix concentration inequalities.
• The proof leverages moment generating functions and matrix Chernoff-type bounds for sums of independent rank-one positive semidefinite operators.
• The analysis accounts for non-i.i.d. task sampling by introducing a harmonic mean of sample sizes, denoted as n̄.
• Theoretical bounds are derived using Theorem 7 and Theorem 8 for matrix concentration, leading to explicit risk bounds with logarithmic dependence on T and n̄.

Experimental results

Research questions

• RQ1 How does the excess risk of trace norm regularized multitask learning scale with the number of tasks and samples per task?
• RQ2 Can generalization bounds be derived that are independent of the input space dimension, even in infinite-dimensional Hilbert spaces?
• RQ3 What is the role of the trace norm in inducing structural dependence across tasks to improve generalization?
• RQ4 How do sums of random positive semidefinite matrices with subexponential moments concentrate, and what implications does this have for learning theory?

Key findings

• The excess risk bound scales as O(√(||C||_∞ / n̄) + √(log(nT)/n̄T)), where ||C||_∞ is the spectral norm of the covariance operator and n̄ is the harmonic mean of task sample sizes.
• The bound is dimension-free, holding even when the input space is infinite-dimensional, such as in reproducing kernel Hilbert spaces.
• A novel concentration inequality for sums of random positive semidefinite matrices is derived, showing that the expected operator norm concentrates around its mean with sub-Gaussian-like behavior.
• The bound on the Rademacher complexity of the trace norm constraint set is shown to scale as O(√(||C||_∞ / n̄) + √(log(nT)/n̄T)), which drives the final risk bound.
• The analysis confirms that trace norm regularization enables better generalization than independent learning when tasks are related, especially with small individual sample sizes.
• The derived bounds are tight in the sense that they reflect explicit dependencies on the number of tasks T, the average sample size n̄, and the data distribution through the covariance operator's spectral norm.