[Paper Review] New perspectives on k-support and cluster norms

Andrew M. McDonald, Massimiliano Pontil+1 more|arXiv (Cornell University)|Jan 1, 2016

Sparse and Compressive Sensing Techniques47 references33 citations

TL;DR

This paper introduces the box-norm, a new regularizer defined as a parameterized infimum of quadratics, showing that the k-support norm is a member of this family and the box-norm arises as a perturbation of it. The authors develop an improved algorithm for computing the proximity operator of the squared box-norm, extend the norms to matrices as the spectral k-support and spectral box-norms, and demonstrate that the spectral box-norm is equivalent to the cluster norm; numerical experiments show state-of-the-art performance in matrix completion and multitask learning with centered variants.

ABSTRACT

We study a regularizer which is defined as a parameterized in_mum of quadratics, and which we call the box-norm. We show that the k-support norm, a regularizer proposed by Argyriou et al. (2012) for sparse vector prediction problems, belongs to this family, and the box-norm can be generated as a perturbation of the former. We derive an improved algorithm to compute the proximity operator of the squared box-norm, and we provide a method to compute the norm. We extend the norms to matrices, introducing the spectral k-support norm and spectral box-norm. We note that the spectral box-norm is essentially equivalent to the cluster norm, a multitask learning regularizer introduced by Jacob et al. (2009a), and which in turn can be interpreted as a perturbation of the spectral k-support norm. Centering the norm is important for multitask learning and we also provide a method to use centered versions of the norms as regularizers. Numerical experiments indicate that the spectral k-support and box-norms and their centered variants provide state of the art performance in matrix completion and multitask learning problems respectively.

Motivation & Objective

• To develop a new family of regularizers based on the infimum of quadratics, termed the box-norm, for sparse vector and matrix learning.
• To establish theoretical connections between the k-support norm, box-norm, and cluster norm, showing that the box-norm is a perturbation of the k-support norm.
• To extend the norms to matrices, introducing the spectral k-support and spectral box-norms for multitask learning and matrix completion.
• To provide efficient algorithms for computing the proximity operator and the norm values of the box-norm and its spectral variants.
• To introduce and evaluate centered versions of the norms to improve performance in multitask learning settings.

Proposed method

• The box-norm is defined as the infimum over a parameterized family of quadratic functions, generalizing the k-support norm.
• An improved algorithm is derived for computing the proximity operator of the squared box-norm using convex optimization techniques.
• The k-support and box-norms are extended to matrices via singular values, resulting in the spectral k-support and spectral box-norms.
• The spectral box-norm is shown to be mathematically equivalent to the cluster norm, a known multitask learning regularizer.
• Centered variants of the norms are constructed by shifting the singular values to ensure zero mean across tasks, improving generalization in multitask learning.
• Numerical methods are provided to compute the norm values efficiently, enabling practical deployment in large-scale learning problems.

Experimental results

Research questions

• RQ1 How can the k-support norm be reinterpreted as part of a broader family of regularizers defined via infima of quadratics?
• RQ2 What is the relationship between the box-norm and the k-support norm, and how does the box-norm emerge as a perturbation of the latter?
• RQ3 Can the box-norm be naturally extended to matrices, and what properties do the resulting spectral norms possess?
• RQ4 Is the spectral box-norm equivalent to the cluster norm, and what implications does this equivalence have for multitask learning?
• RQ5 How do centered versions of the box-norm and spectral box-norm improve performance in multitask learning and matrix completion tasks?

Key findings

• The spectral box-norm is mathematically equivalent to the cluster norm, establishing a new theoretical link between two previously distinct regularizers.
• The proximity operator of the squared box-norm can be computed more efficiently using the proposed algorithm, improving computational scalability.
• The spectral k-support and spectral box-norms achieve state-of-the-art performance in matrix completion tasks, outperforming existing baselines.
• Centered variants of the spectral box-norm and k-support norm significantly improve performance in multitask learning, particularly in low-sample regimes.
• Numerical experiments confirm that the spectral box-norm and its centered variant provide strong empirical results across diverse multitask learning and matrix completion benchmarks.