[Paper Review] Provable Meta-Learning of Linear Representations

Nilesh Tripuraneni, Chi Jin+1 more|arXiv (Cornell University)|Feb 26, 2020

Domain Adaptation and Few-Shot Learning35 references52 citations

TL;DR

This paper studies meta-learning of a shared linear feature representation across multiple tasks in a linear model, providing algorithmic methods with provable recovery guarantees and transfer to a new task, plus information-theoretic lower bounds.

ABSTRACT

Meta-learning, or learning-to-learn, seeks to design algorithms that can utilize previous experience to rapidly learn new skills or adapt to new environments. Representation learning -- a key tool for performing meta-learning -- learns a data representation that can transfer knowledge across multiple tasks, which is essential in regimes where data is scarce. Despite a recent surge of interest in the practice of meta-learning, the theoretical underpinnings of meta-learning algorithms are lacking, especially in the context of learning transferable representations. In this paper, we focus on the problem of multi-task linear regression -- in which multiple linear regression models share a common, low-dimensional linear representation. Here, we provide provably fast, sample-efficient algorithms to address the dual challenges of (1) learning a common set of features from multiple, related tasks, and (2) transferring this knowledge to new, unseen tasks. Both are central to the general problem of meta-learning. Finally, we complement these results by providing information-theoretic lower bounds on the sample complexity of learning these linear features.

Motivation & Objective

• Motivate learning-to-learn by identifying when a common low-dimensional feature space can be recovered from multiple related linear regression tasks.
• Develop algorithms that efficiently learn the shared representation and prove they are sample-efficient.
• Quantify how learning the representation improves sample efficiency on unseen tasks.
• Provide information-theoretic lower bounds to characterize the fundamental limits of feature recovery in multi-task settings.

Proposed method

• Model data as t tasks sharing an unknown r-dimensional linear representation B with orthonormal columns.
• Formulate a two-phase meta-learning framework: meta-train to learn B and meta-test to transfer to a new task with B fixed.
• Use a Burer-Monteiro factorization and a regularized empirical risk to show all local minima yield a good approximation to B.
• Provide a method-of-moments estimator that uses top-r eigenvectors of the matrix (1/n1) sum y_i^2 x_i x_i^T to recover B.
• Analyze a transfer-learning procedure where the learned B is plugged into a least-squares estimator for a new task.
• Establish an information-theoretic lower bound for feature recovery under the multi-task linear model.

Experimental results

Research questions

• RQ1 Can we efficiently learn a shared low-dimensional feature representation from multiple related linear regression tasks?
• RQ2 How well can the learned representation transfer to a new unseen task within the same linear feature space?
• RQ3 What are the sample complexities for learning the shared features and for transferring to a new task, and how do they compare to learning without transfer?
• RQ4 Under what conditions on task diversity and covariate design can first-order methods efficiently recover the representation?
• RQ5 What are the fundamental limits (lower bounds) for recovering the linear feature subspace in this multi-task setting?

Key findings

• Local minimizers of the regularized empirical risk recover the true linear representation up to a small statistical error.
• A method-of-moments estimator using top-r eigenvectors of the y^2 x x^T moment matrix recovers B with error O~(sqrt((tilde{kappa}/tilde{nu}) * dr / n1)).
• Transferring learned features to a new task yields an excess prediction error of O~(dr^2/n1 + r/n2) when using the feature estimate.
• Positive transfer occurs when n1/n2 is large relative to r and the representation’s complexity r is small compared to d.
• The paper provides an information-theoretic lower bound showing unavoidable error in feature recovery and highlighting the near-optimality of the proposed estimators under their assumptions.