[Paper Review] A Representation Learning Perspective on the Importance of Train-Validation Splitting in Meta-Learning

Nikunj Saunshi, Arushi Gupta+1 more|arXiv (Cornell University)|Jun 29, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes that train-validation splitting in meta-learning implicitly regularizes representations to be low-rank and expressive, significantly improving sample efficiency in few-shot learning. Through theoretical analysis and experiments, it shows that tr-val splitting enables O(k) sample complexity for k-dimensional task subspaces, outperforming tr-tr splitting which may require Ω(d) samples without explicit regularization.

ABSTRACT

An effective approach in meta-learning is to utilize multiple "train tasks" to learn a good initialization for model parameters that can help solve unseen "test tasks" with very few samples by fine-tuning from this initialization. Although successful in practice, theoretical understanding of such methods is limited. This work studies an important aspect of these methods: splitting the data from each task into train (support) and validation (query) sets during meta-training. Inspired by recent work (Raghu et al., 2020), we view such meta-learning methods through the lens of representation learning and argue that the train-validation split encourages the learned representation to be low-rank without compromising on expressivity, as opposed to the non-splitting variant that encourages high-rank representations. Since sample efficiency benefits from low-rankness, the splitting strategy will require very few samples to solve unseen test tasks. We present theoretical results that formalize this idea for linear representation learning on a subspace meta-learning instance, and experimentally verify this practical benefit of splitting in simulations and on standard meta-learning benchmarks.

Motivation & Objective

• To understand why train-validation splitting improves sample efficiency in meta-learning beyond empirical success.
• To analyze how tr-val splitting implicitly regularizes representations toward low-rank structures without explicit constraints.
• To contrast tr-val with tr-tr (train-train) splitting in terms of representation quality and sample complexity.
• To provide theoretical justification for the superiority of tr-val in linear representation learning on subspace tasks.
• To validate the hypothesis experimentally on simulations and benchmarks like Omniglot and MiniImageNet.

Proposed method

• Formalizes meta-learning as representation learning where the outer loop optimizes a representation function and the inner loop performs ridge regression on fixed representations.
• Analyzes the tr-val objective in a linear meta-learning setting with tasks defined on a k-dimensional subspace of d-dimensional input space.
• Proves that minimizing the tr-val objective guarantees learning a representation aligned with the true k-dimensional subspace, achieving O(k) sample complexity.
• Compares with the tr-tr variant, showing it cannot guarantee good representation learning without explicit regularization, leading to Ω(d) sample complexity.
• Employs empirical validation using RepLearn and iMAML on Omniglot and MiniImageNet with varying network capacity and regularization.
• Uses t-SNE and singular value analysis to visualize and compare representation clustering and rank between tr-val and tr-tr models.

Experimental results

Research questions

• RQ1 Does train-validation splitting in meta-learning implicitly regularize representations to be low-rank?
• RQ2 How does tr-val splitting compare to tr-tr splitting in terms of sample complexity for few-shot learning?
• RQ3 Can tr-val splitting achieve O(k) sample complexity without explicit low-rank constraints?
• RQ4 Why does tr-tr splitting fail to learn good representations without explicit regularization?
• RQ5 How robust is the tr-val approach to changes in model capacity and dataset choice?

Key findings

• The tr-val objective guarantees learning of an expressive, low-rank representation aligned with the true k-dimensional subspace, resulting in O(k) sample complexity for new tasks.
• In contrast, the tr-tr objective cannot guarantee good representation learning without explicit regularization, potentially leading to Ω(d) sample complexity.
• On Omniglot 5-way 1-shot, tr-val achieves 97.25% meta-test accuracy without regularization, while tr-tr drops to 67.78%.
• With explicit Frobenius norm regularization, tr-tr performance improves but still underperforms tr-val, which maintains 97.19–97.66% accuracy across λ values.
• On MiniImageNet, tr-val consistently outperforms tr-tr, with a 52.25% accuracy at capacity factor ℓ=8, while tr-tr lags at 25.28%.
• t-SNE visualizations show tr-val representations are more clustered than tr-tr, indicating better class separation and lower rank.