[Paper Review] Meta-Learning: A Survey

Joaquin Vanschoren|TU/e Research Portal|Oct 8, 2018

Machine Learning and Data Classification8 references197 citations

TL;DR

This survey consolidates meta-learning approaches that leverage prior experience across tasks to accelerate learning on new tasks, covering evaluations, task characterization, and parameter transfer.

ABSTRACT

Meta-learning, or learning to learn, is the science of systematically observing how different machine learning approaches perform on a wide range of learning tasks, and then learning from this experience, or meta-data, to learn new tasks much faster than otherwise possible. Not only does this dramatically speed up and improve the design of machine learning pipelines or neural architectures, it also allows us to replace hand-engineered algorithms with novel approaches learned in a data-driven way. In this chapter, we provide an overview of the state of the art in this fascinating and continuously evolving field.

Motivation & Objective

• Present an organized overview of meta-learning techniques and how prior experience across tasks is used to guide learning on new tasks.
• Categorize meta-learning methods by the type of meta-data they leverage, from model evaluations to task properties and parameter transfer.
• Discuss practical aspects such as task similarity, configuration space design, and learning curves in meta-learning.
• Highlight connections with AutoML, few-shot learning, and related paradigms like multi-task and ensemble learning.
• Identify challenges and potential directions in leveraging meta-data for faster, data-driven model design.

Proposed method

• Define meta-data as algorithm configurations, evaluations, model parameters, and task meta-features.
• Describe task-independent recommendations and how to build rankings or portfolios of configurations.
• Explain configuration space design via informative defaults and hyperparameter importance.
• Present approaches for configuration transfer using relative landmarks, surrogate models, and warm-started multi-task learning.
• Survey learning curves and how across-task information can speed up early stopping and progress prediction.
• Outline meta-models that map task meta-features to promising configurations or performance predictions.
• Discuss learning meta-features and joint representations, including Siamese networks for task similarity.
• Explore various strategies for warm-starting optimization with similar tasks and meta-data.

Experimental results

Research questions

• RQ1 How can meta-data from prior tasks be leveraged to recommend or warm-start configurations for a new task?
• RQ2 What are effective ways to measure task similarity using evaluations, learning curves, or meta-features?
• RQ3 How can surrogate models, relative landmarks, and multi-task learning facilitate transfer across related tasks?
• RQ4 What role do learning curves and default hyperparameters play in meta-learning for faster optimization?
• RQ5 Which meta-models best predict performance or rank configurations across heterogeneous tasks?

Key findings

• Meta-learning can improve search efficiency by leveraging prior evaluations, meta-features, and learning curves across tasks.
• Task similarity can be quantified via relative landmarks, surrogate predictions, or learned meta-representations to guide optimization.
• Multiple strategies exist for configuring search space design and identifying effective defaults to speed up AutoML systems.
• Warm-starting Bayesian optimization and other optimizers with information from similar tasks accelerates convergence.
• Meta-models that predict performance or rank configurations enable faster pruning and smarter configuration selection across tasks.