[Paper Review] Efficient Representations for Life-Long Learning and Autoencoding

Maria-Florina Balcan, Avrim Blum+1 more|arXiv (Cornell University)|Nov 6, 2014

Machine Learning and AlgorithmsComputer Science33 references43 citations

TL;DR

This paper proposes efficient algorithms for lifelong machine learning that learn shared representations across multiple tasks, enabling faster and data-efficient learning. It introduces methods for discovering low-dimensional subspaces and Boolean combinations of features—using anchor assumptions—to construct sparse, near-optimal autoencoders, with provable sample complexity improvements over learning tasks in isolation.

ABSTRACT

It has been a long-standing goal in machine learning, as well as in AI more generally, to develop life-long learning systems that learn many different tasks over time, and reuse insights from tasks learned, "learning to learn" as they do so. In this work we pose and provide efficient algorithms for several natural theoretical formulations of this goal. Specifically, we consider the problem of learning many different target functions over time, that share certain commonalities that are initially unknown to the learning algorithm. Our aim is to learn new internal representations as the algorithm learns new target functions, that capture this commonality and allow subsequent learning tasks to be solved more efficiently and from less data. We develop efficient algorithms for two very different kinds of commonalities that target functions might share: one based on learning common low-dimensional and unions of low-dimensional subspaces and one based on learning nonlinear Boolean combinations of features. Our algorithms for learning Boolean feature combinations additionally have a dual interpretation, and can be viewed as giving an efficient procedure for constructing near-optimal sparse Boolean autoencoders under a natural "anchor-set" assumption.

Motivation & Objective

• To develop efficient, streaming algorithms for lifelong learning that reuse insights across tasks.
• To identify and learn shared structural commonalities across multiple target functions.
• To reduce sample complexity in online learning by constructing compact, reusable internal representations.
• To provide theoretical guarantees for learning in both linear and nonlinear settings under natural distributional assumptions.
• To enable sparse, overcomplete representations for Boolean feature combinations via anchor-set assumptions.

Proposed method

• Uses a streaming online learning framework where only hypotheses and current representations are retained.
• Applies trace norm regularization and joint optimization for learning shared low-dimensional subspaces in linear settings.
• Employs an anchor-variable assumption to identify minimal product-based metafeatures for Boolean functions.
• Introduces a bicriteria approximation algorithm under the anchor-set assumption for overcomplete sparse representations.
• Uses Schapire and Sellie's exact learning algorithm for polynomials when representation fails.
• Periodically re-compacts learned representations to minimize the number of conjunctive metafeatures.

Experimental results

Research questions

• RQ1 Can we learn shared low-dimensional subspaces across multiple classification tasks in an online, streaming setting with provable sample efficiency?
• RQ2 How can we efficiently discover nonlinear Boolean combinations of features that enable sparse, compact representations for multiple target functions?
• RQ3 What assumptions are necessary to achieve near-optimal performance in learning sparse Boolean autoencoders?
• RQ4 Can we maintain low sample complexity in lifelong learning without storing past training data?
• RQ5 How can we efficiently compactify learned polynomial functions into minimal sets of overlapping conjunctive metafeatures?

Key findings

• The algorithm achieves sample complexity of O(B²k) per task when learning from a compact representation, significantly better than Ω(nm/ϵ) for independent learning.
• For linear subspaces, the method provides sample complexity improvements under log-concave distributions, with error bounds tied to angular separation.
• Under the anchor-variable assumption, the algorithm finds the optimal number of product-based metafeatures for Boolean autoencoding.
• Under the anchor-set assumption, the algorithm provides a bicriteria approximation, approximating both sparsity and number of metafeatures within logarithmic factors.
• The number of tasks requiring full retraining is bounded by n² + k, ensuring scalability even as new tasks arrive.
• The method enables efficient lifelong learning of polynomials by periodically compactifying representations into minimal conjunctive metafeatures.