[Paper Review] Coresets via Bilevel Optimization for Continual Learning and Streaming

Zalán Borsos, Mojmír Mutný+1 more|arXiv (Cornell University)|Jun 6, 2020

Domain Adaptation and Few-Shot Learning61 references78 citations

TL;DR

The paper introduces a bilevel-optimization framework to construct small labeled data coresets, enabling effective neural network training for continual learning and streaming, with a greedy matching-pursuit approach and a proxy RKHS/NTK formulation to scale to deep models.

ABSTRACT

Coresets are small data summaries that are sufficient for model training. They can be maintained online, enabling efficient handling of large data streams under resource constraints. However, existing constructions are limited to simple models such as k-means and logistic regression. In this work, we propose a novel coreset construction via cardinality-constrained bilevel optimization. We show how our framework can efficiently generate coresets for deep neural networks, and demonstrate its empirical benefits in continual learning and in streaming settings.

Motivation & Objective

• Motivate the use of coresets to maintain accurate models under online data arrival and memory constraints in continual learning and streaming.
• Propose a general coreset construction framework based on cardinality-constrained bilevel optimization.
• Enable efficient coreset generation for deep neural networks via a proxy model and RKHS/NTK formulations.
• Demonstrate empirical gains of the proposed coresets in replay-memory-based continual learning and streaming scenarios.

Proposed method

• Formulate coreset selection as a cardinality-constrained bilevel optimization problem where the inner problem minimizes a regularized loss and the outer problem minimizes the loss on the inner solution.
• Solve the bilevel problem via a greedy forward (matching pursuit) approach that iteratively adds data points that most reduce the outer objective, using a local inner-solution and an implicit-gradient calculation.
• Use a proxy model in a reproducing kernel Hilbert space to avoid inverting large Hessians when training neural networks, enabling scalable outer-loop optimization.
• Employ representer-theorem-based reduction so the inner problem depends on at most m data points (the coreset size).
• Connect the method to influence functions and experimental design, providing intuition for the selection criterion and convergence in convex cases.

Experimental results

Research questions

• RQ1 Can a bilevel-optimization formulation yield effective, small coresets for training complex models like neural networks?
• RQ2 How can one efficiently select a small subset of data online to support continual learning and streaming without catastrophic forgetting?
• RQ3 What are the theoretical and empirical properties (convergence, performance) of greedy matching-pursuit style coreset selection in this setting?
• RQ4 How does a proxy RKHS/NTK formulation affect the practicality and performance of coreset construction for deep networks?

Key findings

• The proposed framework yields representative coresets that improve performance in replay-memory-based continual learning and streaming compared to several baselines.
• A greedy matching-pursuit method with a local inner-solution and implicit gradient provides a practical solution to the cardinality-constrained bilevel problem.
• Proxy formulations using RKHS/NTK allow scalable coreset generation for neural networks, significantly speeding up outer-loop computations (e.g., using NTK proxies).
• The method demonstrates competitive or superior performance across continual learning benchmarks (PermMNIST, SplitMNIST, SplitCIFAR-10) and streaming scenarios, including imbalanced streams.
• The approach establishes connections to Bayesian experimental design and empirical influence functions, offering theoretical grounding for the coreset selection strategy.