[Paper Review] Sequoia: A Software Framework to Unify Continual Learning Research

Fabrice Normandin, Florian Golemo+11 more|arXiv (Cornell University)|Aug 2, 2021

Respiratory viral infections research54 references4 citations

TL;DR

Sequoia is a unified software framework that organizes continual learning (CL) research through a hierarchical taxonomy of settings based on shared assumptions, enabling method inheritance and reuse across Continual Supervised Learning (CSL) and Continual Reinforcement Learning (CRL). The framework reduces duplication, enhances reproducibility, and accelerates research by standardizing evaluation and enabling cross-paradigm method transfer.

ABSTRACT

The field of Continual Learning (CL) seeks to develop algorithms that accumulate knowledge and skills over time through interaction with non-stationary environments. In practice, a plethora of evaluation procedures (settings) and algorithmic solutions (methods) exist, each with their own potentially disjoint set of assumptions. This variety makes measuring progress in CL difficult. We propose a taxonomy of settings, where each setting is described as a set of assumptions. A tree-shaped hierarchy emerges from this view, where more general settings become the parents of those with more restrictive assumptions. This makes it possible to use inheritance to share and reuse research, as developing a method for a given setting also makes it directly applicable onto any of its children. We instantiate this idea as a publicly available software framework called Sequoia, which features a wide variety of settings from both the Continual Supervised Learning (CSL) and Continual Reinforcement Learning (CRL) domains. Sequoia also includes a growing suite of methods which are easy to extend and customize, in addition to more specialized methods from external libraries. We hope that this new paradigm and its first implementation can help unify and accelerate research in CL. You can help us grow the tree by visiting www.github.com/lebrice/Sequoia.

Motivation & Objective

• Address the fragmentation in continual learning (CL) research caused by a proliferation of disjointed evaluation settings and inconsistent assumptions.
• Reduce duplication of effort between Continual Supervised Learning (CSL) and Continual Reinforcement Learning (CRL) by unifying their methodological frameworks.
• Improve reproducibility and standardization in CL by providing a shared software infrastructure with clear configuration and evaluation protocols.
• Enable researchers to develop and evaluate CL methods in a modular, extensible, and interoperable environment across diverse settings.
• Accelerate progress in CL by allowing methods developed for a general setting to be directly inherited and applied to more specific, child settings.

Proposed method

• Define each CL setting as a set of assumptions (e.g., task ID availability, task boundary knowledge), forming the basis for a tree-shaped hierarchy.
• Construct a hierarchical taxonomy where more general settings (parents) inherit assumptions, and more restrictive ones (children) add constraints, enabling method reuse via inheritance.
• Implement the framework as Sequoia, a publicly available open-source software library supporting both CSL and CRL with standardized, composable environments and datasets.
• Integrate a growing suite of CL methods, including those from external libraries, with extensible interfaces for custom method development and evaluation.
• Support evaluation across diverse settings using standardized metrics, including final performance, online performance, and normalized runtime.
• Enable cross-paradigm compatibility by allowing methods designed for CSL to be applied in CRL settings and vice versa, particularly for techniques like experience replay.

Experimental results

Research questions

• RQ1 How can the diversity of continual learning settings be systematically organized to reduce redundancy and improve method reusability?
• RQ2 To what extent can a unified software framework reduce duplication of effort between Continual Supervised Learning and Continual Reinforcement Learning?
• RQ3 Can a hierarchical taxonomy of assumptions enable direct inheritance of methods across settings, improving reproducibility and evaluation consistency?
• RQ4 How does the framework support standardized, reproducible evaluation of CL methods across different settings and resource constraints?
• RQ5 What is the impact of shared infrastructure on the development and comparison of new CL algorithms in both supervised and reinforcement learning contexts?

Key findings

• The hierarchical taxonomy of CL settings enables method inheritance, where a method developed for a general setting is automatically applicable to all more specific child settings.
• Sequoia reduces duplication of effort by allowing shared implementations of core techniques—such as experience replay—across both CSL and CRL, as demonstrated in both MonsterKong and MuJoCo benchmarks.
• Empirical evaluation shows a clear trade-off between final performance and online performance, with off-policy algorithms like DQN achieving higher final performance and on-policy ones like PPO achieving better online performance.
• The framework enables effective transfer learning in CRL, as shown by PPO’s transfer matrix in Continual-MonsterKong, where positive rewards above the diagonal indicate successful generalization to unseen tasks.
• Methods like GEM and GDumb achieve the best performance-runtime trade-off in incremental supervised learning, with GDumb serving as a strong baseline despite its non-online inference capability.
• The framework supports customizable environment configurations, such as in MonsterKong, allowing for systematic variation in layout, rewards, and visual features to test policy generalization.