[Paper Review] Continual learning of longitudinal health records

J. Armstrong, David W. Clifton|arXiv (Cornell University)|Dec 22, 2021

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper evaluates continual learning (CL) methods on longitudinal ICU patient data to address catastrophic forgetting in clinical time-series settings. It finds that replay-based methods outperform regularization techniques in long-term stability across diverse domain shifts, especially when handling large numbers of sequential tasks.

ABSTRACT

Continual learning denotes machine learning methods which can adapt to new environments while retaining and reusing knowledge gained from past experiences. Such methods address two issues encountered by models in non-stationary environments: ungeneralisability to new data, and the catastrophic forgetting of previous knowledge when retrained. This is a pervasive problem in clinical settings where patient data exhibits covariate shift not only between populations, but also continuously over time. However, while continual learning methods have seen nascent success in the imaging domain, they have been little applied to the multi-variate sequential data characteristic of critical care patient recordings. Here we evaluate a variety of continual learning methods on longitudinal ICU data in a series of representative healthcare scenarios. We find that while several methods mitigate short-term forgetting, domain shift remains a challenging problem over large series of tasks, with only replay based methods achieving stable long-term performance. Code for reproducing all experiments can be found at https://github.com/iacobo/continual

Motivation & Objective

• Address catastrophic forgetting in clinical machine learning models trained on evolving longitudinal health records.
• Evaluate the performance of various continual learning strategies—especially regularization and replay—on real-world ICU time-series data.
• Investigate domain shifts across time, geography, and patient demographics in multi-variate sequential EHR data.
• Assess the feasibility and robustness of CL methods under clinical constraints such as data privacy and limited storage.
• Provide a comprehensive benchmark for continual learning in critical care using real datasets (eICU-CRD and MIMIC-III)

Proposed method

• Evaluate multiple continual learning strategies—including Elastic Weight Consolidation (EWC), Knowledge Distillation (LwF), and gradient-based methods (GEM, A-GEM)—on sequential ICU data batches.
• Use a task-incremental setup where models are trained on successive data batches representing different hospitals, time periods, or patient demographics.
• Implement early stopping per task to ensure convergence and enable fair comparison across methods.
• Apply replay-based methods with fixed memory buffers to store past examples, and compare performance against regularization-only approaches.
• Use balanced accuracy as the primary metric, with bootstrapped 95% confidence intervals to assess statistical reliability.
• Conduct experiments across multiple domain shift scenarios: age, ward, ethnicity (broad), hospital, and time (seasonal/annual)

Experimental results

Research questions

• RQ1 How do regularization-based continual learning methods perform on longitudinal ICU time-series data compared to replay-based methods under increasing domain shifts?
• RQ2 To what extent do continual learning strategies mitigate catastrophic forgetting when training on sequential, non-stationary clinical data?
• RQ3 Can continual learning methods maintain stable performance across a large number of sequential tasks (e.g., 10+ hospitals) in real-world clinical settings?
• RQ4 How do different model architectures (e.g., Transformers, RNNs) interact with continual learning strategies in multi-variate time-series health data?
• RQ5 What is the impact of data privacy constraints on the applicability of replay-based methods in clinical continual learning?

Key findings

• Regularization methods such as LwF showed strong performance on short sequences of domain shifts (e.g., age, ward, broad ethnicity), but performance degraded significantly as the number of tasks increased.
• For $ n \geq 5 $ tasks in the hospital domain shift scenario, regularization methods performed no better than naive fine-tuning, indicating failure to prevent catastrophic forgetting.
• Replay-based methods, particularly explicit replay with fixed memory buffers, achieved the highest performance across all domain shift scenarios and maintained stable performance even with large numbers of tasks.
• A-GEM and GEM showed relatively stable performance across increasing heterogeneity in hospitals, while regularization methods exhibited high volatility due to overfitting to early task distributions.
• No method achieved performance comparable to the cumulative upper bound (training on all data at once) when $ n \geq 5 $, highlighting the challenge of long-term continual learning in clinical settings.
• Rehearsal methods improved with larger memory capacity, demonstrating that storage size directly impacts performance stability in continual learning for EHR data.