[Paper Review] A distillation-based approach integrating continual learning and federated learning for pervasive services

Anastasiia Usmanova, François Portet+2 more|arXiv (Cornell University)|Sep 9, 2021

Domain Adaptation and Few-Shot Learning31 references33 citations

TL;DR

The paper introduces a distillation-based method (FLwF-2T) for Federated Continual Learning in pervasive HAR on mobile devices, using a two-teacher setup (past client model and server global model) to mitigate catastrophic forgetting.

ABSTRACT

Federated Learning, a new machine learning paradigm enhancing the use of edge devices, is receiving a lot of attention in the pervasive community to support the development of smart services. Nevertheless, this approach still needs to be adapted to the specificity of the pervasive domain. In particular, issues related to continual learning need to be addressed. In this paper, we present a distillation-based approach dealing with catastrophic forgetting in federated learning scenario. Specifically, Human Activity Recognition tasks are used as a demonstration domain.

Motivation & Objective

• Address catastrophic forgetting in Federated Continual Learning (FCL) for pervasive computing tasks such as Human Activity Recognition (HAR).
• Investigate whether FL can share past knowledge across clients and improve performance on unseen tasks.
• Leverage a server-generated global model as a second teacher to enhance general features learned by clients.

Proposed method

• Propose a distillation-based FL method inspired by Learning without Forgetting (LwF) to constrain changes to past knowledge.
• Introduce FLwF-2T, where each client learns from both its past client model (teacher 1) and the server global model (teacher 2) via distillation losses.
• Define a final loss L_FLwF-2T = α L_class + β L_dis_cl + (1−α−β) L_dis_serv, with α, β regulating contributions of teachers and the classification term.
• Use temperature-scaled logits for distillation (temperature T) and soft targets to preserve prior task performance.
• Evaluate on the UCI HAR dataset with a CNN-based architecture suitable for mobile devices.

Experimental results

Research questions

• RQ1 Does Federated Learning help prevent catastrophic forgetting on a client side in a class-incremental setting?
• RQ2 Can FL facilitate sharing past knowledge across clients to improve performance on unknown tasks?
• RQ3 Can the server's global model act as an additional teacher to improve client performance and reduce forgetting?

Key findings

• FLwF-2T generally improves forgetting resistance on Client 1, achieving a lower forgetting measure F_2^1 (0.418) than FLwF (0.595) in the reported CL metrics.
• In generality tests, FLwF-2T/FT + exemplars achieved strong performance with favorable generalization when combining client-specific and server knowledge streams.
• Using two teachers (client past model and server model) can outperform single-teacher baselines in terms of maintaining previously learned information while learning new tasks.
• Exemplar memory further boosts performance across FL and CL metrics for several strategies, with FLwF-2T/FT + ex often yielding strong results.
• The proposed FLwF-2T framework is computationally lightweight, suitable for mobile devices, as it relies on past client models and the server-provided global model during rounds.