[Paper Review] Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning

Linfeng Xu, Qingbo Wu+7 more|arXiv (Cornell University)|Jan 26, 2023

Human Pose and Action Recognition60 references4 citations

TL;DR

This paper introduces UESTC-MMEA-CL, a novel multi-modal egocentric activity dataset combining RGB video, accelerometer, and gyroscope data from 10 participants performing 32 daily activities. It enables continual learning research in first-person activity recognition, demonstrating that exemplar-based methods like iCaRL significantly reduce catastrophic forgetting—achieving 77.8% average accuracy with RGB+Acc+Gyro fusion—while sensor-only modalities suffer severe performance drops without exemplars.

ABSTRACT

With the rapid development of wearable cameras, a massive collection of egocentric video for first-person visual perception becomes available. Using egocentric videos to predict first-person activity faces many challenges, including limited field of view, occlusions, and unstable motions. Observing that sensor data from wearable devices facilitates human activity recognition, multi-modal activity recognition is attracting increasing attention. However, the deficiency of related dataset hinders the development of multi-modal deep learning for egocentric activity recognition. Nowadays, deep learning in real world has led to a focus on continual learning that often suffers from catastrophic forgetting. But the catastrophic forgetting problem for egocentric activity recognition, especially in the context of multiple modalities, remains unexplored due to unavailability of dataset. In order to assist this research, we present a multi-modal egocentric activity dataset for continual learning named UESTC-MMEA-CL, which is collected by self-developed glasses integrating a first-person camera and wearable sensors. It contains synchronized data of videos, accelerometers, and gyroscopes, for 32 types of daily activities, performed by 10 participants. Its class types and scale are compared with other publicly available datasets. The statistical analysis of the sensor data is given to show the auxiliary effects for different behaviors. And results of egocentric activity recognition are reported when using separately, and jointly, three modalities: RGB, acceleration, and gyroscope, on a base network architecture. To explore the catastrophic forgetting in continual learning tasks, four baseline methods are extensively evaluated with different multi-modal combinations. We hope the UESTC-MMEA-CL can promote future studies on continual learning for first-person activity recognition in wearable applications.

Motivation & Objective

• To address the lack of multi-modal, continual learning-ready datasets for egocentric activity recognition, particularly integrating wearable sensor data with first-person video.
• To investigate catastrophic forgetting in multi-modal deep learning for egocentric activity recognition, especially when combining vision and inertial sensor data.
• To evaluate the effectiveness of continual learning strategies—such as iCaRL, EWC, and LwF—on multi-modal egocentric activity recognition under sequential task learning.
• To provide a benchmark dataset and baseline model to accelerate research in continual, multi-modal first-person activity recognition for wearable applications.

Proposed method

• The authors developed a custom wearable glasses system integrating a first-person camera with inertial measurement units (IMUs) to collect synchronized RGB video, accelerometer, and gyroscope data.
• The dataset, named UESTC-MMEA-CL, comprises 32 daily activity classes performed by 10 participants, with full temporal and spatial synchronization across modalities.
• A mid-fusion (TBW-like) architecture is used to combine features from RGB, acceleration, and gyroscope modalities within a shared deep neural network for joint representation learning.
• Four continual learning strategies—fine-tune, iCaRL (exemplar-based), EWC (parameter regularization), and LwF (knowledge distillation)—are evaluated on sequential task learning.
• Statistical analysis of sensor data is performed to validate the auxiliary role of inertial signals in distinguishing activity patterns.
• Experiments are conducted using both uni-modal and multi-modal combinations, with performance measured via average accuracy (A) and forgetting rate (F).

Experimental results

Research questions

• RQ1 How does catastrophic forgetting manifest in multi-modal egocentric activity recognition when learning tasks sequentially?
• RQ2 To what extent do different continual learning strategies—especially exemplar-based vs. exemplar-free—mitigate forgetting in RGB, accelerometer, and gyroscope modalities?
• RQ3 How effective is multi-modal fusion (RGB + Acc + Gyro) in reducing forgetting compared to uni-modal learning?
• RQ4 Why do sensor-only networks (accelerometer and gyroscope) exhibit significantly higher forgetting and lower accuracy than RGB networks in continual learning?
• RQ5 Can the proposed UESTC-MMEA-CL dataset serve as a reliable benchmark for future research in continual, multi-modal egocentric activity recognition?

Key findings

• The iCaRL method achieved the highest average accuracy of 77.8% on the RGB+Acc+Gyro multi-modal combination, with a relatively low forgetting rate of 33.5%.
• Exemplar-based iCaRL significantly outperformed exemplar-free methods EWC and LwF in reducing forgetting, especially in multi-modal settings.
• Sensor-only networks (accelerometer and gyroscope) achieved less than 20% average accuracy even with continual learning, with forgetting rates exceeding 60%.
• The RGB+Gyro combination achieved 77.4% average accuracy with iCaRL, outperforming the standalone RGB network (70.4%) and showing strong complementary effects.
• Fine-tuning resulted in the worst performance, with a 99.0% forgetting rate on the RGB+Acc+Gyro combination, highlighting the severity of catastrophic forgetting without regularization.
• The statistical analysis of sensor data confirmed that inertial signals provide distinct and useful patterns for different activity classes, supporting their integration with vision data.