[Paper Review] Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

Hyeongju Choi, Apoorva Beedu+2 more|arXiv (Cornell University)|Nov 8, 2022

Human Pose and Action Recognition4 citations

TL;DR

This paper proposes a two-stage multi-modal framework that fuses RGB video and IMU sensor data for human activity recognition, using 3D-CNN and 1D-CNN encoders followed by feature fusion and classification. It achieves 96.05% accuracy on UTD-MHAD and 84.05% on MMAct, outperforming single-modality baselines by 22% and 11% respectively, and shows strong robustness in low-data and zero-shot settings.

ABSTRACT

To properly assist humans in their needs, human activity recognition (HAR) systems need the ability to fuse information from multiple modalities. Our hypothesis is that multimodal sensors, visual and non-visual tend to provide complementary information, addressing the limitations of other modalities. In this work, we propose a multi-modal framework that learns to effectively combine features from RGB Video and IMU sensors, and show its robustness for MMAct and UTD-MHAD datasets. Our model is trained in two-stage, where in the first stage, each input encoder learns to effectively extract features, and in the second stage, learns to combine these individual features. We show significant improvements of 22% and 11% compared to video only and IMU only setup on UTD-MHAD dataset, and 20% and 12% on MMAct datasets. Through extensive experimentation, we show the robustness of our model on zero shot setting, and limited annotated data setting. We further compare with state-of-the-art methods that use more input modalities and show that our method outperforms significantly on the more difficult MMact dataset, and performs comparably in UTD-MHAD dataset.

Motivation & Objective

• To improve human activity recognition (HAR) accuracy by effectively fusing complementary information from RGB video and IMU sensor modalities.
• To address limitations of single-modality systems—such as occlusion sensitivity in video and body-position dependency in IMU—through multi-modal learning.
• To demonstrate robust performance under data-scarce and zero-shot conditions, where unseen action classes are excluded during training.
• To outperform state-of-the-art methods using more complex or additional modalities, such as skeleton data, by focusing on efficient fusion of only video and IMU inputs.

Proposed method

• The framework uses a two-stage training process: first, separate encoders (3D-CNN for video, 1D-CNN for IMU) learn modality-specific features.
• The video modality is encoded using a S3D 3D-CNN architecture, while IMU data is processed through three stacked 1D-convolutional blocks with ReLU and dropout.
• In the second stage, the extracted features from both modalities are concatenated and passed through a two-layer MLP for final action classification.
• The model is trained end-to-end with cross-entropy loss, enabling joint optimization of feature learning and fusion.
• The method avoids reliance on post-processing skeleton estimation, using raw video and IMU inputs directly.
• A two-stage design allows individual modality encoders to learn strong representations before fusion, improving overall feature quality.

Experimental results

Research questions

• RQ1 Can a two-stage multi-modal learning framework that fuses RGB video and IMU data significantly improve HAR accuracy compared to single-modality baselines?
• RQ2 How does the proposed method perform under low-data and zero-shot settings, where some action classes are unseen during training?
• RQ3 Does the proposed fusion strategy outperform state-of-the-art methods that use additional modalities such as skeleton data?
• RQ4 How robust is the model when only 25% of the training data is available, and does multi-modal learning maintain an advantage in such settings?
• RQ5 Does video data provide more complementary information than IMU when the latter lacks coverage of certain action classes?

Key findings

• The proposed multi-modal model achieves 96.05% accuracy on the UTD-MHAD dataset, representing a 22% improvement over video-only and 11% over IMU-only baselines.
• On the MMAct dataset, the model reaches 84.05% accuracy, showing a 20% gain over video-only and 12% over IMU-only setups.
• In a zero-shot setting, where 5 action classes are hidden during training, the multi-modal model drops only 10% in accuracy, significantly outperforming single-modality setups that drop by 12–14%.
• With only 25% of the training data, the model maintains strong performance, with a 10% drop in accuracy on both UTD-MHAD and MMAct, demonstrating robustness to data scarcity.
• When IMU data lacks coverage of certain actions but video data includes them, the multi-modal setup (IMU* + RGB) outperforms (RGB* + IMU), confirming video's stronger complementarity in missing-modality scenarios.
• The model outperforms or matches state-of-the-art methods like VSKD and HAMLET, despite using only two modalities and avoiding complex post-processing like skeleton estimation.