[Paper Review] Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities

Kaixuan Chen, Dalin Zhang+4 more|arXiv (Cornell University)|Jan 21, 2020

Context-Aware Activity Recognition Systems195 references319 citations

TL;DR

This survey reviews state-of-the-art deep learning methods for sensor-based human activity recognition, categorizing challenges and mapping deep techniques to open issues, datasets, and future directions.

ABSTRACT

The vast proliferation of sensor devices and Internet of Things enables the applications of sensor-based activity recognition. However, there exist substantial challenges that could influence the performance of the recognition system in practical scenarios. Recently, as deep learning has demonstrated its effectiveness in many areas, plenty of deep methods have been investigated to address the challenges in activity recognition. In this study, we present a survey of the state-of-the-art deep learning methods for sensor-based human activity recognition. We first introduce the multi-modality of the sensory data and provide information for public datasets that can be used for evaluation in different challenge tasks. We then propose a new taxonomy to structure the deep methods by challenges. Challenges and challenge-related deep methods are summarized and analyzed to form an overview of the current research progress. At the end of this work, we discuss the open issues and provide some insights for future directions.

Motivation & Objective

• Motivate sensor-based activity recognition and its application domains (smart homes, healthcare, manufacturing).
• Highlight unique challenges in sensor data (feature extraction, data scarcity, distribution shifts, segmentation, multi-occupant scenarios, privacy and feasibility).
• Provide a taxonomy of deep learning approaches aligned with these challenges and analyze how they address them.
• Present public datasets and modalities to evaluate challenge-specific methods.
• Offer open issues and actionable insights for future research directions.

Proposed method

• Propose a taxonomy of deep learning methods organized by the challenges they address in sensor-based activity recognition.
• Discuss temporal feature extraction using RNNs (LSTM/GRU), CNNs (including 1D, multi-scale, dilated, and modality-specific variants), and hybrid CNN-RNN architectures.
• Explore multimodal feature extraction and fusion strategies (Early Fusion vs. Sensor Fusion), including feature-based and classifier-ensemble approaches.
• Summarize sensor modalities (wearable, ambient, object, and other modalities) and their datasets, with attention to data types and challenges such as class imbalance and distribution discrepancy.
• Review public datasets and their suitability for evaluating various challenges (multimodal, composite activities, multi-occupant scenarios).
• Highlight open issues and potential future directions in deep learning for sensor-based activity recognition.

Experimental results

Research questions

• RQ1 What are the main challenges unique to sensor-based human activity recognition, and how can deep learning methods be tailored to address them?
• RQ2 How do different deep learning architectures and fusion strategies perform across various sensor modalities and datasets?
• RQ3 What public datasets exist for evaluating challenge-specific methods, and what are their characteristics?
• RQ4 What future research directions and open issues can guide the development of more robust, scalable, and privacy-preserving HAR systems?

Key findings

• Deep learning enables end-to-end learning from multimodal sensor data, addressing feature extraction challenges by learning hierarchical representations.
• Temporal feature extraction is effectively handled by LSTM/GRU and CNN variants, including dilated and multi-scale approaches, with attention to time-scale diversity.
• Multimodal fusion strategies (early fusion, sensor fusion, and classifier ensembles) have distinct trade-offs in capturing intra- and inter-modality correlations.
• A wide range of sensor modalities (wearable, ambient, object, and other) and datasets support evaluation across simple to complex activities, including composite and multi-occupant scenarios.
• The survey provides a taxonomy linking challenges to methods, highlights available public datasets, and discusses open issues and future directions.