[Paper Review] Understanding and Improving Deep Neural Network for Activity Recognition

Xue Li, Xiandong Si+5 more|arXiv (Cornell University)|May 18, 2018

Context-Aware Activity Recognition Systems21 references3 citations

TL;DR

This paper proposes a novel visualization-driven approach to understand and improve deep neural networks for sensor-based human activity recognition. By analyzing feature maps from a CNN on smartphone sensor data, the authors identify discriminative features linked to specific activities, which are then fused into a DNN model to achieve a 96.1% classification accuracy—marking the first visualization of abstract sensor-based features in this domain.

ABSTRACT

Activity recognition has become a popular research branch in the field of pervasive computing in recent years. A large number of experiments can be obtained that activity sensor-based data's characteristic in activity recognition is variety, volume, and velocity. Deep learning technology, together with its various models, is one of the most effective ways of working on activity data. Nevertheless, there is no clear understanding of why it performs so well or how to make it more effective. In order to solve this problem, first, we applied convolution neural network on Human Activity Recognition Using Smart phones Data Set. Second, we realized the visualization of the sensor-based activity's data features extracted from the neural network. Then we had in-depth analysis of the visualization of features, explored the relationship between activity and features, and analyzed how Neural Networks identify activity based on these features. After that, we extracted the significant features related to the activities and sent the features to the DNN-based fusion model, which improved the classification rate to 96.1%. This is the first work to our knowledge that visualizes abstract sensor-based activity data features. Based on the results, the method proposed in the paper promises to realize the accurate classification of sensor- based activity recognition.

Motivation & Objective

• To address the lack of interpretability in deep neural networks for activity recognition using sensor data.
• To visualize and analyze the abstract features learned by deep networks from raw sensor signals.
• To identify significant, activity-discriminative features from the learned representations.
• To improve classification performance by fusing these salient features into a DNN-based model.
• To establish a foundation for interpretability and optimization of deep learning in sensor-based activity recognition.

Proposed method

• Applied a convolutional neural network (CNN) to the Human Activity Recognition Using Smart Phones Dataset for end-to-end feature learning.
• Developed a visualization technique to interpret the abstract features extracted by the CNN layers from sensor data.
• Conducted in-depth analysis of visualized features to correlate them with specific human activities.
• Extracted the most discriminative features based on their activity-specific patterns and relevance.
• Fed the selected features into a deep neural network (DNN) fusion model to enhance classification performance.
• Optimized the fusion model using the extracted features to maximize recognition accuracy.

Experimental results

Research questions

• RQ1 How do deep neural networks learn and represent features from raw sensor data in activity recognition?
• RQ2 What is the relationship between learned features and specific human activities?
• RQ3 Which features are most discriminative for different activity classes?
• RQ4 Can visualizing and selecting key features improve the performance of deep learning models in activity recognition?
• RQ5 To what extent can feature fusion enhance classification accuracy beyond standard deep learning approaches?

Key findings

• The visualization of learned features revealed clear spatial and temporal patterns corresponding to distinct human activities.
• Specific feature maps in the CNN were consistently activated by particular activities, indicating strong feature-activity correspondence.
• The most discriminative features were found to capture motion dynamics and temporal transitions unique to each activity.
• Fusing the selected significant features into a DNN model improved classification accuracy to 96.1%.
• This performance gain demonstrates the effectiveness of interpretability-driven feature selection in deep learning for sensor-based recognition.
• The study presents the first known visualization of abstract sensor-based features in deep learning for activity recognition.