[Paper Review] Human Activity Recognition Based on Wearable Sensor Data: A Standardization of the State-of-the-Art

Artur Jordão, Antonio C. Nazaré+2 more|arXiv (Cornell University)|Jun 13, 2018

Context-Aware Activity Recognition Systems27 references80 citations

TL;DR

The paper standardizes evaluation in wearable-sensor human activity recognition by analyzing sample generation and validation protocols, showing bias in common methods and proposing two new sample generation approaches plus dataset standardization.

ABSTRACT

Human activity recognition based on wearable sensor data has been an attractive research topic due to its application in areas such as healthcare and smart environments. In this context, many works have presented remarkable results using accelerometer, gyroscope and magnetometer data to represent the activities categories. However, current studies do not consider important issues that lead to skewed results, making it hard to assess the quality of sensor-based human activity recognition and preventing a direct comparison of previous works. These issues include the samples generation processes and the validation protocols used. We emphasize that in other research areas, such as image classification and object detection, these issues are already well-defined, which brings more efforts towards the application. Inspired by this, we conduct an extensive set of experiments that analyze different sample generation processes and validation protocols to indicate the vulnerable points in human activity recognition based on wearable sensor data. For this purpose, we implement and evaluate several top-performance methods, ranging from handcrafted-based approaches to convolutional neural networks. According to our study, most of the experimental evaluations that are currently employed are not adequate to perform the activity recognition in the context of wearable sensor data, in which the recognition accuracy drops considerably when compared to an appropriate evaluation approach. To the best of our knowledge, this is the first study that tackles essential issues that compromise the understanding of the performance in human activity recognition based on wearable sensor data.

Motivation & Objective

• Identify factors that bias performance in wearable sensor activity recognition evaluations.
• Assess how sample generation and validation protocols affect reported accuracy.
• Implement and compare multiple top-performing methods (handcrafted features to ConvNets) under standardized evaluation.
• Propose two novel data sample generation processes to reduce bias and enable fair comparisons.
• Standardize publicly available wearable sensor datasets to facilitate reproducible evaluation.

Proposed method

• Implement and evaluate multiple state-of-the-art activity recognition methods ranging from handcrafted features to convolutional neural networks.
• Analyze the impact of data-sample generation processes on recognition performance and bias.
• Propose Full-Non-Overlapping-Window and Leave-One-Trial-Out sample generation processes to address bias and data scarcity.
• Standardize datasets with differing sensor configurations to enable uniform evaluation.
• Use 10-fold cross-validation and leave-one-subject-out validation to examine protocol effects on performance.
• Perform statistical validation to compare methods and assess equivalence.

Experimental results

Research questions

• RQ1 How do sample generation processes bias reported activity recognition accuracy in wearable sensor data?
• RQ2 What validation protocols yield robust, less biased performance estimates across diverse datasets?
• RQ3 Do handcrafted and ConvNet-based methods remain competitive under standardized evaluation?
• RQ4 Can new data-sample generation schemes reduce bias without excessively reducing sample size?
• RQ5 How should public datasets be standardized to enable reproducible benchmarking?

Key findings

• Bias from semi-non-overlapping-window sampling can inflate accuracy by causing training and testing overlap; performance drops when bias is removed.
• Full-Non-Overlapping-Window reduces bias but yields fewer samples, potentially lowering accuracy.
• Leave-One-Trial-Out preserves cross-validation structure while preventing overlap within trials, mitigating bias.
• Under standardized evaluation, many methods' accuracies decrease by about ten percentage points on average compared to biased setups.
• ConvNet-based methods require careful architectural choices (channel separation, kernel sizes) and may be impractical on low-sampling-rate datasets.
• Standardization across datasets reveals variability in robustness of methods and emphasizes the need for reproducible benchmarks.