[Paper Review] DeepSense: A Unified Deep Learning Framework for Time-Series Mobile Sensing Data Processing

Shuochao Yao, Shaohan Hu+3 more|arXiv (Cornell University)|Nov 7, 2016

Context-Aware Activity Recognition SystemsComputer Science39 references105 citations

TL;DR

DeepSense integrates CNNs and GRUs to handle regression and classification on noisy time-series mobile sensor data, achieving state-of-the-art results on car tracking, heterogeneous activity recognition, and biometric user identification while remaining practical for on-device deployment.

ABSTRACT

Mobile sensing applications usually require time-series inputs from sensors. Some applications, such as tracking, can use sensed acceleration and rate of rotation to calculate displacement based on physical system models. Other applications, such as activity recognition, extract manually designed features from sensor inputs for classification. Such applications face two challenges. On one hand, on-device sensor measurements are noisy. For many mobile applications, it is hard to find a distribution that exactly describes the noise in practice. Unfortunately, calculating target quantities based on physical system and noise models is only as accurate as the noise assumptions. Similarly, in classification applications, although manually designed features have proven to be effective, it is not always straightforward to find the most robust features to accommodate diverse sensor noise patterns and user behaviors. To this end, we propose DeepSense, a deep learning framework that directly addresses the aforementioned noise and feature customization challenges in a unified manner. DeepSense integrates convolutional and recurrent neural networks to exploit local interactions among similar mobile sensors, merge local interactions of different sensory modalities into global interactions, and extract temporal relationships to model signal dynamics. DeepSense thus provides a general signal estimation and classification framework that accommodates a wide range of applications. We demonstrate the effectiveness of DeepSense using three representative and challenging tasks: car tracking with motion sensors, heterogeneous human activity recognition, and user identification with biometric motion analysis. DeepSense significantly outperforms the state-of-the-art methods for all three tasks. In addition, DeepSense is feasible to implement on smartphones due to its moderate energy consumption and low latency

Motivation & Objective

• Motivate improved exploitation of noisy time-series sensor data in mobile sensing and computing.
• Propose a unified deep learning framework that accommodates both regression and classification tasks.
• Leverage local sensor interactions, multimodal fusion, and temporal modeling to learn physical/noise representations and robust features.
• Demonstrate on-device feasibility in terms of energy and latency.

Proposed method

• Split input sensor measurements into time intervals and transform to the frequency domain per interval.
• Apply per-sensor convolutional subnets to capture intra-interval local interactions and inter-sensor global interactions.
• Merge sensor outputs with a merge convolutional subnet to learn cross-sensor relationships.
• Use a two-layer stacked GRU to learn inter-interval temporal dependencies with dropout and recurrent batch normalization.
• For regression, decode outputs with a shared linear layer to estimate physical quantities per interval.
• For classification, aggregate interval features (e.g., averaging) and apply softmax for category probabilities.

Experimental results

Research questions

• RQ1 Can a unified CNN-RNN framework learn both the physical signal/noise composition for regression and robust feature representations for classification from noisy mobile sensor data?
• RQ2 Does hierarchical sensing with per-sensor CNNs, multimodal fusion, and temporal modeling outperform task-specific baselines across regression and classification tasks?
• RQ3 Is the framework feasible for on-device deployment in terms of energy consumption and latency?
• RQ4 How does DeepSense perform on challenging tasks like car tracking, heterogeneous human activity recognition, and biometric user identification?

Key findings

• DeepSense substantially reduces tracking error in car tracking (CarTrack) compared with baselines, achieving 40.43 ± 5.24 m MAE and 93.8% map-aided accuracy.
• Compared with variants, DeepSense with all components (individual and merge conv nets, stacked GRU) yields the best performance, while removing components degrades results.
• For CarTrack, several baselines (DS-SingleGRU, DS-noIndvConv, DS-noMergeConv) show progressively worse MAE and map-aided accuracy, confirming the value of both per-sensor processing and cross-sensor fusion.
• In HHAR (heterogeneous human activity recognition) and UserID (biometric motion-based identification), DeepSense and variants outperform state-of-the-art baselines by a large margin (at least 10%).
• The framework is demonstrated to be implementable on mobile devices with moderate energy use and low latency (on-device processing).