[Paper Review] Temporal Unet: Sample Level Human Action Recognition using WiFi

Fei Wang, Yunpeng Song+3 more|arXiv (Cornell University)|Apr 19, 2019

Indoor and Outdoor Localization Technologies53 references17 citations

TL;DR

This paper introduces Temporal Unet, a novel deep learning model for sample-level human action recognition using WiFi channel state information (CSI). By leveraging temporal convolutional and deconvolutional layers with skip connections, it achieves high-precision classification of individual CSI samples into action classes, outperforming prior series-level methods with a mean AP of 0.98 for action detection and 0.86 for action classification on benchmark datasets.

ABSTRACT

Human doing actions will result in WiFi distortion, which is widely explored for action recognition, such as the elderly fallen detection, hand sign language recognition, and keystroke estimation. As our best survey, past work recognizes human action by categorizing one complete distortion series into one action, which we term as series-level action recognition. In this paper, we introduce a much more fine-grained and challenging action recognition task into WiFi sensing domain, i.e., sample-level action recognition. In this task, every WiFi distortion sample in the whole series should be categorized into one action, which is a critical technique in precise action localization, continuous action segmentation, and real-time action recognition. To achieve WiFi-based sample-level action recognition, we fully analyze approaches in image-based semantic segmentation as well as in video-based frame-level action recognition, then propose a simple yet efficient deep convolutional neural network, i.e., Temporal Unet. Experimental results show that Temporal Unet achieves this novel task well. Codes have been made publicly available at https://github.com/geekfeiw/WiSLAR.

Motivation & Objective

• To address the lack of fine-grained, sample-level action recognition in WiFi sensing, moving beyond traditional series-level classification.
• To enable precise temporal localization and real-time action recognition by labeling each individual CSI sample with an action class.
• To develop a unified deep learning framework that effectively handles both sample-level action detection and classification tasks.
• To overcome challenges in modeling temporal context and distinguishing subtle action transitions from noise in CSI data.

Proposed method

• Proposes Temporal Unet, a U-Net inspired architecture with temporal convolutional, pooling, and deconvolutional layers applied along the time axis of CSI sequences.
• Uses stacked temporal layers to capture multi-scale contextual information from neighboring samples, enabling robust feature learning across time.
• Employs skip connections between encoder and decoder blocks to preserve spatial-temporal details and improve gradient flow.
• Applies temporal max pooling and strided convolutions to downsample and upsample features while maintaining temporal resolution.
• Trains the model end-to-end using cross-entropy loss for classification and a modified AP-based loss for detection, optimizing for both tasks simultaneously.
• Utilizes CSI data from 3 OFDM subcarriers as input, processing raw time-series distortions induced by human actions.

Experimental results

Research questions

• RQ1 Can a deep learning model achieve accurate sample-level action recognition using only raw WiFi CSI time-series data?
• RQ2 How can temporal context from neighboring samples be effectively modeled to improve classification of individual CSI samples?
• RQ3 Can a single unified architecture perform both sample-level action detection and classification with high performance?
• RQ4 What is the impact of skip connections and multi-scale temporal convolutions on recognition accuracy in fine-grained action recognition?
• RQ5 How does the proposed method compare to existing series-level approaches in terms of temporal precision and real-time applicability?

Key findings

• Temporal Unet achieves a mean AP of 0.98 for sample-level action detection, indicating strong performance even at high success thresholds.
• For sample-level action classification, the model achieves a mean AP of 0.86, with high confidence in distinguishing most action classes.
• The confusion matrix reveals that the primary misclassification occurs between 'hand right' and 'hand left', likely due to symmetry.
• Transitional phases between 'non-action' and 'action' states are the most frequent source of errors in both detection and classification.
• Visualization results confirm that the model accurately identifies action onset and offset times, with confidence curves aligning well with manual annotations.
• The model outperforms prior series-level methods by enabling per-sample labeling, which is critical for real-time and continuous action segmentation.