[Paper Review] Attend And Discriminate: Beyond the State-of-the-Art for Human Activity Recognition using Wearable Sensors

Alireza Abedin, Mahsa Ehsanpour+3 more|arXiv (Cornell University)|Jul 14, 2020

Context-Aware Activity Recognition Systems4 citations

TL;DR

This paper proposes a novel deep learning framework for wearable sensor-based human activity recognition that enhances feature representation through cross-channel attention, center-loss regularization, and data-agnostic mixup augmentation. The method achieves new state-of-the-art performance across four diverse benchmarks, with up to a 6% relative improvement in accuracy by explicitly modeling sensor modality interactions and improving feature compactness and generalization.

ABSTRACT

Wearables are fundamental to improving our understanding of human activities, especially for an increasing number of healthcare applications from rehabilitation to fine-grained gait analysis. Although our collective know-how to solve Human Activity Recognition (HAR) problems with wearables has progressed immensely with end-to-end deep learning paradigms, several fundamental opportunities remain overlooked. We rigorously explore these new opportunities to learn enriched and highly discriminating activity representations. We propose: i) learning to exploit the latent relationships between multi-channel sensor modalities and specific activities; ii) investigating the effectiveness of data-agnostic augmentation for multi-modal sensor data streams to regularize deep HAR models; and iii) incorporating a classification loss criterion to encourage minimal intra-class representation differences whilst maximising inter-class differences to achieve more discriminative features. Our contributions achieves new state-of-the-art performance on four diverse activity recognition problem benchmarks with large margins -- with up to 6% relative margin improvement. We extensively validate the contributions from our design concepts through extensive experiments, including activity misalignment measures, ablation studies and insights shared through both quantitative and qualitative studies.

Motivation & Objective

• To address the underexplored potential of modeling latent relationships between multi-channel wearable sensor modalities and specific human activities.
• To improve generalization in HAR by applying data-agnostic mixup augmentation to multi-modal sensor time-series data, which remains underutilized despite success in computer vision.
• To reduce intra-class variability and enhance inter-class separability in learned representations by incorporating center-loss as a complementary optimization objective.
• To develop a unified, generalizable framework that outperforms state-of-the-art models across diverse wearable HAR benchmarks.

Proposed method

• Introduces a cross-channel interaction encoder (CIE) that uses self-attention to model latent correlations between 79 sensor channels, generating enriched feature maps for improved representation learning.
• Employs an attentional GRU encoder (AGE) to dynamically weigh temporal steps, focusing on the most informative hidden states in short sequences (T=8), particularly the final states.
• Incorporates center-loss as a regularization term to minimize intra-class feature variance while maximizing inter-class margins, enhancing discriminative power.
• Applies mixup augmentation to multi-channel time-series data via linear interpolation between pairs of samples, increasing training data diversity without modality-specific design.
• Combines CIE, AGE, center-loss, and mixup within an end-to-end deep learning pipeline for joint optimization of representation quality and generalization.
• Uses a unified evaluation protocol across four diverse HAR benchmarks to validate the framework’s robustness and generalizability.

Experimental results

Research questions

• RQ1 Can modeling cross-channel sensor interactions through self-attention improve representation learning in wearable-based HAR?
• RQ2 Does incorporating center-loss as a loss criterion lead to more compact and discriminative feature representations in HAR?
• RQ3 Can data-agnostic mixup augmentation effectively regularize deep HAR models trained on small, multi-modal sensor datasets?
• RQ4 How do the individual components (CIE, AGE, center-loss, mixup) contribute to performance gains in a unified framework?
• RQ5 To what extent is the proposed framework generalizable across diverse wearable sensor HAR benchmarks?

Key findings

• The proposed framework achieves new state-of-the-art performance on four diverse HAR benchmarks, with a maximum relative improvement of 6% over prior SoA methods.
• Mixup alone improved accuracy by 5.2% (from 67.2% to 70.7%) on the Opportunity dataset, demonstrating its effectiveness in augmenting multi-modal sensor sequences.
• The combination of mixup and center-loss yielded the highest performance gain, indicating that joint optimization for compactness and separability is highly effective.
• The cross-channel interaction encoder (CIE) with self-attention provided a larger performance gain than the attentional GRU (AGE), especially in short sequences, due to its ability to model inter-sensor dependencies.
• Visualizations of learned attention scores revealed meaningful, interpretable patterns—e.g., strong attention on right-arm IMUs during drinking tasks and on back/left-arm IMUs during bending actions—validating model interpretability.
• Ablation studies confirmed that each component (CIE, AGE, center-loss, mixup) contributes significantly to performance, with CIE and center-loss showing the most substantial individual improvements.