[Paper Review] TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation

Sungho Suh, Vítor Fortes Rey+1 more|arXiv (Cornell University)|Sep 14, 2022

Context-Aware Activity Recognition Systems4 citations

TL;DR

TASKED proposes a Transformer-based adversarial learning framework with self-knowledge distillation for wearable sensor-based human activity recognition, improving cross-subject generalization by aligning feature distributions across domains using MMD regularization and enhancing training stability. It achieves state-of-the-art performance, outperforming prior methods by up to 3.89 percentage points in F1-macro across four public HAR datasets.

ABSTRACT

Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human-computer interaction applications. However, they are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, they still face significant challenging problems with performance degradation due to large gaps in the distribution of training and test data, and behavioral differences between subjects. In this work, we propose a novel Transformer-based Adversarial learning framework for human activity recognition using wearable sensors via Self-KnowledgE Distillation (TASKED), that accounts for individual sensor orientations and spatial and temporal features. The proposed method is capable of learning cross-domain embedding feature representations from multiple subjects datasets using adversarial learning and the maximum mean discrepancy (MMD) regularization to align the data distribution over multiple domains. In the proposed method, we adopt the teacher-free self-knowledge distillation to improve the stability of the training procedure and the performance of human activity recognition. Experimental results show that TASKED not only outperforms state-of-the-art methods on the four real-world public HAR datasets (alone or combined) but also improves the subject generalization effectively.

Motivation & Objective

• Address performance degradation in human activity recognition (HAR) due to distribution shifts between training and test data across different subjects.
• Overcome limitations of CNNs and RNNs in capturing long-range spatial-temporal dependencies in wearable sensor data.
• Improve model generalization to unseen subjects by learning domain-invariant representations without requiring labeled subject data during inference.
• Stabilize training and enhance performance through a teacher-free self-knowledge distillation mechanism.
• Develop a unified framework that jointly optimizes feature extraction, domain alignment, and robustness to individual behavioral variations.

Proposed method

• Employ a Transformer encoder to model spatial and temporal dependencies across multi-sensor time-series data from wearable devices.
• Introduce an adversarial training scheme where a subject discriminator is trained to distinguish between subjects, while the feature extractor is trained to fool it.
• Apply Maximum Mean Discrepancy (MMD) regularization across all subject pairs to align feature distributions across domains and improve domain generalization.
• Implement a teacher-free self-knowledge distillation technique to stabilize training and reduce bias introduced by MMD and adversarial losses.
• Optimize the overall framework end-to-end using a combined loss: cross-entropy for classification, adversarial loss for domain invariance, and distillation loss for feature consistency.
• Use a multi-head attention mechanism with relative positional encoding to preserve spatial relationships between sensors on the body.

Experimental results

Research questions

• RQ1 Can a Transformer-based architecture effectively learn spatial and temporal representations from wearable sensor data for human activity recognition?
• RQ2 To what extent does adversarial learning with MMD regularization improve cross-subject generalization in HAR?
• RQ3 How does self-knowledge distillation enhance training stability and performance in a domain generalization setting for HAR?
• RQ4 Does the proposed framework outperform state-of-the-art methods in terms of accuracy, F1-macro, and F1-weighted on real-world HAR datasets?
• RQ5 Can the model generalize effectively when evaluated on unseen subjects, particularly when subject-specific patterns are not available during inference?

Key findings

• TASKED achieved a 3.08 percentage point improvement in accuracy, 3.89 percentage points in F1-weighted, and 3.39 percentage points in F1-macro over the best-performing state-of-the-art method across four public HAR datasets.
• The model outperformed the Transformer baseline by 1.71 and 2.96 percentage points in accuracy and F1-macro, respectively, when using combined datasets.
• In Bayesian correlated t-tests, TASKED was more than 1% better than METIER in F1-macro with a 70% probability, and the probability of equivalence was only around 30%.
• The framework demonstrated the smallest variance in performance across all metrics, as shown in box-and-whisker plots, indicating consistent and robust results.
• The proposed method generalized effectively to unseen subjects, with significant performance gains even when training and testing data came from different subject distributions.
• Self-knowledge distillation contributed to training stability and prevented overfitting to the MMD and adversarial regularization, leading to improved generalization.