[Paper Review] Importance of user inputs while using incremental learning to personalize human activity recognition models

Pekka Siirtola, Heli Koskimäki+1 more|arXiv (Cornell University)|May 28, 2019

Context-Aware Activity Recognition Systems8 references4 citations

TL;DR

This paper proposes a semi-supervised incremental learning approach for personalizing human activity recognition models using inertial sensor data, where users label only uncertain predictions (12–26% of data), achieving near-supervised accuracy with minimal user effort. The method reduces concept drift and achieves error rates within 2%-units of full-supervision, outperforming non-supervised and user-independent models significantly.

ABSTRACT

In this study, importance of user inputs is studied in the context of personalizing human activity recognition models using incremental learning. Inertial sensor data from three body positions are used, and the classification is based on Learn++ ensemble method. Three different approaches to update models are compared: non-supervised, semi-supervised and supervised. Non-supervised approach relies fully on predicted labels, supervised fully on user labeled data, and the proposed method for semi-supervised learning, is a combination of these two. In fact, our experiments show that by relying on predicted labels with high confidence, and asking the user to label only uncertain observations (from 12% to 26% of the observations depending on the used base classifier), almost as low error rates can be achieved as by using supervised approach. In fact, the difference was less than 2%-units. Moreover, unlike non-supervised approach, semi-supervised approach does not suffer from drastic concept drift, and thus, the error rate of the non-supervised approach is over 5%-units higher than using semi-supervised approach.

Motivation & Objective

• To address concept drift in incremental learning for personalized human activity recognition by incorporating user feedback.
• To evaluate the trade-off between labeling effort and recognition accuracy in personalizing models using inertial sensor data.
• To compare non-supervised, semi-supervised, and supervised learning approaches in incremental model personalization.
• To determine the minimal user input required to achieve near-supervised performance in activity recognition.
• To assess the impact of base classifier choice and confidence thresholding on labeling efficiency and model accuracy.

Proposed method

• The Learn++ ensemble method is used for incremental learning, allowing online model updates without full retraining.
• Three base classifiers—LDA, QDA, and CART—are evaluated for their performance in generating prediction confidence scores.
• A semi-supervised approach selects predictions with confidence above a threshold (0.90 or 0.95) for model update, while uncertain predictions (below threshold) are flagged for user labeling.
• User-labeled instances are used to correct mispredictions and prevent concept drift, with two adjacent windows also labeled for context.
• The method combines streaming data predictions with selective human feedback to refine model accuracy over time.
• Balanced accuracy is computed across nine subjects using data from three body positions (waist, wrist, arm), with leave-one-subject-out cross-validation.

Experimental results

Research questions

• RQ1 Can a semi-supervised incremental learning approach reduce user labeling effort while maintaining high recognition accuracy in personalized human activity recognition?
• RQ2 How does the inclusion of user-labeled uncertain predictions affect model performance compared to non-supervised and fully supervised approaches?
• RQ3 What is the optimal threshold for confidence-based selection of instances requiring user labeling, and how does it vary by base classifier?
• RQ4 To what extent does the semi-supervised approach mitigate concept drift compared to non-supervised incremental learning?
• RQ5 How does the choice of base classifier (LDA, QDA, CART) influence labeling efficiency and final model accuracy?

Key findings

• The semi-supervised approach achieved an average error rate of 4.2–4.3%, compared to 9.5% for the non-supervised approach, demonstrating significant mitigation of concept drift.
• By labeling only 12–26% of instances (depending on classifier), the semi-supervised method achieved error rates within less than 2%-units of the fully supervised approach.
• The error rate of the semi-supervised method was consistently lower than non-supervised learning, with a 5.7–11.4 percentage-point gap depending on the base classifier and body position.
• No significant difference was observed between threshold values of 0.90 and 0.95, as both resulted in similar labeling rates (12–26%) and model performance.
• The method reduced the number of replaced labels to 25–47% of the data, depending on the classifier, while maintaining high accuracy.
• The proposed method outperformed user-independent models by over 10 percentage points in error rate reduction across all classifiers and body positions.