[Paper Review] Unsupervised Human Activity Recognition through Two-stage Prompting with ChatGPT

Qingxin Xia, Takuya Maekawa+1 more|arXiv (Cornell University)|Jun 3, 2023

Context-Aware Activity Recognition Systems4 citations

TL;DR

This paper proposes a two-stage prompt engineering method for ChatGPT to enable unsupervised Human Activity Recognition (HAR) using object usage sequences. By first generating activity descriptions that emphasize distinguishing objects and then refining predictions with knowledge-enhanced prompting, the approach achieves state-of-the-art F1-scores of 91.15% on Opportunity, 100% on 50Salads, and 100% on CMU-MMAC, outperforming zero-shot, retrieval-based, and few-shot baselines.

ABSTRACT

Wearable sensor devices, which offer the advantage of recording daily objects used by a person while performing an activity, enable the feasibility of unsupervised Human Activity Recognition (HAR). Unfortunately, previous unsupervised approaches using the usage sequence of objects usually require a proper description of activities manually prepared by humans. Instead, we leverage the knowledge embedded in a Large Language Model (LLM) of ChatGPT. Because the sequence of objects robustly characterizes the activity identity, it is possible that ChatGPT already learned the association between activities and objects from existing contexts. However, previous prompt engineering for ChatGPT exhibits limited generalization ability when dealing with a list of words (i.e., sequence of objects) due to the similar weighting assigned to each word in the list. In this study, we propose a two-stage prompt engineering, which first guides ChatGPT to generate activity descriptions associated with objects while emphasizing important objects for distinguishing similar activities; then outputs activity classes and explanations for enhancing the contexts that are helpful for HAR. To the best of our knowledge, this is the first study that utilizes ChatGPT to recognize activities using objects in an unsupervised manner. We conducted our approach on three datasets and demonstrated the state-of-the-art performance.

Motivation & Objective

• To enable unsupervised Human Activity Recognition (HAR) without relying on costly human-annotated labels.
• To overcome the limitation of existing prompt engineering in distinguishing similar activities due to uniform word weighting in object sequences.
• To leverage pre-trained knowledge in ChatGPT for automatic, context-aware activity description generation based on object usage.
• To improve HAR performance by guiding ChatGPT to focus on discriminative objects through structured prompting.
• To demonstrate the effectiveness of two-stage prompting in generating robust, generalizable knowledge for HAR in an unsupervised setting.

Proposed method

• The first stage uses a knowledge generation prompt that presents two similar activities and their associated object sequences, prompting ChatGPT to generate descriptive distinctions emphasizing key differentiating objects.
• The second stage employs knowledge prompt engineering to generate final activity predictions, incorporating the generated descriptions as context and requiring an explanation for the prediction.
• The method uses the GPT-3.5 text-Davinci-003 model with temperature=0 and top_p=0.5 for deterministic outputs.
• Object sequences from segmented time periods are used as input, with activity classes provided in the prompt for zero-shot, few-shot, or retrieval-based baselines.
• The approach is designed to enhance context relevance by embedding task-specific knowledge directly into the prompt structure.
• The two-stage process ensures that ChatGPT focuses on salient features for discrimination, improving generalization across diverse datasets.

Figure 1. The overview of the Proposed approach. The pink background rectangles show the input of the model, the orange rectangles show the procedures done by ChatGPT, and the green rectangles show the output results of each prompt for ChatGPT processing.

Experimental results

Research questions

• RQ1 Can large language models like ChatGPT be effectively leveraged for unsupervised HAR using only object usage sequences?
• RQ2 How can prompt engineering be designed to improve the model's ability to distinguish between similar activities with overlapping object usage?
• RQ3 Does two-stage prompting, involving intermediate knowledge generation and refined prediction, yield better HAR performance than standard prompting techniques?
• RQ4 Can the model generate high-quality, discriminative descriptions of activities without human-annotated descriptions?
• RQ5 How does the proposed method generalize across diverse HAR datasets with varying levels of object overlap?

Key findings

• The proposed two-stage prompting method achieved an F1-score of 91.15% on the Opportunity dataset, significantly outperforming the zero-shot baseline (53.61%) and retrieval-based method (48.19%).
• On the 50Salads dataset, the proposed method achieved a perfect F1-score of 100.00%, surpassing the few-shot baseline (90.61%) and retrieval-based method (93.87%).
• On the CMU-MMAC dataset, the proposed method achieved a 100.00% F1-score, outperforming all baselines, including the few-shot method (100.00%) and retrieval-based method (71.05%).
• The confusion matrix for the Opportunity dataset showed that the proposed method successfully distinguished the cleanup activity from others despite high object overlap, indicating effective focus on discriminative features.
• The method demonstrated robustness across datasets with varying degrees of object overlap, suggesting strong generalization capability in unsupervised HAR.
• The results indicate that automatically generated knowledge via two-stage prompting is more effective and consistent than manually prepared few-shot examples or retrieved knowledge, especially in complex, real-world scenarios.

Figure 2. The confusion matrix of comparative prompting methods and the proposed method of Opportunity dataset.