[Paper Review] LLaSA: A Sensor-Aware LLM for Natural Language Reasoning of Human Activity from IMU Data

Sheikh Asif Imran, Mohammad Nur Hossain Khan+2 more|arXiv (Cornell University)|Jun 20, 2024

Context-Aware Activity Recognition Systems4 citations

TL;DR

LLaSA introduces a sensor-aware large multimodal agent that integrates IMU data with LLMs using a novel dataset of 26,288 human activity narrations (SensorCaps) and 257,562 question-answer pairs (OpenSQA). By fusing LIMU-BERT with Llama via parameter-efficient fine-tuning, LLaSA achieves state-of-the-art performance in activity classification and open-ended question answering, enabling detailed, context-aware reasoning about motion data in healthcare and human-computer interaction applications.

ABSTRACT

Wearable systems can recognize activities from IMU data but often fail to explain their underlying causes or contextual significance. To address this limitation, we introduce two large-scale resources: SensorCap, comprising 35,960 IMU--caption pairs, and OpenSQA, with 199,701 question--answer pairs designed for causal and explanatory reasoning. OpenSQA includes a curated tuning split (Tune-OpenSQA) optimized for scientific accuracy, narrative clarity, and diagnostic insight. Leveraging these datasets, we develop LLaSA (Large Language and Sensor Assistant), a family of compact sensor-aware language models (7B and 13B) that generate interpretable, context-rich responses to open-ended questions grounded in raw IMU data. LLaSA outperforms commercial LLMs, including GPT-3.5 and GPT-4o-mini, on benchmark and real-world tasks, demonstrating the effectiveness of domain supervision and model alignment for sensor reasoning. Our code repository and datasets can be found at https://github.com/BASHLab/LLaSA.

Motivation & Objective

• To bridge the gap in instruction-following datasets for inertial measurement unit (IMU) data in multimodal AI.
• To develop a large multimodal agent (LLaSA) capable of understanding and reasoning about human activities using IMU sensor data.
• To enable open-ended question-answering about motion patterns by integrating LLMs with sensor-derived narratives.
• To advance sensor-aware language models for real-world applications in healthcare, sports science, and human-computer interaction.
• To establish a benchmark for evaluating sensor-aware LLMs on activity classification and reasoning tasks.

Proposed method

• Leveraged LIMU-BERT as a foundational encoder for IMU data, using self-supervised pre-training on unlabeled accelerometer and gyroscope signals.
• Applied trainable positional encoding to preserve temporal order in IMU sequences before feeding into the encoder.
• Combined LIMU-BERT with the Llama LLM via parameter-efficient fine-tuning (LoRA) to create a multimodal agent (LLaSA).
• Generated natural language narrations of IMU events using GPT-based models to create the SensorCaps dataset of 26,288 activity descriptions.
• Constructed OpenSQA, an instruction-following dataset with 257,562 question-answer pairs grounded in IMU-derived narratives.
• Evaluated LLaSA on a new benchmark combining closed-ended activity classification and open-ended question answering, using GPT-4o for qualitative assessment.

Figure 1: Architecture of the Large Language and Sensor Assistant (LLaSA) model

Experimental results

Research questions

• RQ1 Can a large language model be effectively fine-tuned to reason about human activities using raw IMU sensor data and natural language descriptions?
• RQ2 How well can a sensor-aware LLM perform on open-ended question-answering tasks involving motion analysis compared to general-purpose LLMs?
• RQ3 To what extent can narrated IMU data improve the quality and specificity of instruction-following capabilities in multimodal agents?
• RQ4 What are the limitations of current sensor-aware LLMs in handling mathematical reasoning and sensor-specific details?
• RQ5 Can a multimodal agent trained on synthetic narrations and QA pairs generalize to real-world activity understanding tasks?

Key findings

• LLaSA outperformed Vicuna-13b in reasoning about activity phases, such as distinguishing stance and swing phases during stair climbing, by providing data-grounded interpretations.
• The model demonstrated strong performance in closed-ended activity classification, achieving state-of-the-art results on the benchmark dataset.
• LLaSA produced more specific and contextually relevant answers than general LLMs, which often gave generic or unrelated responses.
• Despite its strengths, LLaSA occasionally misclassified activities and struggled with mathematical problems involving sensor data, indicating a need for improved numerical reasoning.
• GPT-4o assessment confirmed that LLaSA’s responses were more scientifically and narratively grounded than those of baseline models, though hallucinations and generalization errors were observed.
• The study revealed that sensor embeddings alone are insufficient for mathematical reasoning, suggesting the need for explicit numerical input augmentation in future models.