[Paper Review] From Classification to Clinical Insights: Towards Analyzing and Reasoning About Mobile and Behavioral Health Data With Large Language Models

Zachary Englhardt, Chengqian Ma+8 more|arXiv (Cornell University)|Nov 21, 2023

Digital Mental Health Interventions57 references4 citations

TL;DR

This paper proposes using large language models (LLMs) to generate clinically meaningful insights from mobile and behavioral health data, moving beyond binary classification to interactive, human-AI collaboration. By applying chain-of-thought prompting to multi-sensor data (e.g., step count, sleep), LLMs achieve 61.1% accuracy in depression classification—surpassing prior work—and clinicians express strong interest in using AI-generated reasoning to collaboratively explore patient data with patients, enhancing therapeutic alliance.

ABSTRACT

Passively collected behavioral health data from ubiquitous sensors holds significant promise to provide mental health professionals insights from patient's daily lives; however, developing analysis tools to use this data in clinical practice requires addressing challenges of generalization across devices and weak or ambiguous correlations between the measured signals and an individual's mental health. To address these challenges, we take a novel approach that leverages large language models (LLMs) to synthesize clinically useful insights from multi-sensor data. We develop chain of thought prompting methods that use LLMs to generate reasoning about how trends in data such as step count and sleep relate to conditions like depression and anxiety. We first demonstrate binary depression classification with LLMs achieving accuracies of 61.1% which exceed the state of the art. While it is not robust for clinical use, this leads us to our key finding: even more impactful and valued than classification is a new human-AI collaboration approach in which clinician experts interactively query these tools and combine their domain expertise and context about the patient with AI generated reasoning to support clinical decision-making. We find models like GPT-4 correctly reference numerical data 75% of the time, and clinician participants express strong interest in using this approach to interpret self-tracking data.

Motivation & Objective

• To address the limitations of traditional machine learning in analyzing passive, multi-sensor behavioral health data, which often suffer from poor generalization across devices and weak signal-mental health correlations.
• To explore whether large language models (LLMs) can generate clinically useful, interpretable reasoning from self-tracking data, moving beyond binary classification toward actionable insights.
• To evaluate the feasibility and perceived value of a human-AI collaborative model where clinicians and patients jointly query LLMs to interpret behavioral trends in context.
• To identify challenges in deploying LLMs in clinical mental health settings, including data ownership, model reliability, and risks of over-reliance or simulated therapy.

Proposed method

• Employing chain-of-thought prompting to guide LLMs in reasoning about relationships between trends in multi-sensor data (e.g., step count, sleep duration) and mental health conditions like depression and anxiety.
• Using LLMs to perform binary depression classification on self-tracking data, achieving 61.1% accuracy—higher than state-of-the-art benchmarks.
• Conducting qualitative interviews with mental health clinicians to assess their perceptions of AI-generated reasoning and their willingness to integrate such tools into clinical workflows.
• Designing a collaborative interaction model where both clinicians and patients can query the LLM, with clinicians combining AI insights with clinical context and patient history.
• Evaluating LLMs' numerical accuracy by measuring how often they correctly reference input data values (75% accuracy for GPT-4 in this study).
• Exploring architectural and deployment considerations, including on-device inference and secure data sharing models inspired by Apple and Google’s privacy-first approaches.

Experimental results

Research questions

• RQ1 Can LLMs generate clinically relevant, interpretable reasoning from passive mobile and wearable sensor data, beyond simple classification?
• RQ2 How do clinicians perceive the value of AI-generated insights when interpreting patient self-tracking data in a therapeutic context?
• RQ3 To what extent can LLMs correctly reference and reason about numerical trends in behavioral data, and how reliable is this reasoning in clinical settings?
• RQ4 What are the risks and challenges of integrating LLMs into mental health practice, particularly regarding over-reliance, data ownership, and simulated therapy?

Key findings

• LLMs achieved 61.1% accuracy in binary depression classification using mobile and behavioral health data, exceeding the state of the art.
• Clinicians expressed strong interest in using LLMs to collaboratively explore patient data with patients, viewing this as a way to strengthen the therapeutic alliance.
• GPT-4 correctly referenced numerical data inputs 75% of the time, indicating a notable level of factual consistency in reasoning.
• The primary clinical value lies not in classification, but in AI-generated reasoning that supports clinician-patient dialogue and shared interpretation of behavioral trends.
• Clinicians expressed concern that over-reliance on LLMs could lead to generic, one-size-fits-all interpretations and risk simulating therapy, undermining clinical judgment.
• The study highlights the need for secure, privacy-preserving data architectures that allow both patients and clinicians to actively interact with and contribute to AI-driven insights.