[Paper Review] Rapidly Personalizing Mobile Health Treatment Policies with Limited Data

Sabina Tomkins, Peng Liao+3 more|arXiv (Cornell University)|Feb 23, 2020

Mobile Health and mHealth Applications33 references4 citations

TL;DR

This paper introduces IntelligentPooling, a Thompson Sampling-based reinforcement learning algorithm that adaptively pools data across users using a hierarchical Bayesian Gaussian process model to accelerate personalization in mobile health (mHealth) settings with limited, noisy data. It achieves 26% lower regret than state-of-the-art methods in simulations and demonstrates practical personalization in a live clinical trial with minimal user data.

ABSTRACT

In mobile health (mHealth), reinforcement learning algorithms that adapt to one's context without learning personalized policies might fail to distinguish between the needs of individuals. Yet the high amount of noise due to the in situ delivery of mHealth interventions can cripple the ability of an algorithm to learn when given access to only a single user's data, making personalization challenging. We present IntelligentPooling, which learns personalized policies via an adaptive, principled use of other users' data. We show that IntelligentPooling achieves an average of 26% lower regret than state-of-the-art across all generative models. Additionally, we inspect the behavior of this approach in a live clinical trial, demonstrating its ability to learn from even a small group of users.

Motivation & Objective

• To address the challenge of learning effective, personalized treatment policies in mobile health (mHealth) when data per user is scarce and noisy.
• To balance the trade-off between personalization (high variance) and pooling (potential bias) in reinforcement learning for mHealth.
• To develop a method that dynamically adjusts the degree of personalization based on observed treatment effect variance across and within users.
• To validate the approach in both simulated environments derived from real mHealth data and in a live clinical trial.

Proposed method

• IntelligentPooling uses a mixed-effects Gaussian process (GP) model to jointly model treatment effects across users, incorporating both population-level and individual-level random effects.
• The algorithm employs empirical Bayes to update hyper-parameters of the hierarchical prior, enabling adaptive pooling that evolves with observed data.
• Thompson Sampling is used to select actions, where posterior samples from the GP model inform treatment decisions based on contextual features.
• The model includes user-specific random effects on the intercept and treatment effect terms, allowing for individualized policy learning.
• Features such as activity level, location, engagement, and dosage are used as context to inform treatment decisions.
• The approach learns from all available data up to each decision point, including data from prior and concurrent users, enabling real-time adaptation.

Experimental results

Research questions

• RQ1 Can a reinforcement learning algorithm achieve faster and more accurate personalization in mHealth when individual data is limited and noisy?
• RQ2 How can data from multiple users be pooled in a principled way to improve learning speed without sacrificing personalization accuracy?
• RQ3 To what extent does the algorithm adapt its personalization level based on observed variance in treatment effects across and within users?
• RQ4 Can the proposed method outperform both fully personalized and fully pooled baselines in terms of regret and policy quality?
• RQ5 Is the algorithm viable and effective in a real-world, live clinical trial setting with minimal user data?

Key findings

• IntelligentPooling achieved an average of 26% lower regret than state-of-the-art methods across all generative models in simulation.
• The algorithm demonstrated stronger personalization over time, with a growing difference in treatment probabilities between IntelligentPooling and a pooled baseline (Pearson correlation r = .56, p < .1).
• The posterior mean of the treatment effect coefficient for user B showed a positive effect, while user A showed no evidence of benefit, indicating effective personalization.
• Speed of learning was significantly faster than a person-specific model, as shown by the faster divergence of IntelligentPooling’s posterior from the prior compared to the person-specific approach.
• In the live clinical trial, IntelligentPooling successfully adapted to individual user responses using only a small number of users, showing practical feasibility.
• The method effectively balanced personalization and pooling by adjusting the degree of data sharing based on observed variance in treatment effects.