[Paper Review] Cohesion-based Online Actor-Critic Reinforcement Learning for mHealth Intervention

Feiyun Zhu, Peng Liao+3 more|arXiv (Cornell University)|Mar 25, 2017

Impact of Technology on Adolescents8 references4 citations

TL;DR

This paper proposes a novel cohesion-based online actor-critic reinforcement learning framework for mobile health (mHealth) interventions, leveraging network cohesion among users to improve policy learning in data-scarce environments. By learning user similarity networks from warm-start trajectories and enforcing ℓ₂ constraints on value and policy functions, the method significantly reduces variance and outperforms separate policy learning across multiple mHealth benchmarks.

ABSTRACT

In the wake of the vast population of smart device users worldwide, mobile health (mHealth) technologies are hopeful to generate positive and wide influence on people's health. They are able to provide flexible, affordable and portable health guides to device users. Current online decision-making methods for mHealth assume that the users are completely heterogeneous. They share no information among users and learn a separate policy for each user. However, data for each user is very limited in size to support the separate online learning, leading to unstable policies that contain lots of variances. Besides, we find the truth that a user may be similar with some, but not all, users, and connected users tend to have similar behaviors. In this paper, we propose a network cohesion constrained (actor-critic) Reinforcement Learning (RL) method for mHealth. The goal is to explore how to share information among similar users to better convert the limited user information into sharper learned policies. To the best of our knowledge, this is the first online actor-critic RL for mHealth and first network cohesion constrained (actor-critic) RL method in all applications. The network cohesion is important to derive effective policies. We come up with a novel method to learn the network by using the warm start trajectory, which directly reflects the users' property. The optimization of our model is difficult and very different from the general supervised learning due to the indirect observation of values. As a contribution, we propose two algorithms for the proposed online RLs. Apart from mHealth, the proposed methods can be easily applied or adapted to other health-related tasks. Extensive experiment results on the HeartSteps dataset demonstrates that in a variety of parameter settings, the proposed two methods obtain obvious improvements over the state-of-the-art methods.

Motivation & Objective

• To address the instability of separate policy learning in mHealth due to limited user-specific data.
• To model user similarity through network cohesion, enabling information sharing among similar users.
• To develop an online actor-critic RL framework that learns the cohesion network directly from data, rather than assuming it a priori.
• To reduce variance in policy learning without introducing excessive bias by constraining value and policy functions via network structure.
• To demonstrate the method's superiority over state-of-the-art approaches on real-world mHealth data.

Proposed method

• Proposes a network cohesion-constrained actor-critic RL framework that shares information across users with similar behaviors.
• Uses warm-start trajectories (WST) to learn the user network structure, avoiding reliance on pre-specified social networks.
• Implements a projection step with ℓ₂ regularization on the value function using a graph Laplacian to enforce cohesion.
• Applies a fixed-point update with a cohesion constraint on the policy function, using a parameterized ℓ₂ penalty.
• Introduces two algorithms: Cohesion-RL#1 and Cohesion-RL#2, differing in how the ℓ₂ constraints are applied during value and policy updates.
• Optimizes the model via iterative updates that jointly refine value functions, policies, and the cohesion network using indirect reward signals.

Experimental results

Research questions

• RQ1 Can information sharing among similar users improve policy learning in data-scarce mHealth settings?
• RQ2 How can a user network be learned directly from user behavior data rather than relying on external social network assumptions?
• RQ3 What is the impact of different cohesion constraint strengths on policy performance and stability?
• RQ4 How does the length of warm-start trajectories affect the learning of network cohesion and overall policy quality?
• RQ5 Can cohesion-based RL outperform separate policy learning in online mHealth intervention tasks?

Key findings

• Cohesion-RL#2 consistently outperforms Cohesion-RL#1 and Separate-RL across all parameter settings, demonstrating superior stability and performance.
• When warm-start trajectories are short (T₀ = 5), Cohesion-RL methods reduce average steps from 261.67 to 52.09, showing strong benefit from network cohesion.
• Increasing T₀ from 5 to 20 improves Separate-RL performance by 210.51 steps, highlighting the critical role of initial data quality.
• The method achieves significant performance gains even with limited data, proving effective in real-world mHealth scenarios with sparse user data.
• The network cohesion constraint with μ₁ ∈ [0.001, 10] improves performance over baseline, with μ₁ = 1.0 yielding optimal results.
• The proposed method is robust to parameter variations and maintains high performance across different discount factors (γ).