[Paper Review] Latent-state models for precision medicine

Zekun Xu, Eric B. Laber+2 more|arXiv (Cornell University)|May 26, 2020

Machine Learning in Healthcare88 references4 citations

TL;DR

This paper proposes a partially observable Markov decision process (POMDP) model that incorporates latent health states to estimate optimal dynamic treatment regimes in irregular, longitudinal observational data—particularly for chronic mental illnesses like bipolar disorder. By modeling unobserved patient states that influence treatment response, the method enables estimation of personalized, clinically meaningful treatment strategies where traditional methods fail due to lack of temporal alignment or Markov assumptions.

ABSTRACT

Observational longitudinal studies are a common means to study treatment efficacy and safety in chronic mental illness. In many such studies, treatment changes may be initiated by either the patient or by their clinician and can thus vary widely across patients in their timing, number, and type. Indeed, in the observational longitudinal pathway of the STEP-BD study of bipolar depression, one of the motivations for this work, no two patients have the same treatment history even after coarsening clinic visits to a weekly time-scale. Estimation of an optimal treatment regime using such data is challenging as one cannot naively pool together patients with the same treatment history, as is required by methods based on inverse probability weighting, nor is it possible to apply backwards induction over the decision points, as is done in Q-learning and its variants. Thus, additional structure is needed to effectively pool information across patients and within a patient over time. Current scientific theory for many chronic mental illnesses maintains that a patient's disease status can be conceptualized as transitioning among a small number of discrete states. We use this theory to inform the construction of a partially observable Markov decision process model of patient health trajectories wherein observed health outcomes are dictated by a patient's latent health state. Using this model, we derive and evaluate estimators of an optimal treatment regime under two common paradigms for quantifying long-term patient health. The finite sample performance of the proposed estimator is demonstrated through a series of simulation experiments and application to the observational pathway of the STEP-BD study. We find that the proposed method provides high-quality estimates of an optimal treatment strategy in settings where existing approaches cannot be applied without ad hoc modifications.

Motivation & Objective

• To address the challenge of estimating optimal treatment regimes in observational longitudinal studies with irregular, non-Markovian treatment histories and frequent, variable treatment changes.
• To incorporate patient-specific latent health states—such as unobserved mood states in bipolar disorder—into treatment regime estimation to improve clinical relevance and statistical efficiency.
• To develop a method that enables estimation of optimal treatment regimes under infinite-horizon settings where standard inverse probability weighting or Q-learning approaches are inapplicable due to lack of temporal alignment.
• To provide a theoretically grounded, clinically interpretable framework for precision medicine in chronic mental illness using a POMDP model with latent state dynamics.
• To demonstrate the method's performance through simulations and application to the STEP-BD observational pathway, showing improved estimation over existing approaches in complex data settings.

Proposed method

• Models patient health trajectories using a partially observable Markov decision process (POMDP), where observed outcomes depend on unobserved latent health states.
• Assumes that, conditional on the current latent state and observed covariates, the evolution of patient health is Markovian, allowing for sequential decision-making under uncertainty.
• Defines the information state as the joint distribution of the latent state and current patient measurements, which fully determines the optimal treatment regime.
• Derives estimators for the optimal treatment regime using likelihood-based inference and establishes their asymptotic properties under regularity conditions.
• Applies the model to estimate treatment regimes under two utility criteria: average utility and discounted utility (γ = 0.95), using data from the STEP-BD study.
• Uses decision tree visualization to interpret the estimated optimal treatment rules, with splits based on estimated probabilities of depression, mania, or other mood states.

Experimental results

Research questions

• RQ1 Can a latent-state POMDP model effectively estimate optimal treatment regimes in observational data with irregular, non-Markovian treatment histories?
• RQ2 How does incorporating unobserved patient health states improve the quality and clinical interpretability of estimated dynamic treatment regimes?
• RQ3 What is the finite-sample performance of the proposed estimator compared to existing methods in settings where standard approaches fail?
• RQ4 Does the estimated optimal regime outperform the observed clinical practice in terms of long-term patient health outcomes?
• RQ5 How do different utility criteria (average vs. discounted) affect the structure of the estimated optimal treatment regime?

Key findings

• The proposed POMDP-based method successfully estimates an optimal treatment regime in the STEP-BD observational pathway, where standard methods cannot be applied without ad hoc modifications.
• The estimated optimal regime recommends mood stabilizers alone or in combination with antidepressants, avoiding monotherapy with antidepressants—consistent with clinical guidelines due to the risk of inducing mania.
• For average utility, the value of the estimated regime (1.81, 95% CI: 1.72–1.88) significantly exceeds the observed regime (1.66), with the lower bound of the confidence interval higher than the observed value.
• For discounted utility (γ = 0.95), the estimated regime achieves a value of 33.76 (95% CI: 19.24–46.95), substantially higher than the observed value of 14.40.
• The model estimates clinically meaningful state probabilities: for example, 85% probability of depression in the 'Depression' clinical status, and 55% for mania in the 'Mania' state.
• The estimated optimal treatment regime is qualitatively similar under both average and discounted utility, with antidepressants recommended only when depression is likely or mania is unlikely.