[Paper Review] Evaluating Reinforcement Learning Algorithms in Observational Health Settings

Omer Gottesman, Fredrik Johansson+18 more|arXiv (Cornell University)|May 31, 2018

Machine Learning in HealthcareComputer Science13 references86 citations

TL;DR

This paper analyzes the challenges of evaluating reinforcement learning policies using observational health data, highlighting confounding, representation, and off-policy evaluation issues in sepsis management, and offers best-practice recommendations.

ABSTRACT

Much attention has been devoted recently to the development of machine learning algorithms with the goal of improving treatment policies in healthcare. Reinforcement learning (RL) is a sub-field within machine learning that is concerned with learning how to make sequences of decisions so as to optimize long-term effects. Already, RL algorithms have been proposed to identify decision-making strategies for mechanical ventilation, sepsis management and treatment of schizophrenia. However, before implementing treatment policies learned by black-box algorithms in high-stakes clinical decision problems, special care must be taken in the evaluation of these policies. In this document, our goal is to expose some of the subtleties associated with evaluating RL algorithms in healthcare. We aim to provide a conceptual starting point for clinical and computational researchers to ask the right questions when designing and evaluating algorithms for new ways of treating patients. In the following, we describe how choices about how to summarize a history, variance of statistical estimators, and confounders in more ad-hoc measures can result in unreliable, even misleading estimates of the quality of a treatment policy. We also provide suggestions for mitigating these effects---for while there is much promise for mining observational health data to uncover better treatment policies, evaluation must be performed thoughtfully.

Motivation & Objective

• Motivate careful evaluation of RL policies in healthcare, especially in observational settings where patient lives are not experiments.
• Illustrate how history representation and confounding affect policy estimates.
• Discuss limitations of off-policy evaluation methods and ad-hoc metrics in healthcare RL.
• Provide practical recommendations to mitigate bias and variance in policy evaluation.

Proposed method

• Formalize sepsis management as a RL problem with state, action, and reward definitions from MIMIC III data.
• Demonstrate how state representation choices affect confounding and policy quality.
• Apply off-policy evaluation methods (importance sampling: PDIS, WPDIS, DR, WDR) to retrospective data.
• Compare model-based and IS-based value estimates to assess policy performance.
• Analyze the impact of deterministic vs. stochastic policies on evaluation bias and variance.
• Provide diagnostics on the distribution of importance weights and effective sample size.

Experimental results

Research questions

• RQ1 How do representation choices for patient history influence confounding and the reliability of learned policies?
• RQ2 What are the limitations of off-policy evaluation methods for sequential healthcare decisions in sepsis management?
• RQ3 How do deterministic action policies affect the variance and bias of IS estimators in observational data?
• RQ4 What best practices can mitigate biases when evaluating RL policies with retrospective health data?

Key findings

• Deterministic policies often lead to high-variance IS estimates due to sparse outcomes and few matching trajectories.
• Model-based value estimates are biased but have lower variance than IS estimators in this setting.
• Weighted IS (WDR, WPDIS) reduces variance but introduces bias; unweighted IS shows extremely high variance.
• The effective sample size for evaluating learned policies using IS can be very small, questioning reliability.
• Ad-hoc U-curve analyses can mislead due to confounding and action-bin artifacts; interpretability and clinician input are essential.
• Evaluating policies that are closer to physician practice increases evaluability and reliability of conclusions.