[Paper Review] Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning

Xuefeng Peng, Yi Ding+7 more|arXiv (Cornell University)|Jan 15, 2019

Sepsis Diagnosis and Treatment20 references45 citations

TL;DR

The paper proposes a mixture-of-experts framework that combines kernel-based and deep reinforcement learning to personalize sepsis treatment (IV fluids and vasopressors) using a recurrent history representation and off-policy evaluation, outperforming clinician, kernel-only, and DRL-only policies in a large retrospective cohort.

ABSTRACT

Sepsis is the leading cause of mortality in the ICU. It is challenging to manage because individual patients respond differently to treatment. Thus, tailoring treatment to the individual patient is essential for the best outcomes. In this paper, we take steps toward this goal by applying a mixture-of-experts framework to personalize sepsis treatment. The mixture model selectively alternates between neighbor-based (kernel) and deep reinforcement learning (DRL) experts depending on patient's current history. On a large retrospective cohort, this mixture-based approach outperforms physician, kernel only, and DRL-only experts.

Motivation & Objective

• Motivate personalized sepsis management due to heterogeneous patient responses.
• Develop a framework that combines two RL paradigms to leverage their strengths.
• Encode full patient history with a recurrent autoencoder to inform decisions.
• Incorporate safeguards to keep proposed actions clinically plausible.
• Evaluate policies with off-policy methods and compare to clinicians and single-expert policies.

Proposed method

• Use an LSTM autoencoder to encode the patient’s entire history into a 128-dimensional state.
• Define a per-transition reward based on changes in predicted mortality log-odds to guide RL training.
• Apply off-policy evaluation using the weighted doubly robust (WDR) estimator to assess policies from retrospective data.
• Derive two expert policies: kernel-based nearest-neighbors over surviving patients and a double-DQN with dueling architecture (DRL).
• Restrict the DRL to actions taken by clinicians at least 1% of the time among similar neighbors to improve safety.
• Combine experts with a mixture-of-experts (MoE) gating function that selects between kernel and DRL policies based on context, using a linear gating function and sigmoid for probabilities.

Experimental results

Research questions

• RQ1 Can a mixture of kernel-based and deep RL policies improve sepsis treatment decisions beyond clinician or single-expert policies?
• RQ2 Does incorporating full patient history via a recurrent encoding improve policy performance?
• RQ3 How does the MoE gating strategy affect policy choice and outcomes across patient states?
• RQ4 Is off-policy evaluation (WDR) reliable for comparing these policies on retrospective data?

Key findings

• MoE policy outperforms physician, kernel-only, and DRL-only policies on the test set under the WDR objective.
• Recurrent encoding of patient history improves policy performance for all RL policies compared to non-recurrent encoding.
• Kernel policy tends to be more conservative, while DRL (DQN) suggests more aggressive fluid and vasopressor use; MoE balances these tendencies.
• With non-recurrent encoding, MoE achieves 4.31 (physician V) vs 3.76 (physician) in baseline; with recurrent encoding, MoE reaches 5.72, outperforming all other policies.
• Off-policy evaluation shows that 90% of importance weights are non-zero for MoE, indicating broad data usage, though, as noted, IS-based estimators can have high variance in healthcare settings.
• Training and evaluation times indicate the MoE can compute recommendations at test time in approximately 0.16 seconds per patient trajectory.