[논문 리뷰] Improving Sepsis Treatment Strategies by Combining Deep and Kernel-Based Reinforcement Learning

Xuefeng Peng, Yi Ding외 7명|arXiv (Cornell University)|2019. 01. 15.

Sepsis Diagnosis and Treatment참고 문헌 20인용 수 45

한 줄 요약

논문은 kernel-based와 deep reinforcement learning을 결합한 mixture-of-experts 프레임워크를 제안하여 재발성 데이터에서 재발성 역사 표현과 off-policy 평가를 사용해 패혈증 치료(IV fluids 및 vasopressors)를 개인화하고 임상의, 커널 전용, DRL 전용 정책을 능가한다.

ABSTRACT

Sepsis is the leading cause of mortality in the ICU. It is challenging to manage because individual patients respond differently to treatment. Thus, tailoring treatment to the individual patient is essential for the best outcomes. In this paper, we take steps toward this goal by applying a mixture-of-experts framework to personalize sepsis treatment. The mixture model selectively alternates between neighbor-based (kernel) and deep reinforcement learning (DRL) experts depending on patient's current history. On a large retrospective cohort, this mixture-based approach outperforms physician, kernel only, and DRL-only experts.

연구 동기 및 목표

• Motivate personalized sepsis management due to heterogeneous patient responses.
• Develop a framework that combines two RL paradigms to leverage their strengths.
• Encode full patient history with a recurrent autoencoder to inform decisions.
• Incorporate safeguards to keep proposed actions clinically plausible.
• Evaluate policies with off-policy methods and compare to clinicians and single-expert policies.

제안 방법

• Use an LSTM autoencoder to encode the patient’s entire history into a 128-dimensional state.
• Define a per-transition reward based on changes in predicted mortality log-odds to guide RL training.
• Apply off-policy evaluation using the weighted doubly robust (WDR) estimator to assess policies from retrospective data.
• Derive two expert policies: kernel-based nearest-neighbors over surviving patients and a double-DQN with dueling architecture (DRL).
• Restrict the DRL to actions taken by clinicians at least 1% of the time among similar neighbors to improve safety.
• Combine experts with a mixture-of-experts (MoE) gating function that selects between kernel and DRL policies based on context, using a linear gating function and sigmoid for probabilities.

실험 결과

연구 질문

• RQ1 Can a mixture of kernel-based and deep RL policies improve sepsis treatment decisions beyond clinician or single-expert policies?
• RQ2 Does incorporating full patient history via a recurrent encoding improve policy performance?
• RQ3 How does the MoE gating strategy affect policy choice and outcomes across patient states?
• RQ4 Is off-policy evaluation (WDR) reliable for comparing these policies on retrospective data?

주요 결과

• MoE policy outperforms physician, kernel-only, and DRL-only policies on the test set under the WDR objective.
• Recurrent encoding of patient history improves policy performance for all RL policies compared to non-recurrent encoding.
• Kernel policy tends to be more conservative, while DRL (DQN) suggests more aggressive fluid and vasopressor use; MoE balances these tendencies.
• With non-recurrent encoding, MoE achieves 4.31 (physician V) vs 3.76 (physician) in baseline; with recurrent encoding, MoE reaches 5.72, outperforming all other policies.
• Off-policy evaluation shows that 90% of importance weights are non-zero for MoE, indicating broad data usage, though, as noted, IS-based estimators can have high variance in healthcare settings.
• Training and evaluation times indicate the MoE can compute recommendations at test time in approximately 0.16 seconds per patient trajectory.