[Paper Review] An Optimal Policy for Patient Laboratory Tests in Intensive Care Units

Lifang Cheng, Niranjani Prasad+1 more|arXiv (Cornell University)|Aug 14, 2018

Hemodynamic Monitoring and Therapy7 references4 citations

TL;DR

This paper proposes a batch off-policy reinforcement learning framework that learns an optimal laboratory test ordering policy for ICU patients by balancing clinical utility, cost, and risk. Using multi-objective Q-iteration with a composite reward function, the method reduces test frequency by 44% (e.g., WBC) and advances test timing by over 4 hours on average, improving information gain and enabling earlier treatment initiation.

ABSTRACT

Laboratory testing is an integral tool in the management of patient care in hospitals, particularly in intensive care units (ICUs). There exists an inherent trade-off in the selection and timing of lab tests between considerations of the expected utility in clinical decision-making of a given test at a specific time, and the associated cost or risk it poses to the patient. In this work, we introduce a framework that learns policies for ordering lab tests which optimizes for this trade-off. Our approach uses batch off-policy reinforcement learning with a composite reward function based on clinical imperatives, applied to data that include examples of clinicians ordering labs for patients. To this end, we develop and extend principles of Pareto optimality to improve the selection of actions based on multiple reward function components while respecting typical procedural considerations and prioritization of clinical goals in the ICU. Our experiments show that we can estimate a policy that reduces the frequency of lab tests and optimizes timing to minimize information redundancy. We also find that the estimated policies typically suggest ordering lab tests well ahead of critical onsets--such as mechanical ventilation or dialysis--that depend on the lab results. We evaluate our approach by quantifying how these policies may initiate earlier onset of treatment.

Motivation & Objective

• To address the clinical problem of redundant and costly lab test ordering in ICUs, which contributes to patient harm and increased healthcare costs.
• To develop a data-driven, interpretable policy that optimizes lab test timing and selection by balancing information gain, cost, and clinical risk.
• To improve clinical decision-making by learning from real-world clinician ordering patterns while minimizing information redundancy.
• To evaluate whether learned policies can recommend tests earlier than clinicians, thereby enabling earlier treatment initiation.
• To provide a deployable, low-risk decision support system for clinician-in-the-loop ICU care using off-policy RL.

Proposed method

• The framework models patient trajectories as a Markov decision process (MDP), with state representations derived from sparse, irregularly sampled time series using multi-output Gaussian processes (MOGPs).
• A composite reward function integrates multiple clinical objectives: expected information gain, cost of testing, and impact on downstream treatment timing.
• The method employs batch off-policy reinforcement learning via multi-objective fitted Q-iteration (MO-FQI) to learn policies from historical clinician ordering data.
• Pareto optimality principles are extended to select actions that balance competing objectives across multiple reward components without requiring scalarization.
• Off-policy evaluation uses weighted importance sampling (WIS) estimators to evaluate policy performance on test data without deployment.
• The model estimates future patient states and forecasts lab values with uncertainty quantification, enabling proactive test recommendations.

Experimental results

Research questions

• RQ1 Can a reinforcement learning framework learn a lab test ordering policy that reduces redundant testing while maintaining or improving clinical utility?
• RQ2 To what extent can the learned policy recommend lab tests earlier than clinicians, thereby enabling earlier treatment initiation?
• RQ3 How does the information gain of lab tests recommended by the policy compare to those ordered by clinicians?
• RQ4 Can the framework balance competing clinical objectives—such as cost, risk, and diagnostic utility—through a composite reward function?
• RQ5 Does the policy reduce the total number of lab tests compared to current clinical practice, particularly for high-frequency tests like WBC and lactate?

Key findings

• The MO-FQI policy reduced the total number of lab test recommendations by 44% compared to clinician-ordered practices, with 12,358 recommendations for WBC versus 22,172 actual orders.
• For lactate, which is ordered less frequently, the policy still achieved a 27% reduction in test frequency.
• The average time between lab test recommendation and critical treatment onset increased from 9.1 to 13.2 hours for WBC, indicating earlier detection and intervention potential.
• The mean information gain of lab tests recommended by the policy was significantly higher than that of clinician-ordered tests: 1.53 vs. 0.69 for WBC, and 3.39 vs. 1.63 for BUN.
• The policy consistently outperformed clinicians in all three evaluation metrics—reduction in orders, increased information gain, and earlier treatment timing—across all four tested labs.
• The policy recommended lab tests up to 4.5 hours earlier on average than clinicians, particularly for WBC, creatinine, and BUN, suggesting improved anticipation of clinical deterioration.