[Paper Review] ASAC: Active Sensing using Actor-Critic models

Jinsung Yoon, James Jordon+1 more|arXiv (Cornell University)|Jun 16, 2019

Explainable Artificial Intelligence (XAI)27 references4 citations

TL;DR

ASAC is a deep reinforcement learning framework that uses actor-critic models to solve the active sensing problem in healthcare, where observations are costly. It jointly trains a selector network to decide which variables to measure and a predictor network to provide feedback via predictive error, minimizing KL divergence between full and selected feature distributions while balancing measurement cost.

ABSTRACT

Deciding what and when to observe is critical when making observations is costly. In a medical setting where observations can be made sequentially, making these observations (or not) should be an active choice. We refer to this as the active sensing problem. In this paper, we propose a novel deep learning framework, which we call ASAC (Active Sensing using Actor-Critic models) to address this problem. ASAC consists of two networks: a selector network and a predictor network. The selector network uses previously selected observations to determine what should be observed in the future. The predictor network uses the observations selected by the selector network to predict a label, providing feedback to the selector network (well-selected variables should be predictive of the label). The goal of the selector network is then to select variables that balance the cost of observing the selected variables with their predictive power; we wish to preserve the conditional label distribution. During training, we use the actor-critic models to allow the loss of the selector to be "back-propagated" through the sampling process. The selector network "acts" by selecting future observations to make. The predictor network acts as a "critic" by feeding predictive errors for the selected variables back to the selector network. In our experiments, we show that ASAC significantly outperforms state-of-the-arts in two real-world medical datasets.

Motivation & Objective

• Address the challenge of costly sequential observations in medical settings, where each measurement incurs financial or patient burden.
• Formulate active sensing as a sequential decision-making problem that balances predictive accuracy and measurement cost.
• Develop a deep learning framework capable of handling both static and time-series data with variable selection over time.
• Ensure the selected features preserve the conditional label distribution, even under noisy or costly measurement conditions.
• Enable personalized, cost-effective screening and monitoring by learning optimal measurement policies for individual patients.

Proposed method

• Use a two-stream neural network architecture: a selector network (actor) that selects future observations based on prior measurements, and a predictor network (critic) that predicts the label using selected features.
• Train the selector using policy gradient methods with the critic's predictive error as a reward signal, enabling backpropagation through stochastic sampling.
• Optimize a loss function combining KL divergence between full and selected feature conditional label distributions and a penalty term for measurement cost.
• Introduce a differentiable sampling mechanism that allows gradient-based optimization despite the discrete nature of observation selection.
• Handle both static and time-series settings by modeling sequential decision-making where future observations depend on past selections.
• Incorporate label-dependent costs by modulating measurement cost based on predicted patient state (e.g., sick vs. well), enabling adaptive prioritization of accuracy.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning framework effectively learn to select informative, low-cost measurements in sequential medical observation settings?
• RQ2 How well does ASAC preserve predictive performance while minimizing the number of measurements compared to state-of-the-art methods?
• RQ3 Can ASAC adaptively prioritize accurate measurements when the patient’s condition deteriorates, even if it increases cost?
• RQ4 How robust is ASAC to noisy or imperfect measurements, especially when cheaper, noisier alternatives are available?
• RQ5 Does ASAC maintain performance under missing data, a common issue in real-world clinical datasets?

Key findings

• ASAC significantly outperforms state-of-the-art methods on two real-world medical datasets, reducing observation costs with minimal loss in predictive accuracy.
• On the MIMIC-III dataset, ASAC demonstrated robustness to missing data, maintaining high performance despite incomplete feature availability.
• In synthetic experiments with noisy features, ASAC correctly prioritized true features as cost increased, selecting noisy versions only when cost was low.
• When measurement cost depended on patient status (e.g., higher cost when sick), ASAC increased use of true features by up to 700% when the patient was ill (η=0.1), showing adaptive behavior.
• At γ=0.6, ASAC selected the true version of X² exclusively and began measuring the true X¹ feature occasionally, indicating effective cost-accuracy trade-off learning.
• ASAC successfully balanced predictive power and cost in all tested scenarios, including high-noise, high-cost, and label-dependent cost regimes.