[Paper Review] Machine Learning for Mechanical Ventilation Control

Daniel Suo, Naman Agarwal+13 more|arXiv (Cornell University)|Feb 12, 2021

Respiratory Support and Mechanisms31 references4 citations

TL;DR

This paper proposes a data-driven, differentiable reinforcement learning controller for mechanical ventilation that outperforms standard PID controllers in tracking target pressure waveforms across diverse lung conditions. By training a neural network controller on a learned simulator of ventilator-lung dynamics, the method achieves superior accuracy, robustness, and sample efficiency compared to traditional and baseline RL approaches.

ABSTRACT

We consider the problem of controlling an invasive mechanical ventilator for pressure-controlled ventilation: a controller must let air in and out of a sedated patient's lungs according to a trajectory of airway pressures specified by a clinician. Hand-tuned PID controllers and similar variants have comprised the industry standard for decades, yet can behave poorly by over- or under-shooting their target or oscillating rapidly. We consider a data-driven machine learning approach: First, we train a simulator based on data we collect from an artificial lung. Then, we train deep neural network controllers on these simulators.We show that our controllers are able to track target pressure waveforms significantly better than PID controllers. We further show that a learned controller generalizes across lungs with varying characteristics much more readily than PID controllers do.

Motivation & Objective

• To address the limitations of PID controllers in mechanical ventilation, which often cause overshoot, undershoot, and ringing due to poor adaptability to patient-specific lung dynamics.
• To develop a data-driven control framework that learns from simulated ventilator-lung interactions, reducing reliance on manual tuning and physical experimentation.
• To evaluate and compare model-based and model-free reinforcement learning methods for ventilator control, focusing on sample efficiency and training stability.
• To create open-source, self-contained differentiable simulators to lower the barrier for future research in intelligent ventilator control.
• To demonstrate that learned controllers generalize better across diverse lung characteristics than tuned PID controllers.

Proposed method

• The method employs a differentiable simulator trained on real ventilator data to generate synthetic trajectories of lung pressure under various R and C parameters (resistance and compliance).
• A neural network controller is trained using analytical policy gradient descent, enabling end-to-end backpropagation through the simulator’s dynamics for efficient optimization.
• The controller is trained to minimize the L1 distance between the target pressure waveform and the actual achieved lung pressure at each time step.
• The approach uses a state representation that includes current pressure, target pressure, and time-in-breath, enabling the controller to adapt to dynamic breathing patterns.
• The method is evaluated both in simulation across six ISO-standard lung settings and on a physical test lung, comparing performance against tuned PID and baseline RL algorithms.
• Extensive hyperparameter tuning is performed for baselines (PPO, DQN), while the proposed method requires minimal tuning, relying only on standard learning rate selection.

Experimental results

Research questions

• RQ1 Can a learned neural network controller outperform standard PID controllers in tracking target pressure waveforms across diverse lung characteristics?
• RQ2 How does a differentiable, model-based reinforcement learning approach compare to model-free methods like PPO and DQN in terms of sample efficiency and training stability?
• RQ3 To what extent can a controller trained on a learned simulator generalize to unseen lung settings and perform on a physical ventilator?
• RQ4 Can a single controller trained on multiple lung types outperform the best-tuned PID controller for each individual lung type?
• RQ5 What is the impact of differentiable dynamics on the feasibility of real-time or resource-constrained control system deployment?

Key findings

• The learned controller outperforms the best-tuned PID controller on a physical test lung across all six ISO-standard lung settings, with significantly reduced L1 error in pressure tracking.
• The controller trained on all six lung settings achieved better performance than any individual PID controller tuned per setting, demonstrating superior generalization.
• The analytical policy gradient method converged in under 100 training episodes, while PPO and DQN required tens of thousands of episodes, highlighting a 100x+ sample efficiency gain.
• The proposed method showed stable training and required minimal hyperparameter tuning, unlike PPO and DQN, which needed 720 and 180 hyperparameter trials respectively to reach comparable performance.
• The learned controller achieved faster rise times, minimal overshoot, and no ringing during inspiration, unlike P-only, I-only, and PID controllers that exhibited significant transient errors.
• The differentiable simulator enabled end-to-end training with backpropagation through the dynamics, making the method both sample- and computationally efficient compared to standard RL baselines.