[Paper Review] Neural Ordinary Differential Equations for Intervention Modeling

Daehoon Gwak, Gyuhyeon Sim+4 more|arXiv (Cornell University)|Oct 16, 2020

Machine Learning in Healthcare1 references6 citations

TL;DR

This paper proposes IMODE, a novel Neural ODE framework that disentangles autonomous system dynamics from external interventions by using two separate ODEs—one for observation-driven dynamics and another for intervention effects. Experiments on synthetic and real-world medical time-series data show IMODE outperforms existing methods in modeling intervention effects while enabling interpretable disentanglement of dynamics and intervention impacts.

ABSTRACT

By interpreting the forward dynamics of the latent representation of neural networks as an ordinary differential equation, Neural Ordinary Differential Equation (Neural ODE) emerged as an effective framework for modeling a system dynamics in the continuous time domain. However, real-world systems often involves external interventions that cause changes in the system dynamics such as a moving ball coming in contact with another ball, or such as a patient being administered with particular drug. Neural ODE and a number of its recent variants, however, are not suitable for modeling such interventions as they do not properly model the observations and the interventions separately. In this paper, we propose a novel neural ODE-based approach (IMODE) that properly model the effect of external interventions by employing two ODE functions to separately handle the observations and the interventions. Using both synthetic and real-world time-series datasets involving interventions, our experimental results consistently demonstrate the superiority of IMODE compared to existing approaches.

Motivation & Objective

• To address the limitation of existing Neural ODE models in handling external interventions that alter system dynamics.
• To develop a method that separately models the autonomous evolution of a system and the impact of external interventions.
• To enable interpretable and disentangled modeling of intervention effects in continuous-time dynamic systems.
• To improve prediction accuracy on time-series data involving irregular or impactful interventions, such as medical treatments.

Proposed method

• IMODE employs two distinct ODE functions: one models the autonomous dynamics of the system state based on observations, and the other models the effect of external interventions.
• The intervention ODE is parameterized by a neural network that takes intervention inputs and time, and updates the system's dynamics rather than directly modifying the state.
• The latent state is decomposed into two components: $\mathbf{z}_x$ for autonomous dynamics and $\mathbf{z}_a$ for intervention effects, enabling disentanglement.
• The model is trained end-to-end using variational inference, with the system state evolved via numerical integration of the combined ODEs.
• The framework supports both discrete and continuous interventions, including decaying or permanent effects.
• The architecture is adapted for real-world medical data by using sigmoid activation for multi-treatment prediction and removing gradient reversal layers in continuous intervention settings.

Experimental results

Research questions

• RQ1 Can a Neural ODE-based model effectively disentangle autonomous system dynamics from external interventions?
• RQ2 How does IMODE perform compared to existing intervention modeling baselines on synthetic and real-world time-series data?
• RQ3 Does IMODE's disentangled representation allow for more interpretable modeling of intervention effects?
• RQ4 Can IMODE generalize to complex, real-world medical time-series with irregular and multi-modal interventions?

Key findings

• IMODE consistently outperforms ODE-RNN, RNN-Decay, and other baselines in predicting time-series trajectories under interventions on both synthetic and eICU datasets.
• On the eICU dataset, IMODE achieved the most accurate prediction of blood pressure trajectories, including systolic, diastolic, and mean pressure, especially following norepinephrine administration.
• The $L_2$ norm of $\mathbf{z}_x$ remained stable over time, indicating robust modeling of autonomous dynamics, while $||\mathbf{z}_a||_2$ spiked at intervention times and decayed gradually, confirming disentanglement of intervention effects.
• In synthetic experiments, IMODE correctly captured decaying intervention effects and permanent shifts, whereas baselines failed to model intervention impacts accurately.
• The model’s behavior analysis confirmed that intervention effects are learned separately from system dynamics, validating the disentanglement objective.
• For continuous and unpredictable interventions (e.g., in Moving Ball and Exponential Decay), IMODE outperformed CRN when the treatment classifier was disabled, demonstrating superior generalization.