[Paper Review] Learning Deep Hidden Nonlinear Dynamics from Aggregate Data

Yisen Wang, Bo Dai+4 more|arXiv (Cornell University)|Jul 22, 2018

Generative Adversarial Networks and Image Synthesis24 references3 citations

TL;DR

This paper proposes LEGEND, a deep generative model that learns nonlinear hidden dynamics from aggregate data using Wasserstein distance for likelihood-free and posterior-free inference. By modeling complex diffusion processes through a latent stochastic process and optimizing via distributional alignment, LEGEND achieves state-of-the-art performance in filtering and smoothing tasks on synthetic and real-world datasets, with theoretical convergence and generalization guarantees.

ABSTRACT

Learning nonlinear dynamics from diffusion data is a challenging problem since the individuals observed may be different at different time points, generally following an aggregate behaviour. Existing work cannot handle the tasks well since they model such dynamics either directly on observations or enforce the availability of complete longitudinal individual-level trajectories. However, in most of the practical applications, these requirements are unrealistic: the evolving dynamics may be too complex to be modeled directly on observations, and individual-level trajectories may not be available due to technical limitations, experimental costs and/or privacy issues. To address these challenges, we formulate a model of diffusion dynamics as the {\em hidden stochastic process} via the introduction of hidden variables for flexibility, and learn the hidden dynamics directly on {\em aggregate observations} without any requirement for individual-level trajectories. We propose a dynamic generative model with Wasserstein distance for LEarninG dEep hidden Nonlinear Dynamics (LEGEND) and prove its theoretical guarantees as well. Experiments on a range of synthetic and real-world datasets illustrate that LEGEND has very strong performance compared to state-of-the-art baselines.

Motivation & Objective

• To address the challenge of learning complex nonlinear diffusion dynamics when individual-level longitudinal trajectories are unavailable.
• To model hidden continuous stochastic processes that govern evolving aggregate observations without requiring full individual tracking.
• To develop a likelihood-free and posterior-free learning framework for dynamic parameter estimation and hidden state inference.
• To provide theoretical generalization and convergence bounds for the proposed method.
• To empirically validate the framework on synthetic and real-world datasets, including cell differentiation and bird migration.

Proposed method

• Formulate the underlying dynamics as a continuous stochastic differential equation (SDE) in a hidden state space $X_t$.
• Model observations $Y_t$ as a nonlinear function $\mathbf{f}(X_t)$ of the hidden states, enabling complex observation mappings.
• Use a deep neural network (e.g., RNN with LSTM) to parameterize the drift function $\mathbf{g}(X_t)$ and observation function $\mathbf{f}(X_t)$.
• Optimize model parameters via Wasserstein distance between predicted and true marginal distributions of observations.
• Apply a dynamic generative model that jointly learns the hidden dynamics and observation mapping using distributional matching.
• Bypass likelihood-based estimation and posterior inference by using distributional discrepancy as the training objective.

Experimental results

Research questions

• RQ1 Can a deep generative model learn complex nonlinear dynamics from aggregate observations without individual-level trajectories?
• RQ2 Can Wasserstein distance serve as an effective surrogate for likelihood-based training in hidden dynamic systems?
• RQ3 Does the proposed framework achieve better performance than state-of-the-art baselines in filtering and smoothing tasks?
• RQ4 What theoretical guarantees can be provided for generalization and convergence in this likelihood-free, posterior-free setting?
• RQ5 Can the model capture both marginal distributions and correlation structures in multivariate dynamics?

Key findings

• LEGEND achieved the lowest Wasserstein error on both filtering (D7) and smoothing (D4) tasks for gene expression data, outperforming all baselines.
• The predicted marginal distributions of Krt8 and Krt18 closely matched the true distributions, as shown in Figure 3, indicating accurate modeling of dynamic evolution.
• LEGEND effectively captured the correlation structure between Krt8 and Krt18 over time, as demonstrated in Figure 4, with predictions closer to the ground truth than baselines.
• On bird migration data, LEGEND achieved the lowest Wasserstein error in both filtering (June) and smoothing (April) tasks, proving robustness on real-world complex dynamics.
• Theoretical analysis established generalization and convergence bounds, providing the first such guarantees for this class of likelihood-free, posterior-free dynamic learning frameworks.
• The model demonstrated strong performance even when dynamics were highly nonlinear and affected by irregular environmental factors, such as in bird migration.