[Paper Review] Synthetic data generation for a longitudinal cohort study -- Evaluation, method extension and reproduction of published data analysis results

Lisa Kühnel, Julian Schneider+6 more|arXiv (Cornell University)|May 12, 2023

demographic modeling and climate adaptationDecision Sciences3 citations

TL;DR

This study develops and evaluates VAMBN-MT, an LSTM-enhanced variational autoencoder for generating high-fidelity synthetic longitudinal health data from the DONALD cohort study. It demonstrates that with expert-informed model extensions and sufficient sample size, synthetic data can successfully reproduce real-world trends in sugar intake, achieving 100% statistical significance in large synthetic datasets where smaller ones show variable results.

ABSTRACT

Access to individual-level health data is essential for gaining new insights and advancing science. In particular, modern methods based on artificial intelligence rely on the availability of and access to large datasets. In the health sector, access to individual-level data is often challenging due to privacy concerns. A promising alternative is the generation of fully synthetic data, i.e. data generated through a randomised process that have similar statistical properties as the original data, but do not have a one-to-one correspondence with the original individual-level records. In this study, we use a state-of-the-art synthetic data generation method and perform in-depth quality analyses of the generated data for a specific use case in the field of nutrition. We demonstrate the need for careful analyses of synthetic data that go beyond descriptive statistics and provide valuable insights into how to realise the full potential of synthetic datasets. By extending the methods, but also by thoroughly analysing the effects of sampling from a trained model, we are able to largely reproduce significant real-world analysis results in the chosen use case.

Motivation & Objective

• To evaluate and extend state-of-the-art synthetic data generation methods for longitudinal cohort studies in nutrition.
• To address limitations in existing methods that fail to preserve direct dependencies across time points in longitudinal data.
• To reproduce published real-world analysis results—specifically time and age trends in added sugar intake—using synthetic data.
• To investigate the impact of sampling variability, sample size, and variable grouping on the reliability and utility of synthetic datasets.
• To provide a methodological framework for generating synthetic data that balances privacy, utility, and reproducibility in epidemiological research.

Proposed method

• Adapt the Variational Autoencoder with Multivariate Normalizing Flows (VAMBN) by integrating an LSTM layer to model temporal dependencies across repeated visits in longitudinal data.
• Reframe the data so that all time points for a single individual are encoded together into a single latent representation, preserving inter-visit dependencies.
• Train the model using a hierarchical VAE architecture with shared latent space across time points, enabling joint modeling of correlated variables over time.
• Use a feedforward network as baseline for comparison (VAMBN-Flattened Time points, FT), contrasting it with the LSTM-enhanced version (VAMBN-MT).
• Generate synthetic datasets of varying sizes (1,312 and 10,000 samples) and apply post-processing to assess stability and reproducibility.
• Conduct systematic resampling (100 iterations per model) to evaluate variance, confidence interval width, and statistical significance across repeated synthetic datasets.

Experimental results

Research questions

• RQ1 Can synthetic data generated by VAMBN-MT reproduce the age and time trends in added sugar intake observed in the original DONALD study data?
• RQ2 How does sample size affect the stability and statistical significance of trend detection in synthetic datasets compared to real data?
• RQ3 To what extent does incorporating an LSTM layer improve the modeling of direct dependencies across time points compared to standard feedforward networks?
• RQ4 How does the selection of variables trained together (e.g., grouped by domain or visit) affect the preservation of correlations and downstream analysis validity?
• RQ5 Can synthetic data reliably reproduce non-significant trends from real data, and what does this imply for the utility of synthetic data in detecting subtle effects?

Key findings

• VAMBN-MT significantly outperformed both the baseline VAMBN-FT and standard VAMBN in preserving pairwise correlations and direct dependencies, such as the linear relationship between time and age.
• The model successfully reproduced the age trend in added sugar intake with high fidelity and approximated the time trend well, demonstrating its utility for real-world analysis reproduction.
• Larger synthetic datasets (10,000 samples) produced more stable trend estimates with narrower confidence intervals and 100% statistical significance, compared to smaller datasets (1,312 samples) which showed significance in only 22–70% of cases.
• Sampling variability had a measurable impact: repeated sampling from the same model produced varying results, highlighting the need for large sample sizes to ensure reliable inference.
• The model failed to reproduce the non-significant linear and quadratic time trends in total sugar intake, indicating that synthetic data cannot reliably replicate non-significant real-world findings.
• Proper variable grouping—especially training related variables (e.g., maternal education) together—was essential to preserve correlations, as separate autoencoders led to loss of key relationships.