[Paper Review] CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation

Yusuke Tashiro, Jiaming Song+2 more|arXiv (Cornell University)|Jul 7, 2021

Mental Health Research Topics172 citations

TL;DR

CSDI introduces a conditional score-based diffusion approach for probabilistic time series imputation, outperforming existing methods by leveraging observed values through self-supervised training and attention-based architectures.

ABSTRACT

The imputation of missing values in time series has many applications in healthcare and finance. While autoregressive models are natural candidates for time series imputation, score-based diffusion models have recently outperformed existing counterparts including autoregressive models in many tasks such as image generation and audio synthesis, and would be promising for time series imputation. In this paper, we propose Conditional Score-based Diffusion models for Imputation (CSDI), a novel time series imputation method that utilizes score-based diffusion models conditioned on observed data. Unlike existing score-based approaches, the conditional diffusion model is explicitly trained for imputation and can exploit correlations between observed values. On healthcare and environmental data, CSDI improves by 40-65% over existing probabilistic imputation methods on popular performance metrics. In addition, deterministic imputation by CSDI reduces the error by 5-20% compared to the state-of-the-art deterministic imputation methods. Furthermore, CSDI can also be applied to time series interpolation and probabilistic forecasting, and is competitive with existing baselines. The code is available at https://github.com/ermongroup/CSDI.

Motivation & Objective

• Motivate robust imputation of multivariate time series with missing values in healthcare, environment, and finance.
• Develop a conditional diffusion model that learns the conditional distribution for imputation rather than approximating it.
• Leverage observed data through conditioning and attention mechanisms to capture temporal and feature dependencies.
• Provide a self-supervised training framework to enable learning without ground-truth missing values.
• Demonstrate applicability to imputation, interpolation, and probabilistic forecasting while outperforming baselines.

Proposed method

• Adopt a conditional denoising diffusion probabilistic model (DDPM) to explicitly model p(x_t-1^ta | x_t^ta, x_0^co) using a conditional denoising function epsilon_theta.
• Parameterize the reverse process with mu_theta and sigma_theta, where mu_theta is built from a conditional DDPM denoiser that uses x_t^ta and conditional observations x_0^co.
• Incorporate a self-supervised training scheme inspired by masked language modeling by partitioning observed values into imputation targets and conditional observations during training.
• Use a two-dimensional attention scheme (temporal Transformer and feature Transformer) to capture temporal and feature dependencies within time series data.
• Provide side information including time embeddings and categorical feature embeddings to enhance temporal and feature context.
• Allow zero-padding to fix input spaces and introduce a conditional mask to indicate observed indices, enabling fixed-shape conditioning for the model.

Experimental results

Research questions

• RQ1 Can a conditional diffusion model learn the exact conditional distribution for imputing missing values in multivariate time series?
• RQ2 Does conditioning on observed values and employing self-supervised training improve probabilistic and deterministic imputation over unconditional diffusion models and autoregressive baselines?
• RQ3 How does CSDI perform on healthcare and environmental datasets for probabilistic imputation, deterministic imputation, interpolation, and forecasting compared to state-of-the-art methods?
• RQ4 What architectural choices (attention mechanisms, side information, target strategies) most influence imputation quality and efficiency?

Key findings

• CSDI achieves substantial improvements in probabilistic imputation, reducing CRPS by 40-65% over baselines on healthcare and air quality datasets.
• Deterministic imputations with CSDI reduce MAE by 5-20% relative to state-of-the-art deterministic methods.
• CSDI is effective for time series interpolation and probabilistic forecasting, often competitive with or superior to specialized baselines.
• Self-supervised training leveraging a target-choice strategy enables learning without ground-truth missing values, enabling robust conditional modeling.
• An explicit conditional diffusion framework yields better utilization of observed data than unconditional diffusion approaches in imputation tasks.