[Paper Review] CDSA: Cross-Dimensional Self-Attention for Multivariate, Geo-tagged Time Series Imputation

Jiawei Ma, Zheng Shou+4 more|arXiv (Cornell University)|May 23, 2019

Time Series Analysis and Forecasting35 references39 citations

TL;DR

Introduces Cross-Dimensional Self-Attention (CDSA) to impute missing values in multivariate, geo-tagged time series by modeling attention across time, location, and measurement dimensions; outperforms state-of-the-art imputation and forecasting methods on multiple datasets.

ABSTRACT

Many real-world applications involve multivariate, geo-tagged time series data: at each location, multiple sensors record corresponding measurements. For example, air quality monitoring system records PM2.5, CO, etc. The resulting time-series data often has missing values due to device outages or communication errors. In order to impute the missing values, state-of-the-art methods are built on Recurrent Neural Networks (RNN), which process each time stamp sequentially, prohibiting the direct modeling of the relationship between distant time stamps. Recently, the self-attention mechanism has been proposed for sequence modeling tasks such as machine translation, significantly outperforming RNN because the relationship between each two time stamps can be modeled explicitly. In this paper, we are the first to adapt the self-attention mechanism for multivariate, geo-tagged time series data. In order to jointly capture the self-attention across multiple dimensions, including time, location and the sensor measurements, while maintain low computational complexity, we propose a novel approach called Cross-Dimensional Self-Attention (CDSA) to process each dimension sequentially, yet in an order-independent manner. Our extensive experiments on four real-world datasets, including three standard benchmarks and our newly collected NYC-traffic dataset, demonstrate that our approach outperforms the state-of-the-art imputation and forecasting methods. A detailed systematic analysis confirms the effectiveness of our design choices.

Motivation & Objective

• Address missing data in multivariate, geo-tagged time series from multiple sensors across locations.
• Leverage self-attention to model relationships across time, space, and measurement dimensions.
• Propose CDSA to capture cross-dimensional dependencies efficiently, avoiding full joint attention complexity.
• Demonstrate superior imputation and forecasting performance on real-world datasets compared to RNN-based and traditional methods.

Proposed method

• Proposes three CDSA strategies to model cross-dimensional attention: Independent (per-dimension attention with late fusion), Joint (full cross-dimensional attention), and Decomposed (dimension-specific attention chained together).
• Represents input as X ∈ R^{T×L×M} and reshapes into X_T, X_L, X_M for dimension-specific streams; uses attention maps A_T, A_L, A_M and a decomposed product to form a cross-dimensional attention A.
• Imputes missing values via a Transformer Encoder (8 layers) with residual connections and layer normalization; forecasting uses a Transformer with encoder-decoder (9 layers).
• Normalizes measurements, fills missing entries with zero, and trains with Adam to minimize RMSE for imputation tasks.
• Evaluates on NYC-Traffic, KDD-2015, KDD-2018, and METR-LA datasets, comparing against AR, Kriging, MTSI, BRITS, DCRNN, and others.

Experimental results

Research questions

• RQ1 Can self-attention be effectively adapted to multivariate, geo-tagged time series imputation across time, location, and measurement dimensions?
• RQ2 Which modeling strategy (Independent, Joint, Decomposed) best captures cross-dimensional dependencies while remaining computationally feasible?
• RQ3 Do CDSA-based models improve imputation and forecasting accuracy over state-of-the-art RNN-based and traditional imputation methods across diverse datasets?

Key findings

• CDSA outperforms state-of-the-art methods in imputation and forecasting across multiple real-world datasets.
• The Decomposed CDSA variant achieves the best trade-off between accuracy and computational efficiency, outperforming Independent and Joint/Shared configurations.
• On NYC-Traffic, CDSA reaches RMSE values lower than AR, Kriging, MTSI, BRITS, and DCRNN across missing rates from 20% to 90%.
• On KDD-2015 PM2.5, TEMP, and HUM tasks, CDSA achieves competitive MAE/MRE and superior cross-dataset performance.
• On KDD-2018, CDSA (Decomposed) attains the best MSE at all missing rates where reported.
• On METR-LA forecasting, CDSA improves long-term predictions (60 minutes) relative to several baselines.