[论文解读] ContiFormer: Continuous-Time Transformer for Irregular Time Series Modeling

Yuqi Chen, Kan Ren+4 more|arXiv (Cornell University)|Feb 16, 2024

Neural Networks and Applications被引用 19

一句话总结

ContiFormer 引入了一种连续时间 Transformer，通过将 Neural ODE 动态整合到注意力机制中来对非规则时间序列进行建模，从而实现连续时间表示和可并行计算。

ABSTRACT

Modeling continuous-time dynamics on irregular time series is critical to account for data evolution and correlations that occur continuously. Traditional methods including recurrent neural networks or Transformer models leverage inductive bias via powerful neural architectures to capture complex patterns. However, due to their discrete characteristic, they have limitations in generalizing to continuous-time data paradigms. Though neural ordinary differential equations (Neural ODEs) and their variants have shown promising results in dealing with irregular time series, they often fail to capture the intricate correlations within these sequences. It is challenging yet demanding to concurrently model the relationship between input data points and capture the dynamic changes of the continuous-time system. To tackle this problem, we propose ContiFormer that extends the relation modeling of vanilla Transformer to the continuous-time domain, which explicitly incorporates the modeling abilities of continuous dynamics of Neural ODEs with the attention mechanism of Transformers. We mathematically characterize the expressive power of ContiFormer and illustrate that, by curated designs of function hypothesis, many Transformer variants specialized in irregular time series modeling can be covered as a special case of ContiFormer. A wide range of experiments on both synthetic and real-world datasets have illustrated the superior modeling capacities and prediction performance of ContiFormer on irregular time series data. The project link is https://seqml.github.io/contiformer/.

研究动机与目标

• 激励对观测非均匀采样、持续演化的非规则时间序列进行建模。
• 提出一个将注意力扩展到在连续时间域中运行的连续时间 Transformer。
• 提供理论分析，表明 ContiFormer 将 vanilla Transformer 的变体视为特例。
• 展示在非规则时间序列的插值、分类和预测任务上的强性能。

提出的方法

• 为每个观测定义潜在轨迹并将点积注意力扩展到连续时间。
• 使用常微分方程来定义潜在轨迹，并对查询进行连续时间插值。
• 开发使用跨时间区间的连续内积的连续时间多头注意力(CT-MHA)。
• 将 CT-MHA 集成到带归一化和残差连接的 ContiFormer 层中，通过采样方案实现堆叠。
• 提供重参数化和基于 ODE 的积分，以在处理连续动力学的同时保持类似 Transformer 的并行性。

Figure 1: Architecture of the ContiFormer layer. ContiFormer takes an irregular time series and its corresponding sampled time points as input. Queries, keys, and values are obtained in continuous-time form. The attention mechanism (CT-MHA) performs a scaled inner product in a continuous-time manner

实验结果

研究问题

• RQ1 如何将注意力机制扩展为在连续时间中对非规则时间序列进行操作？
• RQ2 基于 Transformer 的模型是否比离散时间或基于 Neural ODE 的方法更有效地捕捉连续时间动力学？
• RQ3 相对于现有的非规则时间序列 Transformer 变体，ContiFormer 的表示能力有多大？
• RQ4 在不规则采样的插值、分类和事件预测任务中，ContiFormer 的表现如何？

主要发现

• ContiFormer 在合成数据和现实世界数据集上对非规则时间序列的连续时间动力学建模表现出色。
• 该模型在插值、外推、分类和事件预测方面的性能优于来自 RNN、Neural ODE、SSM 以及基于注意力的家族的基线方法。
• 理论分析表明许多 Transformer 变体可以被视为 ContiFormer 内的特例，突显广泛的表示能力。
• 实证结果表明 ContiFormer 在建模连续动力学的同时，保留长程信息并维持可并行计算。

Figure 2: Interpolation and extrapolation of spirals with irregularly-samples time points by Transformer, Neural ODE, and our model.