[Paper Review] PTCT: Patches with 3D-Temporal Convolutional Transformer Network for Precipitation Nowcasting

Ziao Yang, Xiangrui Yang+1 more|arXiv (Cornell University)|Dec 2, 2021

Precipitation Measurement and Analysis4 citations

TL;DR

This paper proposes PTCT, a novel 3D-temporal convolutional Transformer network that splits radar echo frames into patches to overcome CNNs' inductive bias and uses 3D-temporal convolutions to capture short-term spatiotemporal dependencies. PTCT achieves state-of-the-art performance on Radar Echo Guangzhou and HKO-7 datasets by combining patch-based representation with autoregressive inference for high-quality precipitation nowcasting.

ABSTRACT

Precipitation nowcasting is to predict the future rainfall intensity over a short period of time, which mainly relies on the prediction of radar echo sequences. Though convolutional neural network (CNN) and recurrent neural network (RNN) are widely used to generate radar echo frames, they suffer from inductive bias (i.e., translation invariance and locality) and seriality, respectively. Recently, Transformer-based methods also gain much attention due to the great potential of Transformer structure, whereas short-term dependencies and autoregressive characteristic are ignored. In this paper, we propose a variant of Transformer named patches with 3D-temporal convolutional Transformer network (PTCT), where original frames are split into multiple patches to remove the constraint of inductive bias and 3D-temporal convolution is employed to capture short-term dependencies efficiently. After training, the inference of PTCT is performed in an autoregressive way to ensure the quality of generated radar echo frames. To validate our algorithm, we conduct experiments on two radar echo dataset: Radar Echo Guangzhou and HKO-7. The experimental results show that PTCT achieves state-of-the-art (SOTA) performance compared with existing methods.

Motivation & Objective

• Address the limitations of CNNs and RNNs in precipitation nowcasting, including inductive bias and serial computation.
• Overcome the lack of short-term dependency modeling and non-autoregressive generation in existing Transformer-based methods.
• Improve radar echo frame prediction quality by integrating patch-based representation with 3D-temporal convolutions.
• Enable high-fidelity, autoregressive generation of future radar echo sequences for accurate short-term precipitation forecasting.

Proposed method

• Split input radar echo frames into non-overlapping spatial patches to eliminate translation invariance and locality constraints of CNNs.
• Apply 3D-temporal convolutional operations within the self-attention mechanism to model short-term temporal dependencies across consecutive frames.
• Use a patch embedding module that incorporates masking during training to enhance feature representation and robustness.
• Integrate a spatiotemporal encoder with multi-head self-attention to capture long-range spatial and temporal dependencies.
• Employ an autoregressive inference strategy where each predicted frame is fed back to generate the next, ensuring temporal coherence.
• Train the model end-to-end using reconstruction and prediction losses on radar echo sequences.

Experimental results

Research questions

• RQ1 Can patch-based representation improve the modeling of location-variant radar echo motion compared to standard CNNs?
• RQ2 Does integrating 3D-temporal convolution into the Transformer architecture enhance short-term dependency learning in radar echo sequences?
• RQ3 How does the autoregressive inference strategy affect the quality and temporal consistency of generated radar echo frames?
• RQ4 What is the optimal patch size and masking ratio for balancing performance and robustness in radar nowcasting?
• RQ5 To what extent does the proposed PTCT model outperform existing SOTA methods on real-world radar echo datasets?

Key findings

• PTCT achieves state-of-the-art performance on both Radar Echo Guangzhou and HKO-7 datasets, with a SSIM of 0.907 and CSI of 0.531 on Radar Echo Guangzhou.
• Using $20\times20$ patches yields significantly better performance than smaller or single-patch configurations, demonstrating the importance of patching for removing inductive bias.
• The ablation study confirms that both location-variant features and long-range dependencies are essential, as removing either degrades performance substantially.
• Masking up to 50% of patches during training maintains strong performance, indicating robustness to missing or noisy input data.
• 3D-temporal convolution outperforms 2D convolution, with a 6.4% absolute improvement in CSI (0.531 vs. 0.467), proving its effectiveness in capturing short-term dynamics.
• The model maintains good generalization, achieving a POD of 0.701 even with 25% masked patches during inference, suggesting strong reconstruction capability.