[Paper Review] PGN: The RNN's New Successor is Effective for Long-Range Time Series Forecasting

Yanyan Jia, Youfang Lin+4 more|arXiv (Cornell University)|Sep 26, 2024

Medical Coding and Health Information6 citations

TL;DR

PGN introduces Parallel Gated Network as a successor to RNN, and Temporal PGN (TPGN) with two branches to capture long-term and short-term information for efficient long-range time series forecasting, achieving SOTA on five benchmarks.

ABSTRACT

Due to the recurrent structure of RNN, the long information propagation path poses limitations in capturing long-term dependencies, gradient explosion/vanishing issues, and inefficient sequential execution. Based on this, we propose a novel paradigm called Parallel Gated Network (PGN) as the new successor to RNN. PGN directly captures information from previous time steps through the designed Historical Information Extraction (HIE) layer and leverages gated mechanisms to select and fuse it with the current time step information. This reduces the information propagation path to $\mathcal{O}(1)$, effectively addressing the limitations of RNN. To enhance PGN's performance in long-range time series forecasting tasks, we propose a novel temporal modeling framework called Temporal PGN (TPGN). TPGN incorporates two branches to comprehensively capture the semantic information of time series. One branch utilizes PGN to capture long-term periodic patterns while preserving their local characteristics. The other branch employs patches to capture short-term information and aggregate the global representation of the series. TPGN achieves a theoretical complexity of $\mathcal{O}(\sqrt{L})$, ensuring efficiency in its operations. Experimental results on five benchmark datasets demonstrate the state-of-the-art (SOTA) performance and high efficiency of TPGN, further confirming the effectiveness of PGN as the new successor to RNN in long-range time series forecasting. The code is available in this repository: \url{https://github.com/Water2sea/TPGN}.

Motivation & Objective

• Motivate the need for better long-range time series forecasting beyond traditional RNNs due to information propagation and gradient issues.
• Introduce PGN as a parallelized successor to RNN with History Information Extraction to reduce propagation paths to O(1).
• Develop Temporal PGN (TPGN) with two branches to jointly model long-term periodic and short-term information in a 2D representation.
• Demonstrate state-of-the-art forecasting accuracy and computational efficiency on five real-world datasets.

Proposed method

• Propose Parallel Gated Network (PGN) with a History Information Extraction (HIE) layer and a single parallel gate to fuse current and historical information.
• Derive Out = G ⊙ H + (1−G) ⊙ tanh(Wt [X, H] + bt), where H = HIE(Padding(X)) and G, H are learned gates and representations.
• Transform 1D time series into a structured form and apply PGN to extract long-term information along the temporal dimension.
• Introduce Temporal PGN (TPGN) with two branches: a long-term branch that applies PGN across rows to preserve column-wise periodic traits, and a short-term branch that uses patch-based aggregation to capture global short-term information.
• Forecast by concatenating outputs from both branches and mapping through a linear layer to obtain the future sequence.
• Provide complexity analyses: PGN is O(L) in theory with parallelizable operations; TPGN is O(√L) due to the two-branch design.

Experimental results

Research questions

• RQ1 Can PGN effectively replace RNNs by reducing information propagation paths to O(1) while preserving long-term dependencies?
• RQ2 How can 1D time series data be transformed to 2D to better capture long-term periodic patterns and short-term changes?
• RQ3 Does the two-branch Temporal PGN framework (long-term PGN-based and short-term patch-based) improve long-range forecasting performance and efficiency across diverse datasets?

Key findings

• TPGN achieves state-of-the-art performance on five real-world datasets for long-range forecasting.
• TPGN delivers an average MSE improvement of 12.35% over prior best methods and an MAE improvement of 7.25% on long-range tasks.
• PGN reduces the information propagation path to O(1) and enables parallel computation with the same theoretical O(L) complexity as RNN, but higher practical efficiency.
• TPGN maintains a favorable O(√L) complexity, arising from its two-branch architecture that processes long-term and short-term information efficiently.
• Across baselines and tasks, TPGN shows robust improvements as forecasting length increases, with faster performance degradation than competing methods.
• Ablation studies validate the necessity of both long-term PGN-based and short-term patch-based branches.