[Paper Review] Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning

Yuxuan Bian, Xuan Ju+4 more|arXiv (Cornell University)|Feb 7, 2024

Time Series Analysis and Forecasting4 citations

TL;DR

This paper proposes aL LM4TS, a novel framework that adapts Large Language Models (LLMs) for time-series representation learning by reformulating forecasting as a self-supervised multi-patch prediction task. It uses two-stage training—causal next-patch prediction followed by multi-patch fine-tuning—and introduces a patch-wise decoding layer that enhances temporal representation learning, achieving state-of-the-art performance across diverse downstream time-series tasks with superior transferability.

ABSTRACT

In this study, we present aLLM4TS, an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning. Central to our approach is that we reconceive time-series forecasting as a self-supervised, multi-patch prediction task, which, compared to traditional contrastive learning or mask-and-reconstruction methods, captures temporal dynamics in patch representations more effectively. Our strategy encompasses two-stage training: (i). a causal continual pre-training phase on various time-series datasets, anchored on next patch prediction, effectively syncing LLM capabilities with the intricacies of time-series data; (ii). fine-tuning for multi-patch prediction in the targeted time-series context. A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding. Such a design directly transposes individual patches into temporal sequences, thereby significantly bolstering the model's proficiency in mastering temporal patch-based representations. aLLM4TS demonstrates superior performance in several downstream tasks, proving its effectiveness in deriving temporal representations with enhanced transferability and marking a pivotal advancement in the adaptation of LLMs for time-series analysis.

Motivation & Objective

• To address the challenge of learning generalizable, transferable time-series representations across diverse downstream tasks.
• To overcome limitations of traditional self-supervised methods like mask-and-reconstruction and contrastive learning in capturing complex temporal dynamics.
• To adapt the sequence modeling strength of LLMs to time-series data through a task-specific, multi-patch prediction paradigm.
• To improve representation learning by decoupling patch-level decoding from sequence-level dependencies, enhancing model adaptability.
• To demonstrate superior performance and transferability of the proposed framework across multiple time-series benchmarks and tasks.

Proposed method

• Reconceptualizes time-series forecasting as a self-supervised multi-patch prediction task, replacing traditional mask-and-reconstruction or contrastive learning.
• Employs a two-stage training strategy: (1) causal continual pre-training on diverse time-series datasets using next-patch prediction, and (2) fine-tuning for multi-patch prediction in target domains.
• Introduces a patch-wise decoding layer that independently decodes each patch into the time domain using a small, shared linear layer, avoiding sequence-level modeling of temporal relationships.
• Uses a shared patch-level linear decoder matrix $\mathbf{W}_p \in \mathbb{R}^{D \times P}$, reducing parameters by up to 0.34% compared to sequence-level decoders.
• Leverages the inductive bias of LLMs for long-range sequence modeling while focusing on patch-level representations to improve temporal dynamics capture.
• Applies prompt-tuning or fine-tuning techniques to adapt the LLM backbone to time-series inputs after pre-training.

Experimental results

Research questions

• RQ1 Can multi-patch prediction serve as a more effective self-supervised pre-training objective than mask-and-reconstruction or contrastive learning for time-series representation learning?
• RQ2 How does patch-wise decoding compare to sequence-wise decoding in preserving temporal dynamics and improving model performance?
• RQ3 To what extent can a two-stage training strategy—causal next-patch pre-training followed by multi-patch fine-tuning—enhance transferability and performance across diverse time-series tasks?
• RQ4 Does the proposed framework aL LM4TS achieve state-of-the-art performance in forecasting, classification, and anomaly detection on standard benchmarks?
• RQ5 How does the parameter efficiency of the patch-wise decoder compare to conventional sequence-level decoders in terms of model size and training efficiency?

Key findings

• aL LM4TS achieves state-of-the-art performance on multiple time-series benchmarks, including ETTm1 and ETTm2, with average F1 scores of 87.51% on anomaly detection and 0.055 mean MAE on ETTm2 at 50% forecasting horizon.
• The patch-wise decoding layer reduces the number of parameters in the decoder by up to 0.34% compared to sequence-level decoders, significantly improving parameter efficiency.
• The two-stage training strategy—causal next-patch pre-training followed by multi-patch fine-tuning—leads to superior performance across all downstream tasks, outperforming methods like GPT4TS, PatchTST, and FEDformer.
• On the SMD dataset, aL LM4TS achieves a precision of 87.87%, recall of 83.09%, and F1-score of 85.42%, outperforming GPT4TS (F1: 86.72%) and other SOTA models.
• The model demonstrates strong zero-shot and few-shot generalization across diverse data domains, confirming its robust transferability and adaptability.
• The ablation study confirms that the patch-wise decoding design significantly improves performance by isolating temporal dynamics learning to the encoder, avoiding interference from decoder-level sequence modeling.