[Paper Review] CALF: Aligning LLMs for Time Series Forecasting via Cross-modal Fine-Tuning

Peiyuan Liu, Hang Guo+6 more|arXiv (Cornell University)|Mar 12, 2024

Stock Market Forecasting Methods5 citations

TL;DR

CALF introduces a cross-modal fine-tuning framework that aligns temporal time series with pre-trained LLM embeddings through static and dynamic knowledge distillation, achieving state-of-the-art results in both long- and short-term forecasting and improving generalization.

ABSTRACT

Deep learning (e.g., Transformer) has been widely and successfully used in multivariate time series forecasting (MTSF). Unlike existing methods that focus on training models from a single modal of time series input, large language models (LLMs) based MTSF methods with cross-modal text and time series input have recently shown great superiority, especially with limited temporal data. However, current LLM-based MTSF methods usually focus on adapting and fine-tuning LLMs, while neglecting the distribution discrepancy between textual and temporal input tokens, thus leading to sub-optimal performance. To address this issue, we propose a novel Cross-Modal LLM Fine-Tuning (CALF) framework for MTSF by reducing the distribution discrepancy between textual and temporal data, which mainly consists of the temporal target branch with temporal input and the textual source branch with aligned textual input. To reduce the distribution discrepancy, we develop the cross-modal match module to first align cross-modal input distributions. Additionally, to minimize the modality distribution gap in both feature and output spaces, feature regularization loss is developed to align the intermediate features between the two branches for better weight updates, while output consistency loss is introduced to allow the output representations of both branches to correspond effectively. Thanks to the modality alignment, CALF establishes state-of-the-art performance for both long-term and short-term forecasting tasks with low computational complexity, and exhibiting favorable few-shot and zero-shot abilities similar to that in LLMs. Code is available at https://github.com/Hank0626/LLaTA.

Motivation & Objective

• Motivate bridging modality gap between time series data and textual LLM representations to improve forecasting generalization.
• Leverage cross-modal knowledge distillation to transfer static (embedding space) and dynamic (contextual) LLM knowledge to a temporal forecasting model.
• Design a two-branch architecture to integrate temporal data processing with LLM-based textual knowledge.
• Develop training losses and efficiency strategies to enable effective transfer without catastrophic forgetting.

Proposed method

• Two-branch framework with a textual modal branch (LLM-based) and a temporal modal branch (time series processing).
• Static knowledge learning uses PCA to compress word embeddings and cross-attention to align projected time tokens with principal word embeddings, producing aligned text tokens for the LLM branch.
• Dynamic knowledge learning uses two distillation losses: (i) feature regularization to align intermediate representations across modalities and (ii) modal consistency loss to align outputs of both branches.
• Total training loss combines supervised loss with the two distillation losses: L_total = L_sup + lambda1 L_output + lambda2 L_feature.
• Parameter-efficient training uses LoRA and fine-tuning of positional encodings for the temporal branch.
• Ground-truth supervision and cross-modal alignment losses guide knowledge transfer from the textual to the temporal branch.

Experimental results

Research questions

• RQ1 Can cross-modal knowledge distillation reduce the modality gap between time series and text embeddings to improve forecasting performance?
• RQ2 How can static (embedding-based) and dynamic (contextual) knowledge from pre-trained LLMs be leveraged to enhance time series forecasting and generalization?
• RQ3 What are effective loss designs to enforce cross-modal consistency and prevent forgetting during fine-tuning?
• RQ4 Does the proposed LLaTA framework achieve state-of-the-art performance across long-term, short-term, and few/zero-shot forecasting scenarios?

Key findings

• LLaTA achieves state-of-the-art performance on both long- and short-term forecasting across multiple real-world datasets.
• In long-term forecasting, LLaTA delivers top results in the majority of evaluations and consistently outperforms strong baselines including PatchTST and GPT4TS.
• In short-term forecasting on M4 datasets, LLaTA attains best results in 14 of 15 categories and surpasses leading methods.
• LLaTA demonstrates strong few-shot and zero-shot generalization, outperforming baselines in cross-domain transfer scenarios.
• Ablation studies confirm that combining supervised, feature regularization, and modal consistency losses yields the best MSE/MAE performance on tested datasets.