[Paper Review] The impact of data set similarity and diversity on transfer learning success in time series forecasting

Claudia Ehrig, Sonnleitner, Benedikt+3 more|arXiv (Cornell University)|Apr 9, 2024

Forecasting Techniques and Applications4 citations

TL;DR

This study investigates how source-target data set similarity and source data diversity impact transfer learning performance in time series forecasting using pre-trained DeepAR models. It finds that higher similarity improves zero-shot accuracy and reduces bias, while greater diversity enhances uncertainty estimation and accuracy—especially in zero-shot settings, with feature-based measures outperforming shape-based ones.

ABSTRACT

Pre-trained models have become pivotal in enhancing the efficiency and accuracy of time series forecasting on target data sets by leveraging transfer learning. While benchmarks validate the performance of model generalization on various target data sets, there is no structured research providing similarity and diversity measures to explain which characteristics of source and target data lead to transfer learning success. Our study pioneers in systematically evaluating the impact of source-target similarity and source diversity on zero-shot and fine-tuned forecasting outcomes in terms of accuracy, bias, and uncertainty estimation. We investigate these dynamics using pre-trained neural networks across five public source datasets, applied to forecasting five target data sets, including real-world wholesales data. We identify two feature-based similarity and diversity measures, finding that source-target similarity reduces forecasting bias, while source diversity improves forecasting accuracy and uncertainty estimation, but increases the bias.

Motivation & Objective

• To systematically evaluate the impact of source-target similarity and source data diversity on transfer learning success in time series forecasting.
• To identify which data set characteristics—measured via feature-based similarity and diversity—predict better forecasting performance in zero-shot and fine-tuned settings.
• To provide empirical evidence for the effectiveness of selecting similar or diverse source data sets, especially in the absence of domain-specific pre-training.
• To establish a framework for evaluating transfer learning success using measurable, feature-based metrics rather than relying on shape-based or ad hoc similarity measures.

Proposed method

• Pre-trained a DeepAR model on five public time series data sets from diverse domains, including a combined 'Multisource' model to simulate a small foundation model.
• Applied the pre-trained models in both zero-shot and fine-tuned forecasting settings on five target data sets, including real-world wholesales data.
• Computed two feature-based similarity measures using tsfresh and catch22 features to quantify source-target similarity and source diversity.
• Defined source diversity as the variance of time series features within a data set and visualized it in latent PCA space for interpretability.
• Benchmarked forecasting performance against ETS using metrics including accuracy, bias, uncertainty estimation, and training time.
• Used Dynamic Time Warping (DTW) as a shape-based similarity measure for comparison, though no significant correlation was found.

Figure 1: Zero-shot/scratch (red circles) and fine-tuned (blue circles) accuracy in AvgRelRMSE (see Eq. 1 ) per source model across the five target data sets.

Experimental results

Research questions

• RQ1 How does source-target similarity in feature space affect zero-shot and fine-tuned forecasting accuracy and bias?
• RQ2 How does source data diversity, measured via feature variance, influence forecasting accuracy and uncertainty estimation in transfer learning?
• RQ3 Do shape-based similarity measures like DTW correlate with transfer learning performance, or are feature-based measures more predictive?
• RQ4 How do the relationships between similarity/diversity and forecasting performance differ between zero-shot and fine-tuned settings?
• RQ5 Can feature-based similarity and diversity metrics guide the selection of optimal source data sets for transfer learning in new forecasting applications?

Key findings

• Higher source-target similarity in tsfresh features significantly improves zero-shot forecasting accuracy and reduces bias, with the effect being stronger in zero-shot than fine-tuned settings.
• Greater source diversity in catch22 features leads to better zero-shot accuracy and more reliable uncertainty estimation, though it increases bias in the same setting.
• Source diversity in tsfresh features increases zero-shot bias, indicating that not all diversity measures yield consistent improvements across all performance aspects.
• The relationships between similarity, diversity, and performance are much weaker in fine-tuned settings, where model adaptation reduces the influence of source data characteristics.
• DTW-based shape similarity showed no significant correlation with forecasting performance, suggesting it is less informative than feature-based measures for transfer learning success prediction.
• The study supports existing practices of selecting either similar source data or highly diverse source data for transfer learning, with feature-based metrics providing actionable selection criteria.

(a) Sources Tsfresh