[Paper Review] Mixture-of-Linear-Experts for Long-term Time Series Forecasting

Ronghao Ni, Zinan Lin+2 more|arXiv (Cornell University)|Dec 11, 2023

Hydrological Forecasting Using AIEnvironmental Science7 citations

TL;DR

MoLE augments linear-centric LTSF models with a mixture-of-experts router that adaptively weights multiple linear predictors based on time-start embeddings, improving forecasting accuracy across diverse datasets.

ABSTRACT

Long-term time series forecasting (LTSF) aims to predict future values of a time series given the past values. The current state-of-the-art (SOTA) on this problem is attained in some cases by linear-centric models, which primarily feature a linear mapping layer. However, due to their inherent simplicity, they are not able to adapt their prediction rules to periodic changes in time series patterns. To address this challenge, we propose a Mixture-of-Experts-style augmentation for linear-centric models and propose Mixture-of-Linear-Experts (MoLE). Instead of training a single model, MoLE trains multiple linear-centric models (i.e., experts) and a router model that weighs and mixes their outputs. While the entire framework is trained end-to-end, each expert learns to specialize in a specific temporal pattern, and the router model learns to compose the experts adaptively. Experiments show that MoLE reduces forecasting error of linear-centric models, including DLinear, RLinear, and RMLP, in over 78% of the datasets and settings we evaluated. By using MoLE existing linear-centric models can achieve SOTA LTSF results in 68% of the experiments that PatchTST reports and we compare to, whereas existing single-head linear-centric models achieve SOTA results in only 25% of cases.

Motivation & Objective

• Motivate the need for better handling non-stationary temporal patterns in long-term time series forecasting.
• Introduce MoLE, a plug-in mixture-of-experts augmentation for linear-centric LTSF models.
• Demonstrate that MoLE improves forecasting accuracy across multiple datasets and baseline linear models.
• Provide ablations to show the gains come from time-aware specialization rather than model size alone.

Proposed method

• Augment any linear-centric LTSF model with MoLE, which creates multiple expert heads plus a channel-wise router.
• Router uses a two-layer MLP to generate per-channel weights from the embedding of the first timestamp.
• Combine expert outputs via a channel-wise weighted sum followed by a postprocessing layer to produce final forecasts.
• Embed temporal features into a datetime-like embedding and apply simple normalization.
• Experiment with DLinear, RLinear, and RMLP as backbones, comparing single-head vs multi-head MoLE variants.
• Conduct ablations including TimeIn, RandomIn, and RandomOut to assess the role of timestamp conditioning and randomness.

Experimental results

Research questions

• RQ1 Can Mixture-of-Linear-Experts improve long-term forecasting accuracy when applied to linear-centric LTSF models?
• RQ2 Does conditioning the mixer on the first-timestamp embedding yield gains beyond increased model capacity?
• RQ3 How does MoLE performance vary across datasets, forecast horizons, and input sequence lengths?
• RQ4 Are gains primarily due to time-aware specialization of experts rather than mere parameter growth?
• RQ5 What are the effects of hyperparameters (number of heads, dropout, batch size) on MoLE performance?

Key findings

• MoLE improves the forecasting error of DLinear in 32/44 settings (73%).
• MoLE improves RLinear in 38/44 settings (86%).
• MoLE improves RMLP in 33/44 settings (75%).
• Across Weather2K, MoLE-linear models achieve state-of-the-art in all settings evaluated.
• Overall, MoLE enables linear-centric models to reach SOTA in 68% of PatchTST-reported settings (vs 25% for single-head baselines).
• Ablations show gains stem from specialized time-aware experts, with larger gains when input lengths are short relative to prediction lengths.