[Paper Review] Are We Winning the Wrong Game? Revisiting Evaluation Practices for Long-Term Time Series Forecasting

Thanapol Phungtua-eng, Yoshitaka Yamamoto|arXiv (Cornell University)|Mar 9, 2026

Forecasting Techniques and Applications0 citations

TL;DR

The paper argues that long-term time series forecasting evaluation over-relies on aggregated pointwise metrics (MSE/MAE) and proposes a multidimensional, diagnostic evaluation framework focusing on statistical fidelity, structural coherence, and decision relevance, shifting from leaderboard chasing to context-aware forecasting.

ABSTRACT

Long-term time series forecasting (LTSF) is widely recognized as a central challenge in data mining and machine learning. LTSF has increasingly evolved into a benchmark-driven ''GAME,'' where models are ranked, compared, and declared state-of-the-art based primarily on marginal reductions in aggregated pointwise error metrics such as MSE and MAE. Across a small set of canonical datasets and fixed forecasting horizons, progress is communicated through leaderboard-style tables in which lower numerical scores define success. In this GAME, what is measured becomes what is optimized, and incremental error reduction becomes the dominant currency of advancement. We argue that this metric-centric regime is not merely incomplete, but structurally misaligned with the broader objectives of forecasting. In real-world settings, forecasting often prioritizes preserving temporal structure, trend stability, seasonal coherence, robustness to regime shifts, and supporting downstream decision processes. Optimizing aggregate pointwise error does not necessarily imply modeling these structural properties. As a result, leaderboard improvement may increasingly reflect specialization in benchmark configurations rather than a deeper understanding of temporal dynamics. This paper revisits LTSF evaluation as a foundational question in data science: what does it mean to measure forecasting progress? We propose a multi-dimensional evaluation perspective that integrates statistical fidelity, structural coherence, and decision-level relevance. By challenging the current metric monoculture, we aim to redirect attention from winning benchmark tables toward advancing meaningful, context-aware forecasting.

Motivation & Objective

• Challenge the adequacy of current LTSF evaluation practices focused on aggregated pointwise errors.
• Advocate a multidimensional evaluation framework including statistical fidelity, structural coherence, and decision-level relevance.
• Encourage diagnostic reporting and window-level analysis to reveal performance across structural conditions.
• Promote domain-specific, context-aware assessment over universal leaderboard dominance.

Proposed method

• Critically analyze existing benchmark-driven evaluation regimes in LTSF and their incentives.
• Propose a three-dimensional evaluation framework: Statistical Fidelity, Structural Coherence, and Decision-Level Relevance.
• Suggest diagnostic reporting and window-level analysis to assess performance variability across regimes.
• Advocate reporting that emphasizes domain-specific objectives and conditional performance rather than universal leaders.
• Reference and build on emerging benchmarking initiatives (e.g., TSFM-Bench, TIME, TFB) to illustrate diversified evaluation practices.

Experimental results

Research questions

• RQ1 Is the current evaluation game in LTSF aligned with real forecasting objectives beyond marginal MSE/MAE reductions?
• RQ2 How can LTSF evaluation be reframed to balance statistical fidelity, temporal structure, and decision usefulness?
• RQ3 What practical evaluation practices (e.g., window-level diagnostics) can reveal model behavior under diverse structural conditions?
• RQ4 Do universal champions exist across datasets and horizons, or should evaluation be contextualized to domain goals?

Key findings

• Leaderboard-centric progress often reflects benchmark-specific adaptation rather than deeper temporal understanding.
• Pointwise errors fail to capture structural properties like trend preservation and regime robustness.
• A three-dimensional evaluation view (statistical fidelity, structural coherence, decision relevance) provides a richer forecast assessment.
• Window-level diagnostics reveal variability masked by global averages and highlight regime-specific performance.
• No universal champion exists; domain-specific objectives should guide model evaluation and deployment.