[Paper Review] Benchmarking time series classification -- Functional data vs machine learning approaches

Florian Pfisterer, Laura Beggel+3 more|arXiv (Cornell University)|Nov 18, 2019

Time Series Analysis and Forecasting35 references8 citations

TL;DR

This paper benchmarks functional data analysis (FDA) and machine learning (ML) methods for time series classification, integrating diverse FDA techniques and feature extraction pipelines into a unified R framework using the mlr toolbox. It demonstrates that tuned feature extraction combined with traditional ML models (e.g., xgboost, random forest) often outperforms dedicated FDA methods, with hyperparameter tuning reducing error by up to 20.3%, offering a practical, accessible, and high-performing pipeline for non-experts.

ABSTRACT

Time series classification problems have drawn increasing attention in the machine learning and statistical community. Closely related is the field of functional data analysis (FDA): it refers to the range of problems that deal with the analysis of data that is continuously indexed over some domain. While often employing different methods, both fields strive to answer similar questions, a common example being classification or regression problems with functional covariates. We study methods from functional data analysis, such as functional generalized additive models, as well as functionality to concatenate (functional-) feature extraction or basis representations with traditional machine learning algorithms like support vector machines or classification trees. In order to assess the methods and implementations, we run a benchmark on a wide variety of representative (time series) data sets, with in-depth analysis of empirical results, and strive to provide a reference ranking for which method(s) to use for non-expert practitioners. Additionally, we provide a software framework in R for functional data analysis for supervised learning, including machine learning and more linear approaches from statistics. This allows convenient access, and in connection with the machine-learning toolbox mlr, those methods can now also be tuned and benchmarked.

Motivation & Objective

• To provide a comprehensive, accessible benchmark of functional data analysis and machine learning methods for time series classification.
• To unify diverse FDA and ML tools under a single, consistent API using the mlr framework for easier comparison and tuning.
• To evaluate the performance of both specialized FDA methods and feature extraction + ML pipelines across a wide range of time series datasets.
• To guide non-expert practitioners in selecting optimal models through empirical benchmarking and hyperparameter tuning.
• To close the gap in literature by systematically evaluating hyperparameter tuning effects and feature extraction techniques on functional data.

Proposed method

• Integrates multiple R packages for functional data analysis (e.g., fda, FDboost, fda.usc) into a unified interface via the mlr framework.
• Enables end-to-end workflows combining functional data preprocessing, feature extraction (e.g., B-splines, wavelets, Fourier), and traditional ML models (e.g., random forest, xgboost, SVM).
• Employs Bayesian optimization for efficient hyperparameter tuning across all method-feature combinations.
• Uses standardized resampling, performance metrics (MMCE), and parallelized benchmarking across 23 diverse time series datasets.
• Supports both functional modeling (e.g., functional generalized additive models) and non-functional feature-based modeling via transformation pipelines.
• Provides open-source R code and integration with mlr for automated tuning, benchmarking, and model selection.

Experimental results

Research questions

• RQ1 Can classical machine learning models, when applied to extracted vector-valued features from functional data, achieve competitive performance compared to specialized functional data analysis methods?
• RQ2 How do hyperparameter tuning and feature extraction techniques impact the performance of time series classification models?
• RQ3 Do interpretable functional data analysis methods like FDboost perform competitively on standard time series classification benchmarks?
• RQ4 Which combinations of feature extraction and machine learning models yield the most robust and accurate results across diverse time series data?
• RQ5 Is there a practical, unified framework that allows non-experts to easily compare, tune, and deploy both FDA and ML methods for time series classification?

Key findings

• Tuning significantly improves model performance, reducing absolute MMCE by up to 20.3% for Fourier-based feature extraction with xgboost.
• The average reduction in MMCE across all feature extraction techniques was 11% for FDboost, 7.78% for ksvm, 5.69% for xgboost, and 3.59% for ranger.
• A simple random forest without preprocessing achieved an average rank of 15.59 (top 4) across all datasets, serving as a strong baseline.
• Feature extraction methods such as B-spline (bsignal) and wavelet decomposition, when combined with xgboost or random forest, produced highly competitive models.
• The pareto-optimal set of models across datasets included 23 distinct algorithm–feature extraction combinations, indicating high diversity in optimal configurations.
• Despite their interpretability, specialized FDA methods like FDboost did not perform well in the benchmark, suggesting limited generalization to standard time series tasks.