[Paper Review] Merlion: A Machine Learning Library for Time Series

Aadyot Bhatnagar, Paul Kassianik+21 more|arXiv (Cornell University)|Sep 20, 2021

Time Series Analysis and Forecasting30 references18 citations

TL;DR

Merlion is an open-source Python library that provides a unified, production-ready framework for time series forecasting and anomaly detection, featuring standardized data handling, autoML, model ensembling, and a live-retraining simulation for realistic evaluation. It achieves robust performance across diverse datasets, with its proposed ensemble model consistently outperforming individual models in F1 score consistency and generalization.

ABSTRACT

We introduce Merlion, an open-source machine learning library for time series. It features a unified interface for many commonly used models and datasets for anomaly detection and forecasting on both univariate and multivariate time series, along with standard pre/post-processing layers. It has several modules to improve ease-of-use, including visualization, anomaly score calibration to improve interpetability, AutoML for hyperparameter tuning and model selection, and model ensembling. Merlion also provides a unique evaluation framework that simulates the live deployment and re-training of a model in production. This library aims to provide engineers and researchers a one-stop solution to rapidly develop models for their specific time series needs and benchmark them across multiple time series datasets. In this technical report, we highlight Merlion's architecture and major functionalities, and we report benchmark numbers across different baseline models and ensembles.

Motivation & Objective

• Address the lack of standardized, production-ready tools for time series machine learning in industry workflows.
• Unify diverse models and datasets for forecasting and anomaly detection under a single, extensible interface.
• Improve model interpretability and reduce false positives through post-processing rules for anomaly scores.
• Enable automated hyperparameter tuning and model selection via an AutoML module.
• Provide a realistic evaluation framework that simulates live deployment and re-training in production environments.

Proposed method

• Implement a unified API for loading, preprocessing, training, and evaluating models across univariate and multivariate time series.
• Integrate a suite of models including statistical methods (Isolation Forest, Random Cut Forest), deep learning (LSTM autoencoders, VAEs, DAGMM), and classical approaches (Prophet, pmdarima).
• Apply post-processing rules such as threshold calibration and smoothing to improve anomaly score interpretability and reduce false positives.
• Use ensemble learning to combine outputs of multiple models (e.g., RCF + VAE) for improved robustness and generalization.
• Simulate real-world deployment by re-training models periodically (e.g., weekly) and evaluating performance on rolling windows.
• Support visualization of predictions and anomaly scores to aid qualitative analysis and debugging.

Experimental results

Research questions

• RQ1 How can a unified, extensible framework improve the development and benchmarking of time series models across diverse datasets?
• RQ2 To what extent does autoML improve model performance across different forecasting and anomaly detection tasks?
• RQ3 Can post-processing rules significantly reduce false positive rates while maintaining or improving anomaly detection F1 scores?
• RQ4 How effective is model ensembling in achieving consistent, robust performance across multiple time series datasets?
• RQ5 Does periodic re-training in a simulated production pipeline lead to measurable improvements in model generalization and stability?

Key findings

• The Merlion ensemble model achieved the lowest average gap (0.051 ± 0.038) in F1 score relative to the best-performing model per dataset, outperforming individual models in consistency.
• The LSTM encoder-decoder model achieved the highest F1 score on the MSL dataset (0.381), while the Random Cut Forest performed best on SMD (0.500), indicating dataset-specific model superiority.
• Re-training models weekly had no statistically significant impact on F1 performance for most models (p > 0.05), except for DAGMM (p = 0.032), suggesting limited benefit from frequent retraining in this setup.
• The autoML module consistently improved performance across multiple forecasting models, demonstrating its value in hyperparameter optimization.
• Post-processing rules significantly enhanced interpretability and reduced false positives, improving practical usability in production settings.
• Merlion’s evaluation pipeline successfully simulated real-world deployment, enabling reliable benchmarking across multiple datasets and model configurations.