[Paper Review] PyODDS: An End-to-end Outlier Detection System with Automated Machine Learning

Yuening Li, Daochen Zha+3 more|arXiv (Cornell University)|Mar 12, 2020

Anomaly Detection Techniques and Applications22 references4 citations

TL;DR

PyODDS is an end-to-end, automated outlier detection system that integrates automated machine learning (AutoML) to optimize outlier detection pipelines, including algorithm selection and hyper-parameter tuning. It leverages a defined search space and Bayesian optimization strategy to achieve state-of-the-art performance on real-world datasets, outperforming handcrafted models and random search baselines in F1-score and convergence speed.

ABSTRACT

Outlier detection is an important task for various data mining applications. Current outlier detection techniques are often manually designed for specific domains, requiring large human efforts of database setup, algorithm selection, and hyper-parameter tuning. To fill this gap, we present PyODDS, an automated end-to-end Python system for Outlier Detection with Database Support, which automatically optimizes an outlier detection pipeline for a new data source at hand. Specifically, we define the search space in the outlier detection pipeline, and produce a search strategy within the given search space. PyODDS enables end-to-end executions based on an Apache Spark backend server and a light-weight database. It also provides unified interfaces and visualizations for users with or without data science or machine learning background. In particular, we demonstrate PyODDS on several real-world datasets, with quantification analysis and visualization results.

Motivation & Objective

• To address the high human effort required in manually configuring outlier detection pipelines across diverse domains.
• To automate the joint optimization of algorithm selection, hyper-parameter tuning, and contamination ratio estimation in outlier detection.
• To enable end-to-end execution using Apache Spark and a lightweight database for scalable, real-world deployment.
• To provide unified, user-friendly interfaces and visualizations for both technical and non-technical users.
• To demonstrate the effectiveness of AutoML in real-world data mining tasks beyond model training, specifically in outlier detection pipelines.

Proposed method

• PyODDS defines a search space that includes multiple outlier detection algorithms, their hyper-parameters (discrete, ordinal, continuous), and the contamination ratio as configurable components.
• It employs a Bayesian optimization-based search strategy to efficiently explore the search space and identify high-performing pipeline configurations.
• The system uses Apache Spark as a backend for distributed data processing and TDengine as a lightweight database for data storage and retrieval.
• It provides a unified API and visualization layer that displays raw data, outlier scores, time series decomposition (level, trend, seasonality, residual), and probability density functions.
• The framework supports flexible time-slice segmentation via query functions and secure server connection through authentication.
• The system evaluates pipeline performance using F1-score, precision, and recall, with customizable reward functions for false positives and false negatives.

Experimental results

Research questions

• RQ1 Can an automated system jointly optimize algorithm selection and hyper-parameter tuning to achieve superior outlier detection performance compared to handcrafted models?
• RQ2 How does the search strategy in PyODDS compare to random search in terms of convergence speed and final performance on real-world datasets?
• RQ3 To what extent can PyODDS generalize across diverse data sources without manual intervention?
• RQ4 Can the system provide interpretable visualizations that help non-expert users understand detection results and data patterns?
• RQ5 How effective is the integration of AutoML principles into end-to-end outlier detection pipelines for real-world data mining applications?

Key findings

• PyODDS achieved a highest F1-score of 96.68 on the NAB benchmark, matching the performance of the best handcrafted models and outperforming random search baselines.
• The system discovered a top-performing pipeline configuration (OCSVM with specific hyper-parameters) that achieved 96.68 F1-score, 95.27 recall for low false positives, and 94.76 for low false negatives.
• Compared to random search, PyODDS converged faster and identified higher-performing architectures (top-5) across multiple datasets, demonstrating superior search efficiency.
• The visual analytics component effectively illustrated data distribution, outlier scores, and residual components, enabling users to interpret anomalies such as spikes and extreme values.
• The system successfully reduced the need for expert intervention by automating pipeline configuration, including contamination ratio estimation and algorithm selection.
• The integration of Apache Spark and TDengine enabled scalable, real-time processing of time-series data, supporting large-scale deployment.