[Paper Review] Time-Series Classification in Smart Manufacturing Systems: An Experimental Evaluation of State-of-the-Art Machine Learning Algorithms

Mojtaba A. Farahani, Matt McCormick+2 more|arXiv (Cornell University)|Oct 4, 2023

Time Series Analysis and Forecasting93 references4 citations

TL;DR

This study evaluates state-of-the-art machine learning and deep learning algorithms for time-series classification (TSC) in smart manufacturing systems. Using 22 diverse manufacturing datasets, it identifies ResNet, DrCIF, InceptionTime, and ARSENAL as top performers, achieving over 96.6% average accuracy, highlighting the effectiveness of convolutional kernels in capturing temporal patterns in industrial time-series data.

ABSTRACT

Manufacturing is gathering extensive amounts of diverse data, thanks to the growing number of sensors and rapid advances in sensing technologies. Among the various data types available in SMS settings, time-series data plays a pivotal role. Hence, TSC emerges is crucial in this domain. The objective of this study is to fill this gap by providing a rigorous experimental evaluation of the SoTA ML and DL algorithms for TSC tasks in manufacturing and industrial settings. We first explored and compiled a comprehensive list of more than 92 SoTA algorithms from both TSC and manufacturing literature. Following, we selected the 36 most representative algorithms from this list. To evaluate their performance across various manufacturing classification tasks, we curated a set of 22 manufacturing datasets, representative of different characteristics that cover diverse manufacturing problems. Subsequently, we implemented and evaluated the algorithms on the manufacturing benchmark datasets, and analyzed the results for each dataset. Based on the results, ResNet, DrCIF, InceptionTime, and ARSENAL are the top-performing algorithms, boasting an average accuracy of over 96.6% across all 22 manufacturing TSC datasets. These findings underscore the robustness, efficiency, scalability, and effectiveness of convolutional kernels in capturing temporal features in time-series data, as three out of the top four performing algorithms leverage these kernels for feature extraction. Additionally, LSTM, BiLSTM, and TS-LSTM algorithms deserve recognition for their effectiveness in capturing features within time-series data using RNN-based structures.

Motivation & Objective

• To address the gap in empirical evaluation of state-of-the-art (SoTA) machine learning algorithms for time-series classification (TSC) in industrial manufacturing settings.
• To compile and evaluate a comprehensive set of 36 representative SoTA TSC algorithms across diverse manufacturing data characteristics.
• To establish a benchmark dataset collection of 22 real-world manufacturing time-series datasets representative of industrial problems.
• To compare the performance, robustness, and scalability of deep learning and traditional ML models in industrial TSC tasks.
• To identify the most effective architectures and techniques for industrial time-series classification, particularly focusing on temporal feature extraction.

Proposed method

• Curated a comprehensive list of 92 SoTA TSC and manufacturing-specific algorithms from recent literature.
• Selected 36 representative algorithms based on diversity, publication impact, and relevance to industrial applications.
• Collected and curated 22 real-world manufacturing time-series datasets covering various industrial processes and fault types.
• Implemented and evaluated all 36 algorithms using standardized training, validation, and testing protocols across all 22 datasets.
• Applied consistent evaluation metrics, including accuracy, to ensure fair comparison across models.
• Conducted ablation and analysis to identify performance trends based on model architecture, especially convolutional vs. recurrent structures.

Experimental results

Research questions

• RQ1 Which state-of-the-art machine learning and deep learning models achieve the highest classification accuracy on real-world manufacturing time-series datasets?
• RQ2 How do convolutional neural network-based models compare to recurrent neural network-based models in terms of performance on industrial TSC tasks?
• RQ3 What is the relative impact of architectural design—particularly the use of convolutional kernels—on TSC performance in manufacturing contexts?
• RQ4 How robust and scalable are the top-performing models across datasets with varying characteristics such as length, dimensionality, and noise levels?
• RQ5 Which models consistently outperform others across diverse industrial time-series classification problems?

Key findings

• ResNet, DrCIF, InceptionTime, and ARSENAL achieved an average accuracy of over 96.6% across all 22 manufacturing TSC datasets, establishing them as the top-performing models.
• Three of the top four models (ResNet, DrCIF, InceptionTime) rely on convolutional kernels, demonstrating the superior ability of these architectures to extract meaningful temporal features from industrial time-series data.
• LSTM, BiLSTM, and TS-LSTM models showed strong performance, confirming the effectiveness of recurrent architectures in modeling sequential dependencies in time-series data.
• The study confirms that model performance is highly dependent on data characteristics, with convolutional models showing greater robustness across diverse industrial data types.
• ARSENAL, a deep ensemble method, achieved high accuracy by aggregating predictions from multiple estimators, highlighting the value of ensemble learning in industrial TSC.
• The results indicate that architectural choices significantly influence performance, with convolutional models outperforming recurrent models in most cases, particularly in terms of accuracy and scalability.