[Paper Review] A Transformer-based Framework For Multi-variate Time Series: A Remaining Useful Life Prediction Use Case

Oluwaseyi Ogunfowora, Homayoun Najjaran|arXiv (Cornell University)|Aug 19, 2023

Currency Recognition and Detection4 citations

TL;DR

This paper proposes a native encoder-transformer framework for multi-variate time series forecasting with a focus on remaining useful life (RUL) prediction. It introduces an expanding window data preparation method and optimizes normalization and positional encoding, achieving a 137.65% average performance improvement over the next-best SOTA model on the C-MAPSS benchmark datasets.

ABSTRACT

In recent times, Large Language Models (LLMs) have captured a global spotlight and revolutionized the field of Natural Language Processing. One of the factors attributed to the effectiveness of LLMs is the model architecture used for training, transformers. Transformer models excel at capturing contextual features in sequential data since time series data are sequential, transformer models can be leveraged for more efficient time series data prediction. The field of prognostics is vital to system health management and proper maintenance planning. A reliable estimation of the remaining useful life (RUL) of machines holds the potential for substantial cost savings. This includes avoiding abrupt machine failures, maximizing equipment usage, and serving as a decision support system (DSS). This work proposed an encoder-transformer architecture-based framework for multivariate time series prediction for a prognostics use case. We validated the effectiveness of the proposed framework on all four sets of the C-MAPPS benchmark dataset for the remaining useful life prediction task. To effectively transfer the knowledge and application of transformers from the natural language domain to time series, three model-specific experiments were conducted. Also, to enable the model awareness of the initial stages of the machine life and its degradation path, a novel expanding window method was proposed for the first time in this work, it was compared with the sliding window method, and it led to a large improvement in the performance of the encoder transformer model. Finally, the performance of the proposed encoder-transformer model was evaluated on the test dataset and compared with the results from 13 other state-of-the-art (SOTA) models in the literature and it outperformed them all with an average performance increase of 137.65% over the next best model across all the datasets.

Motivation & Objective

• To develop a robust, end-to-end transformer-based framework for multi-variate time series prediction in prognostics.
• To adapt the native transformer architecture—originally designed for NLP—effectively for time series data with non-i.i.d. and non-stationary characteristics.
• To improve RUL prediction accuracy by addressing challenges such as auto-correlation, non-stationarity, and complex degradation patterns in multi-variate sensor data.
• To evaluate the model’s performance on the standard C-MAPSS benchmark datasets for turbofan engine degradation.
• To identify optimal hyperparameters and data preprocessing strategies (e.g., normalization, positional encoding) for transformer-based time series modeling.

Proposed method

• Proposes a pure encoder-transformer architecture without additional attention heads or residual connections, leveraging self-attention to model long-range dependencies in multi-variate time series.
• Introduces a novel expanding window method for data preparation that enhances model awareness of early degradation stages and improves performance over traditional sliding window approaches.
• Employs a clustering-based normalization technique to extract relevant features and improve data stationarity across multi-variate sensor channels.
• Compares layer normalization versus batch normalization, and fixed versus learnable positional encodings to optimize transformer performance on time series.
• Applies three data transformation methods (min-max scaling, z-score, and clustering-based) to assess their impact on model generalization and prediction accuracy.
• Uses the C-MAPSS dataset with four distinct engine degradation scenarios (FD001–FD004) for comprehensive evaluation under standardized conditions.

Experimental results

Research questions

• RQ1 Can a native encoder-transformer architecture achieve state-of-the-art performance in multi-variate time series RUL prediction without architectural augmentation?
• RQ2 How does the expanding window data preparation method compare to the sliding window method in improving transformer model performance on RUL prediction tasks?
• RQ3 What is the optimal combination of normalization, positional encoding, and input transformation for transformer models in time series forecasting?
• RQ4 Does the proposed framework outperform existing SOTA models across diverse degradation patterns in the C-MAPSS benchmark?
• RQ5 Why does the expanding window method show limited improvement on FD001 and FD003 datasets despite strong gains on others?

Key findings

• The proposed encoder-transformer model achieved an average RUL prediction score of 444.14 across all C-MAPSS datasets, outperforming the next-best model by 137.65% on average.
• The model achieved an average RMSE of 13.66, significantly lower than the next-best model (MSTformer) with 14.755 RMSE.
• The expanding window method led to a substantial performance improvement over the sliding window method, particularly on FD002 and FD004, where it reduced RMSE by over 10%.
• Learnable positional encodings and layer normalization outperformed fixed positional encodings and batch normalization in the transformer architecture for this time series task.
• The clustering-based normalization technique enhanced feature representation and contributed to improved model generalization across different engine degradation scenarios.
• The model showed robust performance across all four C-MAPSS datasets, with the most consistent predictions on FD002 and FD004, where early predictions were more accurate and less penalized.