[Paper Review] Time Series Data Cleaning with Regular and Irregular Time Intervals

Xi Wang, Chen Wang|arXiv (Cornell University)|Apr 17, 2020

Data Quality and Management137 references4 citations

TL;DR

This survey presents a comprehensive classification of time series data cleaning techniques, focusing on observed value errors in both regular and irregular time intervals. It reviews state-of-the-art methods for single-point big/small errors and continuous errors, emphasizing automated cleaning over manual or discard-based approaches, and identifies key future directions including multidimensional cleaning, domain-agnostic anomaly detection, and specialized tools for industrial and blockchain applications.

ABSTRACT

Errors are prevalent in time series data, especially in the industrial field. Data with errors could not be stored in the database, which results in the loss of data assets. Handling the dirty data in time series is non-trivial, when given irregular time intervals. At present, to deal with these time series containing errors, besides keeping original erroneous data, discarding erroneous data and manually checking erroneous data, we can also use the cleaning algorithm widely used in the database to automatically clean the time series data. This survey provides a classification of time series data cleaning techniques and comprehensively reviews the state-of-the-art methods of each type. In particular, we have a special focus on the irregular time intervals. Besides we summarize data cleaning tools, systems and evaluation criteria from research and industry. Finally, we highlight possible directions time series data cleaning.

Motivation & Objective

• To address the growing challenge of time series data errors in industrial and financial applications, where sensor and system limitations lead to unreliable data.
• To classify observed value errors into three types: single-point big errors, single-point small errors, and continuous errors, enabling targeted cleaning strategies.
• To review and systematize existing automated data cleaning techniques for time series, especially in the context of irregular time intervals where traditional database methods fall short.
• To identify gaps in current tools and evaluation criteria, and to propose future research directions for scalable, intelligent, and application-specific cleaning solutions.

Proposed method

• Classifying time series errors into three categories: single-point big errors (large deviation at isolated points), single-point small errors (minor deviation at isolated points), and continuous errors (multiple consecutive erroneous points).
• Surveying and analyzing existing automated cleaning algorithms, including statistical, machine learning, and model-based approaches, with emphasis on their applicability to irregular time intervals.
• Evaluating the performance of cleaning methods using domain-specific criteria, such as accuracy, computational efficiency, and robustness to noise and gaps.
• Highlighting the limitations of current relational database-centric tools for time series, and advocating for new tools leveraging stream processing, distributed computing, and real-time data pipelines.
• Proposing the integration of advanced machine learning techniques into cleaning algorithms to improve accuracy and adaptability, especially in low-data or weak-knowledge scenarios.
• Designing anomaly detection algorithms tailored for time series that can identify errors without relying on labeled ground truth, enabling proactive cleaning in industrial deployments.

Experimental results

Research questions

• RQ1 How can time series data errors be systematically classified to support targeted cleaning strategies, especially in the presence of irregular time intervals?
• RQ2 What are the strengths and limitations of existing automated cleaning algorithms for different types of observed value errors in time series?
• RQ3 How can machine learning and statistical models be leveraged to improve the accuracy and scalability of time series data cleaning in real-world industrial settings?
• RQ4 What are the key challenges in designing efficient, distributed, and real-time time series cleaning tools that go beyond traditional relational database systems?
• RQ5 How can anomaly detection be enhanced to identify errors in time series when ground truth is unavailable or limited, particularly in domains with weak domain knowledge?

Key findings

• The correct rate of stock data from Yahoo Finance is only 93%, highlighting the prevalence of observed value errors in real-world time series.
• Discarding erroneous data is common but suboptimal, as it leads to loss of valuable data assets, especially when errors are sparse or localized.
• Manual cleaning is accurate but impractical due to high time and labor costs, with up to 30–80% of data warehousing project time spent on cleaning.
• Existing data cleaning tools are largely unsuitable for time series due to their focus on relational databases, creating a need for specialized time series cleaning systems.
• Future cleaning algorithms should support multidimensional time series and integrate machine learning to improve robustness and generalization, especially in low-data regimes.
• Anomaly detection must be reimagined for time series with weak supervision, enabling automated error identification without relying on labeled true values.