[Paper Review] Two-Sample Testing for Event Impacts in Time Series

Erik Scharwächter, Emmanuel Müller|arXiv (Cornell University)|Jan 31, 2020

Time Series Analysis and Forecasting4 citations

TL;DR

This paper proposes EITEST, a non-parametric two-sample testing framework to detect shared statistical information between a time series and event occurrences, without assuming a specific detection model. By testing for distributional differences in time series values at increasing lags after events using multiple two-sample tests (via Kolmogorov-Smirnov or MMD statistics), EITEST identifies informative time series across univariate, multivariate, and non-numeric domains—demonstrating superior or comparable performance to existing methods in simulations and real-world applications on social media and smart home data.

ABSTRACT

In many application domains, time series are monitored to detect extreme events like technical faults, natural disasters, or disease outbreaks. Unfortunately, it is often non-trivial to select both a time series that is informative about events and a powerful detection algorithm: detection may fail because the detection algorithm is not suitable, or because there is no shared information between the time series and the events of interest. In this work, we thus propose a non-parametric statistical test for shared information between a time series and a series of observed events. Our test allows identifying time series that carry information on event occurrences without committing to a specific event detection methodology. In a nutshell, we test for divergences of the value distributions of the time series at increasing lags after event occurrences with a multiple two-sample testing approach. In contrast to related tests, our approach is applicable for time series over arbitrary domains, including multivariate numeric, strings or graphs. We perform a large-scale simulation study to show that it outperforms or is on par with related tests on our task for univariate time series. We also demonstrate the real-world applicability of our approach on datasets from social media and smart home environments.

Motivation & Objective

• To develop a general-purpose statistical test to detect whether a time series carries information about exogenous events, without relying on a specific event detection algorithm.
• To address the challenge of identifying informative time series when the relationship between events and time series is complex or subtle, such as changes in mean, variance, or tail behavior.
• To enable detection of event impacts in arbitrary data domains, including multivariate numeric, string, and graph time series, where visual inspection is infeasible.
• To provide a computationally efficient method scalable to long time series, overcoming limitations of existing independence tests.

Proposed method

• The method reduces the problem of detecting shared information between time series and events to a sequence of multiple two-sample tests, comparing the distribution of time series values at different lags after event occurrences.
• For each lag k ∈ {1, ..., K}, the method tests whether the distribution of X_t given E_{t−k} = 1 differs from the distribution given E_{t−k} = 0, using a non-parametric two-sample test.
• Two variants are proposed: EITEST-KS using the Kolmogorov-Smirnov statistic, and EITEST-MMD using the Maximum Mean Difference (MMD) statistic, which supports arbitrary domains via kernel methods.
• The method controls the family-wise error rate across multiple lags using a closed testing procedure, ensuring statistical validity under multiple comparisons.
• The approach is linear in time series length T, making it scalable to long sequences where other tests fail due to computational cost.
• The method is agnostic to the event detection algorithm, focusing solely on identifying time series that are statistically associated with events.

Experimental results

Research questions

• RQ1 Can a non-parametric two-sample testing framework detect subtle statistical associations between time series and event occurrences without assuming a specific detection model?
• RQ2 How well does the proposed method perform compared to existing causal inference and independence testing methods in detecting event impacts across different types of time series behavior?
• RQ3 Can the method detect event impacts in non-numeric time series such as strings or graphs, and what are the requirements for kernel selection in such cases?
• RQ4 How does the method scale with long time series, and does it maintain statistical power and computational feasibility?

Key findings

• EITEST-KS and EITEST-MMD both reject the null hypothesis of independence between German Twitter activity and global earthquakes with p < .0001, confirming a strong statistical association.
• For earthquakes in China, EITEST-KS and EITEST-MMD do not reject independence (p = .4090 and p = .4225, respectively), suggesting limited public awareness in Germany.
• GC-VAR detects an association with Chinese earthquakes (p = .0090), contradicting the EITEST results and indicating potential false positives in that method.
• TE-KSG produces inconsistent p-values across runs, showing unreliable performance on both tasks.
• In simulations, EITEST outperforms or matches existing methods in detecting all three types of event impacts: mean shifts, variance changes, and tail fattening.
• The method is linear in time series length T, enabling efficient application to long sequences where other tests become computationally infeasible.