[Paper Review] Detecting Nonlinearity in Data with Long Coherence Times

James Theiler, Paul S. Linsay+1 more|ArXiv.org|Feb 26, 1993

Complex Systems and Time Series Analysis1 references49 citations

TL;DR

This paper identifies that long coherence times in time series data can lead to false detection of nonlinearity using standard surrogate data and predictor comparison methods. The authors demonstrate analytically and numerically that linear processes with long coherence times—such as near-unity ARMA models—can mimic nonlinear behavior, undermining the reliability of common nonlinearity tests, and caution against interpreting such signals as evidence of chaos or nonlinear dynamics.

ABSTRACT

We consider the limitations of two techniques for detecting nonlinearity in time series. The first technique compares the original time series to an ensemble of surrogate time series that are constructed to mimic the linear properties of the original. The second technique compares the forecasting error of linear and nonlinear predictors. Both techniques are found to be problematic when the data has a long coherence time; they tend to indicate nonlinearity even for linear time series. We investigate the causes of these difficulties both analytically and with numerical experiments on ``real'' and computer-generated data. In particular, although we do see some initial evidence for nonlinear structure in the SFI dataset E, we are inclined to dismiss this evidence as an artifact of the long coherence time.

Motivation & Objective

• . The paper investigates why standard nonlinearity detection techniques fail when applied to time series with long coherence times.
• . It identifies that linear processes with long coherence times can produce spurious nonlinear signatures in standard tests.
• . The objective is to clarify the limitations of surrogate data and predictor comparison methods under long-coherence conditions.
• . The authors aim to provide practical guidelines for distinguishing true nonlinearity from artifacts caused by long coherence times.
• . They seek to improve the reliability of nonlinear time series analysis in real-world data with finite, noisy, and coherent measurements.

Proposed method

• . The authors compare two nonlinearity detection techniques: surrogate data generation and linear vs. nonlinear predictor comparison.
• . Surrogate data are generated using two approaches: Fourier transform-based methods and iterative amplitude-adjusted Fourier transform (IAFT) methods.
• . The study evaluates these methods on both synthetic data and real-world datasets, including the SFI competition dataset E.dat.
• . The authors analyze the auto-correlation structure of time series to assess coherence time and its impact on statistical testing.
• . They use numerical experiments to simulate linear processes with long coherence times and test their performance under standard nonlinearity detection protocols.
• . The analysis includes plotting auto-correlation envelopes and assessing the decay of correlation over time to quantify coherence.

Experimental results

Research questions

• RQ1 . Why do standard nonlinearity detection methods falsely identify linear time series with long coherence times as nonlinear?
• RQ2 . To what extent do surrogate data generation methods fail to preserve the linear properties of time series with long coherence times?
• RQ3 . Can long coherence times in linear processes produce statistically significant nonlinear signatures in standard tests?
• RQ4 . How does the choice of coherence time threshold (e.g., 5% of data length) affect the reliability of nonlinearity detection?
• RQ5 . What criteria should be used to distinguish true nonlinearity from artifacts induced by long coherence in real-world time series?

Key findings

• . Linear time series with long coherence times—such as near-unity ARMA(2,2) models—can produce spurious nonlinear signatures in standard nonlinearity tests.
• . Surrogate data generation methods, particularly IAFT, struggle to accurately mimic the linear properties of time series with long coherence times, leading to false nonlinearity detection.
• . The auto-correlation function of long-coherence linear processes decays slowly, often remaining significant for timescales comparable to the data length, violating the stationarity assumption of standard tests.
• . The SFI dataset E.dat shows apparent nonlinearity, but this is likely an artifact of long coherence time rather than true nonlinear dynamics.
• . The study finds that nonlinearity detection is unreliable when the coherence time is a significant fraction of the data length, especially when auto-correlation remains above 0.05 for T > 0.1N.
• . The authors conclude that long coherence times can fool both surrogate data and predictor comparison methods into indicating nonlinearity, even in purely linear processes.