[Paper Review] AR-Net: A simple Auto-Regressive Neural Network for time-series

Oskar Triebe, Nikolay Laptev+1 more|arXiv (Cornell University)|Nov 27, 2019

Time Series Analysis and Forecasting22 references50 citations

TL;DR

AR-Net presents a neural-network-based approach that mimics Classic-AR coefficients using a feed-forward network, achieving linear scaling with AR order and enabling sparse, interpretable long-range autoregressive models.

ABSTRACT

In this paper we present a new framework for time-series modeling that combines the best of traditional statistical models and neural networks. We focus on time-series with long-range dependencies, needed for monitoring fine granularity data (e.g. minutes, seconds, milliseconds), prevalent in operational use-cases. Traditional models, such as auto-regression fitted with least squares (Classic-AR) can model time-series with a concise and interpretable model. When dealing with long-range dependencies, Classic-AR models can become intractably slow to fit for large data. Recently, sequence-to-sequence models, such as Recurrent Neural Networks, which were originally intended for natural language processing, have become popular for time-series. However, they can be overly complex for typical time-series data and lack interpretability. A scalable and interpretable model is needed to bridge the statistical and deep learning-based approaches. As a first step towards this goal, we propose modelling AR-process dynamics using a feed-forward neural network approach, termed AR-Net. We show that AR-Net is as interpretable as Classic-AR but also scales to long-range dependencies. Our results lead to three major conclusions: First, AR-Net learns identical AR-coefficients as Classic-AR, thus being equally interpretable. Second, the computational complexity with respect to the order of the AR process, is linear for AR-Net as compared to a quadratic for Classic-AR. This makes it possible to model long-range dependencies within fine granularity data. Third, by introducing regularization, AR-Net automatically selects and learns sparse AR-coefficients. This eliminates the need to know the exact order of the AR-process and allows to learn sparse weights for a model with long-range dependencies.

Motivation & Objective

• Bridge traditional AR models and neural networks to model time-series with long-range dependencies.
• Provide an interpretable, scalable alternative to Classic-AR for high-order autoregressive processes.
• Enable sparse AR-coefficient learning to avoid needing the exact AR order.
• Demonstrate that AR-Net learns AR-coefficients identically to Classic-AR while offering computational benefits.

Proposed method

• Model AR dynamics with a feed-forward neural network whose first layer weights correspond to AR-coefficients.
• Use Mean Squared Error as the training objective to align with Classic-AR fitting.
• Introduce a sparsity-oriented regularization to encourage zeroing of inconsequential coefficients.
• Compare AR-Net to Classic-AR on synthetic data generated from noisy AR processes.
• Evaluate both coefficient precision (sTPE) and one-step-ahead forecasting MSE.
• Discuss linear vs quadratic complexity in AR-order fitting between AR-Net and Classic-AR.

Experimental results

Research questions

• RQ1 Can AR-Net learn AR-coefficients identically to Classic-AR when fitted on the same data?
• RQ2 Does AR-Net scale linearly with AR order and enable long-range dependencies without intractable computation?
• RQ3 Can regularization induce sparse AR-coefficients without sacrificing forecasting accuracy?
• RQ4 How does AR-Net perform on small datasets or when the true AR order is unknown?
• RQ5 What is the computational advantage of AR-Net relative to least-squares AR as p grows?

Key findings

• AR-Net learns AR-coefficients identically to Classic-AR and achieves similar one-step-ahead MSE.
• AR-Net’s training complexity scales linearly with AR order p, while Classic-AR scales quadratically, making AR-Net preferable for long-range dependencies.
• Regularization enables sparse AR-coefficients, effective even when true sparsity is very high (up to 0.003 sparsity in experiments).
• In sparse setups, AR-Net yields superior sTPE precision compared to Classic-AR, while forecasting performance remains near-identical.
• AR-Net remains effective on small datasets and strong noise, with comparable MSE but better coefficient sparsity than Classic-AR.