[Paper Review] Hierarchical Bayesian Mixture Models for Time Series Using Context Trees as State Space Partitions

Ioannis Papageorgiou, Ioannis Kontoyiannis|arXiv (Cornell University)|Jun 6, 2021

Bayesian Methods and Mixture Models33 references4 citations

TL;DR

This paper proposes a hierarchical Bayesian mixture model for real-valued time series that partitions the state space using context trees derived from recent observations, assigning distinct autoregressive (AR) models to each partition. It enables exact Bayesian inference, including MAP model selection with precise posterior probabilities, outperforming state-of-the-art methods in model selection and forecasting on simulated and real data.

ABSTRACT

A general Bayesian framework is introduced for mixture modelling and inference with real-valued time series. At the top level, the state space is partitioned via the choice of a discrete context tree, so that the resulting partition depends on the values of some of the most recent samples. At the bottom level, a different model is associated with each region of the partition. This defines a very rich and flexible class of mixture models, for which we provide algorithms that allow for efficient, exact Bayesian inference. In particular, we show that the maximum a posteriori probability (MAP) model (including the relevant MAP context tree partition) can be precisely identified, along with its exact posterior probability. The utility of this general framework is illustrated in detail when a different autoregressive (AR) model is used in each state-space region, resulting in a mixture-of-AR model class. The performance of the associated algorithmic tools is demonstrated in the problems of model selection and forecasting on both simulated and real-world data, where they are found to provide results as good or better than state-of-the-art methods.

Motivation & Objective

• To develop a flexible, general Bayesian framework for modeling real-valued time series with non-linear dynamics.
• To address the challenge of selecting appropriate state space partitions for mixture models in time series analysis.
• To enable exact Bayesian inference—specifically, MAP model selection and posterior probability computation—within a rich class of mixture models.
• To demonstrate the effectiveness of the framework in time series forecasting and model selection using autoregressive components.

Proposed method

• The state space is partitioned using a discrete context tree, where each node represents a region defined by a suffix of recent observations.
• A different autoregressive (AR) model is assigned to each region of the partition, forming a mixture-of-AR model.
• Hierarchical Bayesian priors are placed on both the context tree structure and the AR model parameters to enable coherent inference.
• Exact Bayesian inference is achieved through dynamic programming and recursive computation of marginal likelihoods over tree structures.
• The maximum a posteriori (MAP) context tree and associated model parameters are identified via an efficient search algorithm over the space of possible trees.
• Posterior probabilities of the MAP model are computed exactly, enabling rigorous model comparison and selection.

Experimental results

Research questions

• RQ1 Can a hierarchical Bayesian framework with context tree-based state space partitioning enable exact Bayesian inference in time series mixture models?
• RQ2 How does the performance of the proposed mixture-of-AR model compare to state-of-the-art methods in time series forecasting and model selection?
• RQ3 Can the MAP context tree and associated model parameters be identified exactly and efficiently within this framework?
• RQ4 What is the impact of context tree structure on model flexibility and predictive accuracy in real-valued time series?

Key findings

• The proposed framework enables exact Bayesian inference, including precise computation of the maximum a posteriori (MAP) model and its posterior probability.
• The MAP context tree and associated AR models can be identified exactly using an efficient algorithm, avoiding approximation methods.
• On both simulated and real-world time series, the method achieves forecasting and model selection performance at least as good as, and often better than, state-of-the-art approaches.
• The use of context trees as state space partitions allows the model to capture complex, non-linear dependencies in time series data through localized AR dynamics.