[Paper Review] Mixed Membership Models for Time Series

Emily B. Fox, Michael I. Jordan|arXiv (Cornell University)|Sep 13, 2013

Bayesian Methods and Mixture Models30 references3 citations

TL;DR

This paper introduces Bayesian nonparametric mixed membership models for collections of time series, framing each time series as an entity with partial membership in a shared library of dynamic regimes (e.g., behaviors or states). It proposes two frameworks—Dirichlet-multinomial and beta-Bernoulli—enabling flexible, sparse, and shared learning across multiple time series, with key contributions in modeling structural changes and regime overlaps without pre-specifying the number of states.

ABSTRACT

In this article we discuss some of the consequences of the mixed membership perspective on time series analysis. In its most abstract form, a mixed membership model aims to associate an individual entity with some set of attributes based on a collection of observed data. Although much of the literature on mixed membership models considers the setting in which exchangeable collections of data are associated with each member of a set of entities, it is equally natural to consider problems in which an entire time series is viewed as an entity and the goal is to characterize the time series in terms of a set of underlying dynamic attributes or "dynamic regimes". Indeed, this perspective is already present in the classical hidden Markov model, where the dynamic regimes are referred to as "states", and the collection of states realized in a sample path of the underlying process can be viewed as a mixed membership characterization of the observed time series. Our goal here is to review some of the richer modeling possibilities for time series that are provided by recent developments in the mixed membership framework.

Motivation & Objective

• To reframe time series analysis through the lens of mixed membership models, treating each time series as an entity with partial membership in a shared set of dynamic regimes.
• To address the challenge of modeling multiple time series with overlapping dynamic behaviors while avoiding pre-specification of the number of states.
• To extend traditional hidden Markov models and switching state-space models by enabling information sharing across time series via hierarchical priors.
• To develop nonparametric frameworks that allow for an unbounded number of dynamic regimes, adapting to data complexity without model selection.
• To enable decoupling of regime presence from regime prevalence through a beta-Bernoulli formulation, supporting sparse and interpretable membership structures.

Proposed method

• Adapts the Dirichlet-multinomial framework from topic modeling to time series, using a global prior over transition and emission parameters shared across time series.
• Proposes a beta-Bernoulli framework where each time series independently selects a subset of dynamic regimes from a global library before generating its state sequence.
• Employs Bayesian nonparametric priors (e.g., Dirichlet process) to allow an infinite number of potential dynamic regimes, with the actual number determined by the data.
• Uses Markov switching processes as the core mechanism, where state transitions follow a discrete-time Markov process with time-series-specific transition distributions.
• Applies dynamic programming techniques (e.g., from Fearnhead, 2006) to efficiently explore changepoints in multivariate time series dependency structures.
• Integrates time-varying mixture weights in autoregressive models to allow nonstationary processes, with components selected via Dirichlet process mixtures or kernel-based weighting based on past observations.

Experimental results

Research questions

• RQ1 How can mixed membership models be adapted to time series data to allow for partial, shared, and dynamic regime memberships across multiple time series?
• RQ2 What are the implications of using a nonparametric Bayesian framework for modeling the number of dynamic regimes in time series without pre-specifying it?
• RQ3 How can the beta-Bernoulli framework improve modeling sparsity and decouple regime presence from prevalence in time series membership?
• RQ4 In what ways can hierarchical priors over transition and emission parameters enable information sharing across multiple time series?
• RQ5 How can changepoint detection and structural break modeling be enhanced through product partition models and dynamic programming in multivariate time series?

Key findings

• The proposed nonparametric mixed membership models allow for an unbounded number of dynamic regimes, with the actual number learned from data rather than pre-specified.
• The beta-Bernoulli framework enables sparse time series membership, where only a subset of the global regime library is used per time series, supporting interpretable and efficient modeling.
• The Dirichlet-multinomial approach facilitates information sharing across time series by pooling transition and emission parameters through a global prior distribution.
• Empirical results show that the models can effectively discover shared dynamic behaviors (e.g., exercise types in motion data) across individuals, even with partial overlap in routines.
• The integration of dynamic programming techniques enables efficient changepoint detection in multivariate time series, particularly when dependency structures evolve over time.
• The models outperform classical HMMs and fixed-state models in capturing overlapping and heterogeneous dynamic regimes across multiple time series, especially in high-dimensional or sparse settings.