[Paper Review] Maximum likelihood estimation of a finite mixture of logistic regression models in a continuous data stream

Maurits Kaptein, P.E. Ketelaar|arXiv (Cornell University)|Feb 28, 2018

Bayesian Methods and Mixture Models4 citations

TL;DR

This paper proposes oFMLR, an online maximum likelihood estimation method for finite mixtures of logistic regression models designed to cluster customers in real-time from continuous data streams. By adapting the EM algorithm for streaming data, oFMLR enables interpretable, scalable segmentation of dichotomous responses—such as click-through behavior—while maintaining computational efficiency and model interpretability in high-velocity marketing data environments.

ABSTRACT

In marketing we are often confronted with a continuous stream of responses to marketing messages. Such streaming data provide invaluable information regarding message effectiveness and segmentation. However, streaming data are hard to analyze using conventional methods: their high volume and the fact that they are continuously augmented means that it takes considerable time to analyze them. We propose a method for estimating a finite mixture of logistic regression models which can be used to cluster customers based on a continuous stream of responses. This method, which we coin oFMLR, allows segments to be identified in data streams or extremely large static datasets. Contrary to black box algorithms, oFMLR provides model estimates that are directly interpretable. We first introduce oFMLR, explaining in passing general topics such as online estimation and the EM algorithm, making this paper a high level overview of possible methods of dealing with large data streams in marketing practice. Next, we discuss model convergence, identifiability, and relations to alternative, Bayesian, methods; we also identify more general issues that arise from dealing with continuously augmented data sets. Finally, we introduce the oFMLR [R] package and evaluate the method by numerical simulation and by analyzing a large customer clickstream dataset.

Motivation & Objective

• Address the challenge of analyzing high-velocity, high-volume customer clickstream data in marketing, which traditional batch methods struggle to handle due to computational and temporal constraints.
• Develop a method that enables real-time clustering of customers based on binary responses (e.g., clicks) using finite mixture models, suitable for continuous data streams.
• Provide a model that maintains interpretability—unlike black-box machine learning methods—by leveraging the well-understood framework of logistic regression.
• Enable continuous model updating and policy adaptation through online estimation, supporting dynamic marketing decision-making.
• Offer a practical, open-source R package (oFMLR) to facilitate adoption by marketing researchers and practitioners for real-world data stream analysis.

Proposed method

• Adapt the Expectation-Maximization (EM) algorithm for online, or streaming, estimation by updating model parameters incrementally as new data points arrive.
• Use stochastic gradient descent (SGD) principles to approximate the M-step of EM, enabling efficient parameter updates without reprocessing the entire dataset.
• Implement a learning rate schedule that allows the model to 'forget' older data, supporting adaptation to non-stationary data streams.
• Apply online moving averages of the log-likelihood (Eq. 15) as a convergence diagnostic to monitor model stability during streaming estimation.
• Use online bootstrap methods (e.g., Owen & Eckles, 2012) to quantify uncertainty in estimated parameters and assess model robustness.
• Integrate model selection via multiple runs with different numbers of components (K), using information criteria or likelihood-based diagnostics to identify optimal segmentations.

Experimental results

Research questions

• RQ1 Can a finite mixture of logistic regression models be effectively estimated in an online, streaming fashion to support real-time customer segmentation?
• RQ2 How does the performance of the online EM algorithm (oFMLR) compare to batch estimation in terms of convergence, accuracy, and computational efficiency on streaming data?
• RQ3 To what extent can oFMLR identify meaningful, interpretable customer segments in high-velocity clickstream data without relying on domain-specific assumptions?
• RQ4 What diagnostic tools are effective for monitoring convergence and model stability during online estimation of mixture models?
• RQ5 How can uncertainty in model parameters and cluster assignments be quantified in an online setting to support responsible model interpretation?

Key findings

• The oFMLR method successfully enables real-time estimation of finite mixture logistic regression models on continuous data streams, reducing computation time significantly compared to batch methods.
• Numerical simulations demonstrate that oFMLR converges to parameter estimates close to those obtained via batch EM, validating its accuracy under controlled conditions.
• The method maintains interpretability by relying on logistic regression components, allowing researchers to directly interpret segment-specific effects and covariate impacts.
• Model convergence and stability are sensitive to initial parameter values and learning rate choices, highlighting the need for careful initialization and diagnostic monitoring.
• The online moving average of the log-likelihood serves as a useful convergence diagnostic, with stable values indicating model convergence during streaming estimation.
• Empirical analysis on a large clickstream dataset confirms oFMLR’s ability to identify distinct customer response patterns, supporting actionable segmentation for marketing policy.