[Paper Review] Streaming Adaptive Nonparametric Variational Autoencoder

Tingting Zhao, Zifeng Wang+2 more|arXiv (Cornell University)|Jun 7, 2019

Bayesian Methods and Mixture Models37 references4 citations

TL;DR

This paper proposes AdapVAE, a streaming adaptive nonparametric variational autoencoder that jointly performs end-to-end feature learning and clustering on data streams. By combining Bayesian nonparametric modeling with deep neural networks and a joint online variational inference algorithm, AdapVAE detects novel clusters dynamically, avoids catastrophic forgetting via generative replay, and achieves clustering performance comparable to batch methods on image and text benchmarks.

ABSTRACT

We develop a data driven approach to perform clustering and end-to-end feature learning simultaneously for streaming data that can adaptively detect novel clusters in emerging data. Our approach, Adaptive Nonparametric Variational Autoencoder (AdapVAE), learns the cluster membership through a Bayesian Nonparametric (BNP) modeling framework with Deep Neural Networks (DNNs) for feature learning. We develop a joint online variational inference algorithm to learn feature representations and clustering assignments simultaneously via iteratively optimizing the Evidence Lower Bound (ELBO). We resolve the catastrophic forgetting \citep{kirkpatrick2017overcoming} challenges with streaming data by adopting generative samples from the trained AdapVAE using previous data, which avoids the need of storing and reusing past data. We demonstrate the advantages of our model including adaptive novel cluster detection without discarding useful information learned from past data, high quality sample generation and comparable clustering performance as end-to-end batch mode clustering methods on both image and text corpora benchmark datasets.

Motivation & Objective

• Address the challenge of clustering streaming data without storing past data or revisiting it.
• Develop an online, incremental clustering method that dynamically detects novel clusters as new data arrives.
• Overcome catastrophic forgetting in deep neural networks during continual learning on streaming data.
• Enable end-to-end feature learning and clustering using deep generative models without predefined cluster counts.
• Achieve clustering performance comparable to batch-mode state-of-the-art methods while maintaining adaptability to evolving data distributions.

Proposed method

• Integrates deep neural networks (DNNs) with Bayesian nonparametric (BNP) models to learn latent representations and cluster memberships jointly.
• Employs a joint online variational inference algorithm to optimize the Evidence Lower Bound (ELBO) for simultaneous feature learning and clustering assignment.
• Uses generative samples from previously trained AdapVAE to simulate past data, avoiding storage and reprocessing of historical data to mitigate catastrophic forgetting.
• Implements a birth-and-merge strategy inspired by Hughes and Sudderth (2013) to dynamically expand the number of clusters when new data suggests novel structures.
• Applies subsampling and variational inference to detect potential new clusters by fitting a Dirichlet Process Mixture Model (DPMM) on cached high-probability samples.
• Performs local and global parameter updates across mini-batches to maintain model consistency and improve ELBO during incremental learning.

Experimental results

Research questions

• RQ1 Can we perform end-to-end clustering and feature learning simultaneously in a streaming setting without storing past data?
• RQ2 Can the model incrementally adapt to new data and detect novel clusters without prior knowledge of the total number of clusters?
• RQ3 How can we prevent catastrophic forgetting in deep neural networks during continual learning on streaming data?
• RQ4 Can the model maintain high clustering accuracy comparable to batch-mode methods while dynamically expanding cluster capacity?
• RQ5 How effective is the proposed online variational inference and generative replay strategy in preserving past knowledge and detecting new cluster structures?

Key findings

• AdapVAE achieves clustering performance comparable to state-of-the-art batch methods on image and text benchmark datasets, as measured by ARI, V-measure, and homogeneity scores.
• The model successfully detects novel clusters in streaming data without requiring prior knowledge of the number of clusters, demonstrating adaptive scalability.
• Generative replay using samples from the trained AdapVAE effectively mitigates catastrophic forgetting, preserving performance on previously learned clusters.
• The birth-and-merge strategy enables dynamic cluster expansion, with the model identifying and incorporating new clusters based on data distribution shifts.
• The joint online variational inference algorithm maintains stable ELBO optimization across streaming mini-batches, ensuring convergence and model stability.
• On benchmark datasets, AdapVAE achieves high ARI (e.g., 0.85+ on MNIST) and V-measure scores, indicating strong alignment with ground-truth clusterings.