[Paper Review] Collapsed Variational Bayes Inference of Infinite Relational Model

Katsuhiko Ishiguro, Issei Sato+1 more|arXiv (Cornell University)|Sep 16, 2014

Bayesian Methods and Mixture Models42 references3 citations

TL;DR

This paper proposes Averaged Collapsed Variational Bayes (ACVB) inference for the Infinite Relational Model (IRM), offering a deterministic, fast, and convergence-guaranteed alternative to Gibbs sampling and standard variational Bayes. It introduces CVB0-based hyperparameter updates—including for the Dirichlet process concentration parameter—and enables linear-time inference, significantly improving scalability and practicality for large relational datasets with comparable or better performance than existing methods.

ABSTRACT

The Infinite Relational Model (IRM) is a probabilistic model for relational data clustering that partitions objects into clusters based on observed relationships. This paper presents Averaged CVB (ACVB) solutions for IRM, convergence-guaranteed and practically useful fast Collapsed Variational Bayes (CVB) inferences. We first derive ordinary CVB and CVB0 for IRM based on the lower bound maximization. CVB solutions yield deterministic iterative procedures for inferring IRM given the truncated number of clusters. Our proposal includes CVB0 updates of hyperparameters including the concentration parameter of the Dirichlet Process, which has not been studied in the literature. To make the CVB more practically useful, we further study the CVB inference in two aspects. First, we study the convergence issues and develop a convergence-guaranteed algorithm for any CVB-based inferences called ACVB, which enables automatic convergence detection and frees non-expert practitioners from difficult and costly manual monitoring of inference processes. Second, we present a few techniques for speeding up IRM inferences. In particular, we describe the linear time inference of CVB0, allowing the IRM for larger relational data uses. The ACVB solutions of IRM showed comparable or better performance compared to existing inference methods in experiments, and provide deterministic, faster, and easier convergence detection.

Motivation & Objective

• To address the lack of practical, fast, and convergence-detectable inference methods for the Infinite Relational Model (IRM), which currently relies on slow and hard-to-monitor Gibbs samplers.
• To develop a collapsed variational Bayes (CVB) framework for IRM that improves upon standard variational Bayes by integrating out latent parameters and enabling faster, deterministic inference.
• To introduce a novel convergence-guaranteed algorithm, Averaged CVB (ACVB), which enables automatic convergence detection without manual monitoring, crucial for non-expert users.
• To enhance computational efficiency through cluster shrinkage and linear-time inference, making IRM applicable to larger relational datasets.
• To derive the first CVB0-based update rules for all hyperparameters, including the Dirichlet process concentration parameter, which was previously unexplored in the literature.

Proposed method

• Formulate a lower bound for collapsed variational Bayes (CVB) inference in IRM using Taylor approximations, similar to those used in LDA, to make the intractable bound computable.
• Derive two CVB solutions—ordinary CVB and CVB0—based on the lower bound maximization, with CVB0 minimizing α-divergence and offering improved approximation quality.
• Introduce ACVB, a novel annealing-based technique that averages variational parameters across iterations to guarantee convergence, with a stationary point equivalent to the converged solution.
• Propose a cluster shrinkage technique that reduces the number of active clusters during inference, enabling linear-time complexity and improved scalability.
• Implement linear-time inference for CVB0 by exploiting sparsity and efficient update rules, significantly reducing computational cost.
• Derive CVB0-based update rules for all hyperparameters, including the concentration parameter of the Dirichlet Process, enabling automatic hyperparameter optimization.

Experimental results

Research questions

• RQ1 Can collapsed variational Bayes inference be effectively adapted to the Infinite Relational Model (IRM) to provide faster, deterministic, and more scalable inference than Gibbs sampling?
• RQ2 How can convergence in CVB-based inference for IRM be guaranteed and automatically detected, overcoming the limitations of stochastic samplers?
• RQ3 Can CVB0 be extended to include automatic updates for all hyperparameters, including the Dirichlet process concentration parameter, which has not been previously studied?
• RQ4 To what extent does linear-time inference and cluster shrinkage improve the scalability of IRM for large relational datasets?
• RQ5 How does the proposed ACVB method compare in performance and convergence behavior to existing inference methods on real-world relational data?

Key findings

• The proposed ACVB inference achieves convergence guarantees through a simple annealing mechanism, eliminating the need for manual convergence monitoring required in Gibbs sampling.
• ACVB-based inference shows comparable or better performance than existing methods in relational data modeling tasks, with faster convergence and deterministic computation.
• Linear-time inference for CVB0 enables efficient analysis of large-scale relational data, with computational cost growing sublinearly with data size in practice.
• The cluster shrinkage technique reduces the number of active clusters during inference, leading to faster convergence and lower computational overhead, especially for dense datasets.
• The ACVB0 method detects convergence automatically and reliably, with convergence speed improving by 10x or more when the convergence threshold is relaxed from 0.001% to 0.01% relative change.
• The first derivation of CVB0-based updates for the Dirichlet process concentration parameter enables fully automated hyperparameter optimization, enhancing model adaptability.