[Paper Review] On the Model Shrinkage Effect of Gamma Process Edge Partition Models

Iku Ohama, Issei Sato+2 more|arXiv (Cornell University)|Sep 26, 2017

Bayesian Methods and Mixture Models10 references3 citations

TL;DR

This paper identifies a critical flaw in the model shrinkage mechanism of the Edge Partition Model (EPM) using a gamma process prior, where gamma hyperpriors disrupt automatic atom pruning. To resolve this, the authors propose two improved models—CEPM with constrained gamma priors and DEPM with Dirichlet priors—enabling full marginalization of infinite atoms, leading to a truly infinite IDEPM model that achieves state-of-the-art performance in link prediction, convergence speed, and mixing efficiency via collapsed Gibbs sampling.

ABSTRACT

The edge partition model (EPM) is a fundamental Bayesian nonparametric model for extracting an overlapping structure from binary matrix. The EPM adopts a gamma process ($Γ$P) prior to automatically shrink the number of active atoms. However, we empirically found that the model shrinkage of the EPM does not typically work appropriately and leads to an overfitted solution. An analysis of the expectation of the EPM's intensity function suggested that the gamma priors for the EPM hyperparameters disturb the model shrinkage effect of the internal $Γ$P. In order to ensure that the model shrinkage effect of the EPM works in an appropriate manner, we proposed two novel generative constructions of the EPM: CEPM incorporating constrained gamma priors, and DEPM incorporating Dirichlet priors instead of the gamma priors. Furthermore, all DEPM's model parameters including the infinite atoms of the $Γ$P prior could be marginalized out, and thus it was possible to derive a truly infinite DEPM (IDEPM) that can be efficiently inferred using a collapsed Gibbs sampler. We experimentally confirmed that the model shrinkage of the proposed models works well and that the IDEPM indicated state-of-the-art performance in generalization ability, link prediction accuracy, mixing efficiency, and convergence speed.

Motivation & Objective

• To diagnose why the model shrinkage effect in the Edge Partition Model (EPM) fails to work properly in practice.
• To identify the root cause: gamma priors on EPM hyperparameters that interfere with the internal gamma process's shrinkage mechanism.
• To propose new generative constructions—CEPM and DEPM—that restore proper model shrinkage by redefining prior dependencies.
• To develop a truly infinite, fully marginalized version of DEPM (IDEPM) that enables efficient inference via collapsed Gibbs sampling.
• To empirically validate that the proposed models achieve superior generalization, link prediction accuracy, mixing efficiency, and convergence speed.

Proposed method

• Propose CEPM with constrained gamma priors to decouple hyperprior influence from the internal gamma process, ensuring shrinkage depends only on the gamma process.
• Introduce DEPM by replacing gamma hyperpriors with Dirichlet priors on the EPM's latent factors, enabling full conjugacy and marginalization.
• Derive a closed-form marginal likelihood for DEPM by integrating out all parameters, including infinite atoms of the gamma process, leading to IDEPM.
• Implement a collapsed Gibbs sampler for IDEPM by marginalizing out all model parameters, including the infinite atoms and hyperparameters.
• Use conjugate priors and conditional distributions to enable efficient posterior inference, including sampling for hyperparameters α₁, α₂, γ₀, and c₀.
• Leverage the l₁-constraint on factor matrices to simplify the marginal likelihood and ensure tractability in the infinite limit.

Experimental results

Research questions

• RQ1 Why does the standard EPM fail to achieve proper model shrinkage despite using a gamma process prior?
• RQ2 How do gamma priors on EPM hyperparameters interfere with the shrinkage effect of the internal gamma process?
• RQ3 Can a modified EPM construction ensure that the expected number of active atoms is finite and data-adaptive?
• RQ4 Is it possible to fully marginalize the infinite atoms of the gamma process in the EPM framework?
• RQ5 Does the resulting truly infinite model (IDEPM) outperform existing models in inference efficiency and predictive performance?

Key findings

• The EPM's model shrinkage fails due to interference from gamma hyperpriors, which distort the expected number of active atoms by multiplying with the gamma process intensity.
• The expectation of the EPM’s intensity function depends on both the gamma process and hyperpriors, making it impossible to guarantee finite active atoms.
• The proposed CEPM and DEPM models restore proper shrinkage by ensuring the intensity function depends only on the gamma process prior.
• The DEPM allows full marginalization of all parameters, including infinite atoms, enabling a truly infinite model (IDEPM) with a closed-form marginal likelihood.
• The IDEPM model achieves state-of-the-art performance in link prediction accuracy, generalization ability, mixing efficiency, and convergence speed on synthetic and real-world data.
• The collapsed Gibbs sampler for IDEPM efficiently infers all latent variables and hyperparameters, including γ₀ and c₀, via conjugate posterior updates.