[Paper Review] Discriminative Nonparametric Latent Feature Relational Models with Data Augmentation

Bei Chen, Ning Chen+3 more|arXiv (Cornell University)|Dec 7, 2015

Bayesian Methods and Mixture Models32 references3 citations

TL;DR

This paper proposes discriminative nonparametric latent feature relational models (DLFRM) that use data augmentation and Gibbs sampling to enable scalable, end-to-end link prediction in large networks with unknown latent feature dimensions. By integrating regularized Bayesian inference with both logistic and hinge losses, the method handles class imbalance and achieves state-of-the-art performance on real-world networks with up to hundreds of thousands of entities and millions of links.

ABSTRACT

We present a discriminative nonparametric latent feature relational model (LFRM) for link prediction to automatically infer the dimensionality of latent features. Under the generic RegBayes (regularized Bayesian inference) framework, we handily incorporate the prediction loss with probabilistic inference of a Bayesian model; set distinct regularization parameters for different types of links to handle the imbalance issue in real networks; and unify the analysis of both the smooth logistic log-loss and the piecewise linear hinge loss. For the nonconjugate posterior inference, we present a simple Gibbs sampler via data augmentation, without making restricting assumptions as done in variational methods. We further develop an approximate sampler using stochastic gradient Langevin dynamics to handle large networks with hundreds of thousands of entities and millions of links, orders of magnitude larger than what existing LFRM models can process. Extensive studies on various real networks show promising performance.

Motivation & Objective

• Address the challenge of unknown latent feature dimensionality in link prediction without relying on model selection or fixed K.
• Handle class imbalance in real-world networks by assigning distinct regularization parameters to positive and negative links.
• Unify the analysis of both smooth logistic log-loss and non-smooth hinge loss within a single regularized Bayesian inference framework.
• Develop an efficient Gibbs sampler via data augmentation to avoid restrictive assumptions of variational methods.
• Scale the method to large networks using stochastic gradient Langevin dynamics (SGLD) for approximate inference.

Proposed method

• Formulate link prediction as a discriminative Bayesian model using the RegBayes framework to combine prediction loss with posterior inference.
• Apply data augmentation to transform the non-conjugate posterior inference into a conditionally conjugate form, enabling simple Gibbs sampling.
• Use the Indian Buffet Process (IBP) as a nonparametric prior over latent features to allow automatic discovery of feature dimensionality.
• Incorporate separate regularization parameters $c^+$ and $c^-$ for positive and negative links to mitigate data imbalance.
• Develop a stochastic approximate Gibbs sampler using SGLD to scale to networks with 100k+ entities and 1M+ links.
• Unify the treatment of logistic log-loss and hinge loss under the same framework via regularized inference.

Experimental results

Research questions

• RQ1 Can a nonparametric latent feature model automatically infer the correct number of latent features without prior specification?
• RQ2 How can class imbalance in real-world networks be effectively addressed in latent feature models for link prediction?
• RQ3 Can both smooth (logistic) and non-smooth (hinge) loss functions be unified under a single inference framework for discriminative link prediction?
• RQ4 Can data augmentation enable efficient, non-truncated Gibbs sampling without restrictive mean-field or truncation assumptions?
• RQ5 Can the proposed method scale to large-scale networks with hundreds of thousands of entities and millions of links?

Key findings

• The proposed DLFRM with data augmentation and Gibbs sampling converges quickly and stably, with test AUC stabilizing within 150 burn-in steps.
• Setting $c^+ / c^-$ to 10 significantly improves AUC on imbalanced networks compared to equal regularization, demonstrating effective handling of imbalance.
• The model automatically learns a stable number of latent features, with low variance across independent runs, indicating robustness to initialization.
• The stochastic variant (stoDLFRM l) achieves the best AUC performance across all datasets, outperforming all baselines including proximity-based methods and other LFRMs.
• The diagonal variant (diagDLFRM l) achieves competitive performance with significantly reduced training time due to linear scaling in feature dimension K.
• The method scales to networks with up to 100k entities and 1M+ links, orders of magnitude larger than what previous LFRM models can handle.