[Paper Review] Composite Likelihood Estimation for Restricted Boltzmann machines

Muneki Yasuda, S. Kataoka+2 more|arXiv (Cornell University)|Jun 24, 2014

Machine Learning and Algorithms3 citations

TL;DR

This paper proposes a systematic composite likelihood (CL) method for training Restricted Boltzmann Machines (RBMs) by using all k-variable blocks in increasing order of k, ensuring the CL monotonically approaches the true log-likelihood. The method improves estimation accuracy over pseudo-likelihood while remaining computationally feasible, with numerical results showing third-order CL matching exact maximum likelihood performance after 50k iterations.

ABSTRACT

Learning the parameters of graphical models using the maximum likelihood estimation is generally hard which requires an approximation. Maximum composite likelihood estimations are statistical approximations of the maximum likelihood estimation which are higher-order generalizations of the maximum pseudo-likelihood estimation. In this paper, we propose a composite likelihood method and investigate its property. Furthermore, we apply our composite likelihood method to restricted Boltzmann machines.

Motivation & Objective

• To develop a systematic block selection strategy for composite likelihood estimation that ensures convergence toward the true log-likelihood as block size increases.
• To improve parameter estimation accuracy in RBMs beyond traditional pseudo-likelihood methods.
• To provide a computationally feasible alternative to maximum likelihood estimation for RBMs by leveraging higher-order composite likelihoods.
• To empirically validate the proposed method using synthetic data and compare its performance against exact maximum likelihood and lower-order CL methods.

Proposed method

• The method uses all possible k-variable subsets of visible and hidden units as blocks, forming the family 𝒞_k for k = 1, 2, ..., n.
• Composite likelihood is defined as the average of conditional log-likelihoods over all k-sized blocks, with normalization by the number of blocks.
• Gradients for parameter updates are derived using conditional expectations over each block, enabling efficient optimization via gradient ascent.
• The approach generalizes pseudo-likelihood (k=1) and approaches full likelihood (k=n) as k increases.
• The method explicitly formulates the gradient of the k-th order CL for RBMs, including energy functions and conditional probabilities.
• Numerical optimization uses gradient ascent with a fixed learning rate, applied to synthetic data with known ground truth.

Experimental results

Research questions

• RQ1 Does using all k-variable blocks in composite likelihood estimation lead to a monotonic improvement toward the true log-likelihood as k increases?
• RQ2 How does the estimation accuracy of higher-order composite likelihood compare to pseudo-likelihood and exact maximum likelihood in RBMs?
• RQ3 Can the proposed systematic block selection strategy yield better parameter estimates than existing CL or PL methods?
• RQ4 What is the trade-off between computational cost and estimation accuracy when increasing k in the composite likelihood framework?

Key findings

• After 50,000 iterations, the third-order composite likelihood estimation achieved a true log-likelihood of -1.741, matching the exact maximum likelihood result.
• The mean absolute error (MAD) for weight parameters decreased from 0.360 (k=1) to 0.103 (k=3), showing significant improvement with higher-order CL.
• Second-order CL estimation reduced MADs to 0.223 for β and 0.192 for w, indicating strong performance gains over first-order methods.
• Higher-order CLs converged faster and yielded better estimates than first-order CL, with the third-order CL closely matching the exact ML solution.
• The composite likelihood with all k-sized blocks forms an upper bound on the true log-likelihood and increases monotonically with k.
• The method enables a tunable trade-off between computational cost and estimation accuracy by selecting k.