[Paper Review] Relaxations for inference in restricted Boltzmann machines

Sida I. Wang, Roy Frostig+2 more|arXiv (Cornell University)|Dec 21, 2013

Machine Learning and Algorithms7 references3 citations

TL;DR

This paper proposes a randomized relax-and-round algorithm for approximate MAP inference in restricted Boltzmann machines (RBMs), using low-rank semidefinite relaxations (LRP_k) to balance efficiency and accuracy. The method outperforms annealed Gibbs sampling in escaping local optima and enables effective log-partition function estimation via importance sampling, with empirical results showing competitive performance on MNIST and synthetic RBMs.

ABSTRACT

We propose a relaxation-based approximate inference algorithm that samples near-MAP configurations of a binary pairwise Markov random field. We experiment on MAP inference tasks in several restricted Boltzmann machines. We also use our underlying sampler to estimate the log-partition function of restricted Boltzmann machines and compare against other sampling-based methods.

Motivation & Objective

• To develop a scalable and accurate inference method for restricted Boltzmann machines (RBMs) that avoids local optima common in Gibbs sampling.
• To bridge the gap between convex relaxations (like SDP) and non-convex QP by introducing low-rank relaxations (LRP_k) with tunable width k.
• To enable efficient estimation of the log-partition function in RBMs using the proposed sampler as a proposal distribution.
• To evaluate the method on real-world and synthetic RBM benchmarks, comparing against annealed Gibbs and importance sampling.

Proposed method

• The method uses low-rank semidefinite relaxation (LRP_k), where the optimization problem is formulated as maximizing tr(X^T A X) subject to ||X_i||_2 ≤ 1 for each row i.
• It employs randomized rounding: a random vector g is drawn from the unit sphere, and the final binary configuration is obtained via x_i = sign(X_i^T g).
• The algorithm initializes with LRP_2 (k=2) and uses projected gradient descent to find a locally optimal X.
• For log-partition estimation, it uses importance sampling with the rrr-MAP sampler as a proposal, computing Z(A) ≈ E[exp(x^T A x)/p_X(x)] over 10,000 samples.
• The probability p_X(x) is computed efficiently in O(n) time after O(n log n) preprocessing by sorting row vectors of X by angle.
• The method is applied to both MAP inference and log-partition estimation, with comparisons to annealed Gibbs and AIS on MNIST and synthetic RBMs.

Experimental results

Research questions

• RQ1 Can low-rank relaxations (LRP_k) provide a better trade-off between computational efficiency and solution quality than full SDP or QP relaxations for MAP inference in RBMs?
• RQ2 Does the randomized rounding of LRP_k solutions yield near-MAP configurations that outperform annealed Gibbs sampling in escaping local optima?
• RQ3 Can the rrr-MAP sampler serve as an effective proposal distribution for importance sampling to estimate the log-partition function of RBMs?
• RQ4 How does the performance of the rrr-MAP method scale with increasing RBM size and width k?

Key findings

• On the MNIST dataset, the rrr-MAP sampler achieved a log-partition estimate of 438.40, outperforming AIS (436.37) and rrr-low (436.69).
• For a small RBM (Random-S), rrr-IS estimated log Z as 5092.4, close to the true value of 5127.6, demonstrating accuracy on small instances.
• The rrr-MAP sampler produced a lower bound on log Z (rrr-low) of 436.69, which is tighter than the AIS estimate of 436.37, indicating improved sampling quality.
• The method successfully initialized local search algorithms, yielding better results than either rrr-MAP or local search alone.
• The rrr-MAP algorithm was significantly faster than Gurobi for large RBMs, with a 10x time limit still outperformed by rrr-MAP in solution quality.
• Despite its simplicity, the rrr-MAP sampler produced competitive log-partition estimates, though importance sampling performance was limited by sparse support in low-k settings.