[Paper Review] Optimization With Parity Constraints: From Binary Codes to Discrete Integration

Stefano Ermon, Carla P. Gomes+2 more|arXiv (Cornell University)|Sep 26, 2013

Error Correcting Code Techniques30 references18 citations

TL;DR

This paper proposes a novel approach to approximate probabilistic inference by reducing partition function computation to a series of MAP inference problems augmented with random parity constraints. By leveraging connections to binary code decoding and using an ILP formulation with sparsification and LP relaxation, the method achieves high-probability bounds on the partition function that are significantly tighter than variational methods.

ABSTRACT

Many probabilistic inference tasks involve summations over exponentially large sets. Recently, it has been shown that these problems can be reduced to solving a polynomial number of MAP inference queries for a model augmented with randomly generated parity constraints. By exploiting a connection with max-likelihood decoding of binary codes, we show that these optimizations are computationally hard. Inspired by iterative message passing decoding algorithms, we propose an Integer Linear Programming (ILP) formulation for the problem, enhanced with new sparsification techniques to improve decoding performance. By solving the ILP through a sequence of LP relaxations, we get both lower and upper bounds on the partition function, which hold with high probability and are much tighter than those obtained with variational methods.

Motivation & Objective

• To address the computational challenge of summing over exponentially large sets in probabilistic inference.
• To reduce the problem of computing partition functions to a polynomial number of MAP inference queries with parity constraints.
• To improve the accuracy of bounds on the partition function using integer programming and sparsification techniques.
• To establish theoretical connections between discrete integration and max-likelihood decoding of binary codes.

Proposed method

• The method reduces the computation of a partition function to solving a sequence of MAP inference problems on an augmented model with randomly generated parity constraints.
• It formulates the problem as an Integer Linear Program (ILP), exploiting structural properties from binary code decoding to enhance solvability.
• Sparsification techniques are introduced to improve decoding performance and reduce computational overhead.
• The ILP is solved via a sequence of linear programming relaxations to generate both lower and upper bounds on the partition function.
• High-probability bounds are derived using probabilistic guarantees from coding theory, ensuring tightness with high confidence.

Experimental results

Research questions

• RQ1 Can partition function computation in probabilistic models be reduced to a polynomial number of MAP inference queries?
• RQ2 How can parity constraints be leveraged to improve the accuracy of bounds on the partition function?
• RQ3 What is the relationship between discrete integration and max-likelihood decoding of binary codes?
• RQ4 Can ILP formulations with sparsification yield tighter bounds than traditional variational methods?

Key findings

• The proposed method achieves bounds on the partition function that are significantly tighter than those obtained with variational inference techniques.
• The bounds hold with high probability due to theoretical guarantees derived from coding theory.
• Sparsification techniques improve the decoding performance and scalability of the ILP formulation.
• The use of LP relaxations over the ILP enables efficient computation of both lower and upper bounds.