[Paper Review] Optimality of Approximate Inference Algorithms on Stable Instances

Hunter Lang, David Sontag+1 more|arXiv (Cornell University)|Nov 6, 2017

Neural Networks and Applications3 citations

TL;DR

This paper establishes theoretical conditions under which two widely used approximate inference algorithms—LP relaxation and α-expansion—provably recover the optimal MAP solution in Ferromagnetic Potts Models. It introduces (p,q)-stability conditions: LP relaxation is exact under (2,1)-stability, and α-expansion is exact under (1,2)-stability, explaining their strong empirical performance on real-world instances beyond worst-case guarantees.

ABSTRACT

Approximate algorithms for structured prediction problems---such as LP relaxations and the popular alpha-expansion algorithm (Boykov et al. 2001)---typically far exceed their theoretical performance guarantees on real-world instances. These algorithms often find solutions that are very close to optimal. The goal of this paper is to partially explain the performance of alpha-expansion and an LP relaxation algorithm on MAP inference in Ferromagnetic Potts models (FPMs). Our main results give stability conditions under which these two algorithms provably recover the optimal MAP solution. These theoretical results complement numerous empirical observations of good performance.

Motivation & Objective

• To explain the gap between theoretical worst-case guarantees and strong empirical performance of approximate inference algorithms in structured prediction.
• To identify structural properties of real-world instances that make MAP inference tractable despite NP-hardness.
• To formalize stability conditions under which popular algorithms like LP relaxation and α-expansion achieve exact solutions.
• To provide theoretical justification for the robustness and near-optimality of α-expansion and LP relaxations in practice.

Proposed method

• Introduces (p,q)-stability as a formal condition for instance tractability, based on perturbations of edge and node costs.
• Analyzes LP relaxation by showing that fractional solutions are tight under (2,1)-stability, using rounding arguments and integrality gap analysis.
• Applies the α-expansion algorithm to stable instances, proving that all expansion moves preserve optimality and the algorithm converges to the global optimum.
• Uses adversarial perturbations to test stability: for (1,2)-stable instances, the LP solution remains fractional; for (2,1)-stable instances, α-expansion may fail.
• Employs a rounding framework that maps ε-close fractional solutions to integral labelings via expansion moves.
• Proves that on (1,2)-stable instances, α-expansion always reaches the optimal solution regardless of initialization, due to monotonic improvement via expansion moves.

Experimental results

Research questions

• RQ1 Under what conditions do LP relaxations for MAP inference in Ferromagnetic Potts Models yield integral, optimal solutions in practice?
• RQ2 Why does the α-expansion algorithm consistently find solutions close to optimal despite being a 2-approximation in the worst case?
• RQ3 Can stability conditions explain the observed robustness of α-expansion to initial labeling choices?
• RQ4 Are there formal stability criteria under which α-expansion and LP relaxation provably recover the optimal solution?
• RQ5 Do the algorithms fail on instances that are stable under the opposite stability condition?

Key findings

• LP relaxation is provably exact on (2,1)-stable instances, where node costs are at most twice the edge costs in perturbed configurations.
• α-expansion algorithm recovers the optimal solution on (1,2)-stable instances, regardless of initialization, due to monotonic improvement via expansion moves.
• A counterexample shows that LP relaxation can yield fractional solutions on (1,2)-stable instances, proving the condition is not sufficient for LP tightness.
• Another counterexample demonstrates that α-expansion may fail to find the optimal solution on (2,1)-stable instances, showing the stability condition is necessary and not sufficient for α-expansion.
• On (2,2)-stable instances, both LP relaxation and α-expansion are exact, indicating a regime of strong stability where both algorithms achieve optimality.
• The results explain the empirical robustness of α-expansion: on stable instances, the energy landscape ensures all expansion moves lead toward the global optimum.