[Paper Review] LP-SparseMAP: Differentiable Relaxed Optimization for Sparse Structured Prediction

Vlad Niculae, André F. T. Martins|arXiv (Cornell University)|Jan 13, 2020

Tensor decomposition and applications61 references4 citations

TL;DR

LP-SparseMAP introduces a differentiable, relaxed optimization framework for sparse structured prediction that extends SparseMAP to handle intractable structured models via a local polytope relaxation using factor graphs. It enables end-to-end training with efficient forward and backward passes using ADMM and closed-form gradients, achieving state-of-the-art performance on dependency parsing, natural language inference, and multilabel classification tasks with complex constraints.

ABSTRACT

Structured prediction requires manipulating a large number of combinatorial structures, e.g., dependency trees or alignments, either as latent or output variables. Recently, the SparseMAP method has been proposed as a differentiable, sparse alternative to maximum a posteriori (MAP) and marginal inference. SparseMAP returns a combination of a small number of structures, a desirable property in some downstream applications. However, SparseMAP requires a tractable MAP inference oracle. This excludes, e.g., loopy graphical models or factor graphs with logic constraints, which generally require approximate inference. In this paper, we introduce LP-SparseMAP, an extension of SparseMAP that addresses this limitation via a local polytope relaxation. LP-SparseMAP uses the flexible and powerful domain specific language of factor graphs for defining and backpropagating through arbitrary hidden structure, supporting coarse decompositions, hard logic constraints, and higher-order correlations. We derive the forward and backward algorithms needed for using LP-SparseMAP as a hidden or output layer. Experiments in three structured prediction tasks show benefits compared to SparseMAP and Structured SVM.

Motivation & Objective

• To enable differentiable structured prediction in models with intractable MAP inference, such as those with logic constraints or loopy structures.
• To extend the SparseMAP framework to support expressive, constrained structured models using a local polytope relaxation.
• To provide an efficient, modular, and end-to-end trainable structured hidden layer for neural networks.
• To derive globally convergent ADMM algorithms for the forward pass and compact backward passes without computation graph unrolling.
• To support a wide range of factor types (e.g., budget, OR, negation) via a generic interface requiring only a MAP oracle.

Proposed method

• LP-SparseMAP formulates structured prediction as a relaxed optimization problem over the local polytope, enabling differentiable inference in models where exact MAP is intractable.
• It uses a factor graph representation to model complex dependencies, constraints (e.g., budget, valency), and higher-order correlations.
• The forward pass employs an ADMM algorithm with global convergence guarantees, solving the relaxed dual problem efficiently.
• The backward pass computes gradients using a compact, closed-form expression that reuses quantities from the forward pass, avoiding backpropagation through the ADMM iterations.
• New factor types are integrated via a generic interface that requires only a MAP oracle, enabling plug-and-play extensibility.
• Specialized solvers are derived for common factors (e.g., pairwise, logical OR, negation, budget) to improve efficiency and accuracy.

Experimental results

Research questions

• RQ1 Can a differentiable structured prediction method be designed to handle intractable MAP problems with complex constraints?
• RQ2 Can the SparseMAP framework be extended to models requiring approximate inference, such as loopy graphical models or logic-constrained structures?
• RQ3 Can an efficient, globally convergent ADMM-based forward pass be derived for relaxed structured prediction?
• RQ4 Can the backward pass be computed in closed form without unrolling the computation graph?
• RQ5 Does the proposed method outperform existing approaches like SparseMAP and Structured SVM in end-to-end learning with structured latent variables?

Key findings

• LP-SparseMAP achieves state-of-the-art performance on inducing latent dependency trees in arithmetic expression parsing, outperforming SparseMAP and Structured SVM.
• In natural language inference, LP-SparseMAP with constrained parsing improves F1 scores over baselines, particularly under hard logic constraints.
• On multilabel classification tasks (bibtex and bookmarks), LP-SparseMAP achieves higher F1 and accuracy than LP-MAP and SparseMAP, especially with complex label correlations.
• The method demonstrates robustness and efficiency, converging within 10–100 ADMM iterations with a primal and dual tolerance of 1e-6.
• The backward pass achieves accurate gradients using only 100 power iterations, with no need for automatic differentiation through the ADMM loop.
• The library implementation supports C++ and Python, enabling modular integration of new factor types via a MAP oracle interface.