[Paper Review] Generalized Collective Inference with Symmetric Clique Potentials

Rahul Gupta, Sunita Sarawagi+1 more|ArXiv.org|Jul 3, 2009

Advanced Graph Neural Networks23 references3 citations

TL;DR

This paper proposes a generalized collective inference framework that uses symmetric clique potentials to enforce agreement on complex, decomposable properties across multiple instances, enabling more accurate joint labeling. It introduces property-aware message passing and exact/approximate algorithms for various symmetric potentials—achieving faster, more accurate inference than state-of-the-art methods like TRW-S and graph cuts, especially for Potts and majority potentials.

ABSTRACT

Collective graphical models exploit inter-instance associative dependence to output more accurate labelings. However existing models support very limited kind of associativity which restricts accuracy gains. This paper makes two major contributions. First, we propose a general collective inference framework that biases data instances to agree on a set of {\em properties} of their labelings. Agreement is encouraged through symmetric clique potentials. We show that rich properties leads to bigger gains, and present a systematic inference procedure for a large class of such properties. The procedure performs message passing on the cluster graph, where property-aware messages are computed with cluster specific algorithms. This provides an inference-only solution for domain adaptation. Our experiments on bibliographic information extraction illustrate significant test error reduction over unseen domains. Our second major contribution consists of algorithms for computing outgoing messages from clique clusters with symmetric clique potentials. Our algorithms are exact for arbitrary symmetric potentials on binary labels and for max-like and majority-like potentials on multiple labels. For majority potentials, we also provide an efficient Lagrangian Relaxation based algorithm that compares favorably with the exact algorithm. We present a 13/15-approximation algorithm for the NP-hard Potts potential, with runtime sub-quadratic in the clique size. In contrast, the best known previous guarantee for graphs with Potts potentials is only 1/2. We empirically show that our method for Potts potentials is an order of magnitude faster than the best alternatives, and our Lagrangian Relaxation based algorithm for majority potentials beats the best applicable heuristic -- ICM.

Motivation & Objective

• Address the limitation of existing collective models that only support simple associative dependencies, such as identical labeling for repeated words.
• Enable richer forms of inter-instance agreement by introducing a framework that encourages agreement on complex, decomposable properties of labelings.
• Develop efficient inference algorithms tailored to symmetric clique potentials, including exact and approximation methods for different potential types.
• Demonstrate the framework's effectiveness in domain adaptation, particularly for bibliographic information extraction on unseen domains.
• Provide a scalable, message-passing-based inference procedure that leverages property-aware cluster updates for tractable computation.

Proposed method

• Propose a generalized collective inference framework using symmetric clique potentials to enforce agreement on decomposable properties across multiple MRFs.
• Define a cluster graph with separator variables based on property values, enabling joint MAP inference via message passing.
• Design property-aware message passing: modify messages within MRF clusters to reflect property-specific constraints, using cluster-specific algorithms.
• Develop exact combinatorial algorithms for cardinality-based, max-like, and majority-like potentials on binary and multi-label settings.
• Introduce a Lagrangian relaxation-based algorithm for majority potentials that is both efficient and more accurate than existing heuristics like ICM.
• Propose the α-pass algorithm for Potts potentials with a tight approximation guarantee of 13/15 and sub-quadratic runtime in clique size.

Experimental results

Research questions

• RQ1 Can a generalized collective inference framework be designed to support complex, non-trivial associative properties beyond simple label repetition?
• RQ2 How can efficient and exact inference be achieved for symmetric clique potentials, especially for non-trivial properties like majority or Potts potentials?
• RQ3 Can property-aware message passing outperform standard belief propagation in terms of accuracy and speed for structured prediction tasks?
• RQ4 To what extent does the proposed framework improve generalization in domain adaptation scenarios?
• RQ5 Can approximation algorithms for complex potentials (e.g., Potts) achieve better trade-offs between accuracy, runtime, and approximation guarantees than existing methods?

Key findings

• The proposed framework achieves significant test error reduction in domain adaptation tasks, with up to 82.2% token-F1 on the Address dataset and 89.6% on Cora using majority potentials.
• For Potts potentials, the α-pass algorithm achieves a 13/15 approximation guarantee and runs in sub-quadratic time, outperforming TRW-S and graph-cut methods in both speed and accuracy.
• The Lagrangian relaxation-based algorithm for majority potentials is two orders of magnitude faster than exact inference and outperforms the ICM heuristic in multiple scenarios.
• CI (collective inference) reaches optimal accuracy in just one iteration, while TRW-S requires around 20 iterations—making CI roughly an order of magnitude faster per accuracy level.
• The framework demonstrates that property conformance through symmetric potentials provides an effective inference-only solution for domain adaptation without requiring model retraining.
• Empirical comparisons show that collective inference with majority potentials consistently outperforms Potts-based methods and stacking baselines, especially on unseen domains.