[Paper Review] The Stochastic complexity of spin models: How simple are simple spin models?

Alberto Beretta, Claudia Battistin+3 more|arXiv (Cornell University)|Feb 24, 2017

Bayesian Modeling and Causal Inference3 citations

TL;DR

This paper investigates the stochastic complexity of spin models with arbitrary-order interactions using information-theoretic principles. It shows that simplicity arises not from low-order interactions, but from localized statistical dependencies across non-overlapping variable groups—making such models simple, while fully connected pairwise models are highly complex due to extensive interaction structures.

ABSTRACT

Simple models, in information theoretic terms, are those with a small stochastic complexity. We study the stochastic complexity of spin models with interactions of arbitrary order. Invariance with respect to bijections within the space of operators allows us to classify models in complexity classes. This invariance also shows that simplicity is not related to the order of the interactions, but rather to their mutual arrangement. Models where statistical dependencies are localized on non-overlapping groups of few variables (and that afford predictions on independencies that are easy to falsify) are simple. On the contrary, fully connected pairwise models, which are often used in statistical learning, are highly complex because of their extended set of interactions.

Motivation & Objective

• To understand the stochastic complexity of spin models with arbitrary interaction orders.
• To classify spin models into complexity classes based on invariance under bijections in operator space.
• To determine whether low-order interactions inherently imply simplicity in spin systems.
• To contrast the complexity of localized dependency models with fully connected pairwise models.

Proposed method

• Applies information-theoretic stochastic complexity to quantify model complexity in spin systems.
• Uses invariance under bijections in the space of operators to define equivalence classes of models.
• Classifies models based on the structural arrangement of interactions rather than their order.
• Analyzes how interaction patterns affect falsifiability of statistical independencies.
• Compares localized, non-overlapping group interactions against fully connected pairwise models.
• Employs formal invariance principles to show that complexity depends on interaction arrangement, not interaction degree.

Experimental results

Research questions

• RQ1 What determines the stochastic complexity of spin models with arbitrary-order interactions?
• RQ2 How does invariance under bijections in operator space affect model classification?
• RQ3 Why are fully connected pairwise models considered highly complex despite their simplicity in form?
• RQ4 Can models with localized dependencies be considered simpler even with higher-order interactions?
• RQ5 What role does the arrangement of statistical dependencies play in model simplicity?

Key findings

• Models with statistical dependencies localized on non-overlapping groups of few variables exhibit low stochastic complexity.
• Simplicity in spin models is not determined by the order of interactions, but by their structural arrangement.
• Fully connected pairwise models are highly complex due to their extensive and interwoven interaction sets.
• Models with localized dependencies produce predictions on independencies that are easy to falsify, enhancing their simplicity.
• Invariance under bijections in operator space reveals that complexity is invariant to relabeling, emphasizing structural over order-based distinctions.
• The study establishes that interaction arrangement, not interaction order, is the key determinant of stochastic complexity.