[Paper Review] Coding-theorem Like Behaviour and Emergence of the Universal Distribution from Resource-bounded Algorithmic Probability

Héctor Zenil, Liliana Badillo+2 more|arXiv (Cornell University)|Nov 6, 2017

Computability, Logic, AI Algorithms36 references37 citations

TL;DR

This paper investigates how algorithmic probability and the universal distribution emerge from resource-bounded computational models below Turing universality. By simulating finite-state machines, context-free grammars, and linear-bounded automata, it demonstrates that even weak models produce output distributions strongly correlated with the universal distribution, accounting for up to 60% of simplicity/complexity bias—offering a practical, computable approximation to algorithmic complexity that outperforms traditional entropy and compression methods.

ABSTRACT

Previously referred to as `miraculous' in the scientific literature because of its powerful properties and its wide application as optimal solution to the problem of induction/inference, (approximations to) Algorithmic Probability (AP) and the associated Universal Distribution are (or should be) of the greatest importance in science. Here we investigate the emergence, the rates of emergence and convergence, and the Coding-theorem like behaviour of AP in Turing-subuniversal models of computation. We investigate empirical distributions of computing models in the Chomsky hierarchy. We introduce measures of algorithmic probability and algorithmic complexity based upon resource-bounded computation, in contrast to previously thoroughly investigated distributions produced from the output distribution of Turing machines. This approach allows for numerical approximations to algorithmic (Kolmogorov-Chaitin) complexity-based estimations at each of the levels of a computational hierarchy. We demonstrate that all these estimations are correlated in rank and that they converge both in rank and values as a function of computational power, despite fundamental differences between computational models. In the context of natural processes that operate below the Turing universal level because of finite resources and physical degradation, the investigation of natural biases stemming from algorithmic rules may shed light on the distribution of outcomes. We show that up to 60\% of the simplicity/complexity bias in distributions produced even by the weakest of the computational models can be accounted for by Algorithmic Probability in its approximation to the Universal Distribution.

Motivation & Objective

• To investigate how the universal distribution and algorithmic probability emerge in subuniversal computational models below Turing completeness.
• To assess the convergence and coding-theorem-like behavior of algorithmic complexity approximations across the Chomsky hierarchy.
• To evaluate whether finite, resource-limited models can approximate the universal distribution with high fidelity, despite lacking full computational power.
• To compare the performance of resource-bounded algorithmic probability against traditional estimators like Shannon entropy and lossless compression.
• To explore the implications for natural processes that operate below Turing universality due to physical or resource constraints.

Proposed method

• Simulating output distributions from finite-state transducers (FSTs), context-free grammars (CFGs), and linear-bounded automata (LBAs) at increasing computational power levels.
• Computing algorithmic probability approximations via empirical frequency distributions of generated strings across models.
• Applying the algorithmic coding theorem to relate string frequency to algorithmic complexity, using the inverse of the probability as a complexity estimator.
• Comparing these approximations to the empirical distribution of small universal Turing machines (TM(5,2)) as a benchmark for the universal distribution.
• Using rank correlation and value convergence metrics to assess similarity between distributions across models.
• Evaluating performance against standard complexity estimators: Shannon entropy and lossless compression (Compress).

Experimental results

Research questions

• RQ1 To what extent do resource-bounded models below Turing universality reproduce the statistical properties of the universal distribution?
• RQ2 How quickly and consistently do algorithmic complexity approximations converge across different computational models in the Chomsky hierarchy?
• RQ3 Can finite approximations of algorithmic probability explain a significant portion of the simplicity/complexity bias observed in weak computational models?
• RQ4 How do non-halting models compare to halting models in terms of string coverage and distributional similarity to the universal distribution?
• RQ5 What is the relative performance of resource-bounded algorithmic probability compared to Shannon entropy and lossless compression in estimating algorithmic complexity?

Key findings

• The output distributions of FSTs, CFGs, and LBAs show strong rank correlation with the universal distribution derived from small Turing machines, indicating emergent coding-theorem-like behavior.
• Even the weakest model, the finite-state transducer (FST), accounts for up to 60% of the simplicity/complexity bias observed in its output distribution through algorithmic probability approximation.
• Linear-bounded automata (LBA) produce the most accurate approximation to the universal distribution among subuniversal models, both in rank and value convergence.
• Despite their computational cost, CFGs produce distributions highly correlated with the universal distribution, though they are less efficient than LBAs for large-scale sampling.
• Resource-bounded algorithmic probability approximations outperform both Shannon entropy and lossless compression in estimating algorithmic complexity, especially for strings with high algorithmic randomness.
• Non-halting models such as LBAs show convergence to a distribution closer to LBA than to full Turing machines, suggesting a natural asymptotic limit below universality.