[Paper Review] Information, learning and falsification

David Balduzzi|arXiv (Cornell University)|Oct 17, 2011

Computability, Logic, AI Algorithms6 references3 citations

TL;DR

This paper introduces effective information—a computable, non-universal analog of Kolmogorov complexity that quantifies information in terms of the sharpness of a system's output categories. By linking algorithmic information, Shannon information, and statistical learning capacities like VC-entropy and Rademacher complexity, it shows that effective information measures information gain and hypothesis falsification, offering a unified framework grounded in physical systems and causal inference.

ABSTRACT

There are (at least) three approaches to quantifying information. The first, algorithmic information or Kolmogorov complexity, takes events as strings and, given a universal Turing machine, quantifies the information content of a string as the length of the shortest program producing it. The second, Shannon information, takes events as belonging to ensembles and quantifies the information resulting from observing the given event in terms of the number of alternate events that have been ruled out. The third, statistical learning theory, has introduced measures of capacity that control (in part) the expected risk of classifiers. These capacities quantify the expectations regarding future data that learning algorithms embed into classifiers. This note describes a new method of quantifying information, effective information, that links algorithmic information to Shannon information, and also links both to capacities arising in statistical learning theory. After introducing the measure, we show that it provides a non-universal analog of Kolmogorov complexity. We then apply it to derive basic capacities in statistical learning theory: empirical VC-entropy and empirical Rademacher complexity. A nice byproduct of our approach is an interpretation of the explanatory power of a learning algorithm in terms of the number of hypotheses it falsifies, counted in two different ways for the two capacities. We also discuss how effective information relates to information gain, Shannon and mutual information.

Motivation & Objective

• To bridge the conceptual and quantitative gap between algorithmic information (Kolmogorov complexity), Shannon information, and statistical learning theory capacities.
• To develop a computable measure of information that avoids the non-computability of Kolmogorov complexity while preserving its core intuition of information as minimal description length.
• To interpret learning processes, such as empirical risk minimization, as physical systems generating information through hypothesis falsification.
• To unify key concepts in information theory and learning theory by showing that empirical VC-entropy and Rademacher complexity emerge naturally from effective information.
• To provide a causal, physical interpretation of information gain and falsification in terms of the number of hypotheses ruled out by a learning algorithm’s output.

Proposed method

• Define effective information as the Kullback-Leibler divergence between the actual repertoire (posterior over inputs given an output) and the uniform prior over inputs: $ ei(\mathfrak{m},y) = D[\hat{p}_{\mathfrak{m}}(X|y) \| p_{\text{unif}}(X)] $.
• For deterministic systems $ f: \mathcal{X} \to \mathcal{Y} $, effective information reduces to $ ei(f,y) = \log_2|\mathcal{X}| - \log_2|f^{-1}(y)| $, quantifying the inverse size of the pre-image.
• Establish a non-universal analog of Kolmogorov complexity by replacing a universal Turing machine with a physical system $ f $, where effective information becomes $ ei(f,y) = -\log_2 p_f(y) $, with $ p_f(y) $ the effective probability of output $ y $.
• Apply the framework to empirical risk minimization in learning theory, modeling the learning algorithm $ \mathfrak{L}_{\mathcal{F},\mathcal{D}} $ as a physical system mapping labelings to empirical risk.
• Show that empirical VC-entropy equals $ l - \text{effective information} $, where $ l $ is the number of data points, linking it to the number of hypotheses falsified.
• Derive that mutual information is the expected effective information over the output distribution, thus unifying Shannon and effective information.

Experimental results

Research questions

• RQ1 How can a computable measure of information be constructed that preserves the core intuition of algorithmic information while avoiding its non-computability?
• RQ2 In what way does effective information unify algorithmic information, Shannon information, and statistical learning capacities such as VC-entropy and Rademacher complexity?
• RQ3 How does the effective information generated during empirical risk minimization relate to the number of hypotheses falsified by the learning process?
• RQ4 Can effective information serve as a physical, causal measure of information gain that explains the generalization performance of classifiers?
• RQ5 What is the relationship between effective information and falsification in the sense of Popperian scientific theory evaluation?

Key findings

• Effective information provides a non-universal analog of Kolmogorov complexity, computable as $ ei(f,y) = -\log_2 p_f(y) $, where $ p_f(y) $ is the effective probability of output $ y $ under a deterministic system $ f $.
• For a deterministic function $ f $, effective information is $ ei(f,y) = \log_2|\mathcal{X}| - \log_2|f^{-1}(y)| $, showing that sharper (smaller) pre-images yield higher information.
• Empirical VC-entropy is equal to $ l - \text{effective information} $, where $ l $ is the number of data points, meaning the effective information quantifies the number of hypotheses falsified.
• Empirical Rademacher complexity is also expressible in terms of effective information, providing a new interpretation of this capacity as information gain from hypothesis falsification.
• Shannon entropy and mutual information emerge as the expected values of effective information over uniform and channel-induced distributions, respectively.
• The framework interprets information gain as the number of hypotheses ruled out by a learning algorithm, aligning with Popper’s falsificationist criterion of scientific theory power.