[Paper Review] Formally Justifying MDL-based Inference of Cause and Effect

Alexander Marx, Jilles Vreeken|arXiv (Cornell University)|May 5, 2021

Computability, Logic, AI Algorithms30 references4 citations

TL;DR

This paper formally bridges the gap between the algorithmic independence of conditionals (AIC) postulate and practical two-part Minimum Description Length (MDL) methods for cause-effect inference. It derives a Kolmogorov complexity-based two-part formulation of AIC that directly maps to MDL encodings and proves that this formulation yields equivalent causal inference results as the original AIC postulate, ensuring theoretical consistency in MDL-based causal discovery.

ABSTRACT

The algorithmic independence of conditionals, which postulates that the causal mechanism is algorithmically independent of the cause, has recently inspired many highly successful approaches to distinguish cause from effect given only observational data. Most popular among these is the idea to approximate algorithmic independence via two-part Minimum Description Length (MDL). Although intuitively sensible, the link between the original postulate and practical two-part MDL encodings is left vague. In this work, we close this gap by deriving a two-part formulation of this postulate, in terms of Kolmogorov complexity, which directly links to practical MDL encodings. To close the cycle, we prove that this formulation leads on expectation to the same inference result as the original postulate.

Motivation & Objective

• To formally link the algorithmic independence of conditionals (AIC) postulate with practical two-part MDL encodings used in cause-effect inference.
• To resolve the ambiguity in how MDL approximations relate to the original AIC postulate formulated in terms of Kolmogorov complexity.
• To establish that MDL-based inference using two-part descriptions is theoretically justified by the AIC principle.
• To analyze the implications of joint data-model encodings for causal inference, emphasizing model independence from data.

Proposed method

• Derives a two-part Kolmogorov complexity formulation of the AIC postulate, explicitly including both the model complexity and the data complexity given the model.
• Uses the chain rule of Kolmogorov complexity to express the total description length as $ K(P_X) + K(P_{Y|X}) + H(P_{XY}) $ for the causal direction, up to an additive constant.
• Proves that this formulation leads to the same causal inference outcome as the original AIC postulate in expectation, by leveraging the algorithmic independence of mechanisms.
• Analyzes joint encodings where model and data are described together, showing that the model must be independent of the data to preserve asymmetry.
• Demonstrates that $ K(P_{Y|X} \mid x) \approx K(P_{Y|X}) $ under AIC, but $ K(P_{X|Y} \mid y) \not\approx K(P_{X|Y}) $ in the anti-causal direction, preserving asymmetry.
• Establishes a one-to-one mapping between the derived two-part Kolmogorov complexity formulation and standard two-part MDL approximations used in practice.

Experimental results

Research questions

• RQ1 How can the algorithmic independence of conditionals postulate be formally expressed in terms of two-part Kolmogorov complexity?
• RQ2 Does the two-part MDL approximation used in practice correspond to the original AIC postulate in terms of causal inference performance?
• RQ3 What conditions must joint data-model encodings satisfy to preserve the asymmetry between causal and anti-causal directions?
• RQ4 Why does including data in the model description break the asymmetry required for causal inference?
• RQ5 Can the theoretical foundation of MDL-based cause-effect inference be rigorously justified via Kolmogorov complexity?

Key findings

• The paper derives a two-part Kolmogorov complexity formulation of the AIC postulate that explicitly includes both model complexity and data complexity given the model.
• This formulation is proven to lead to the same causal inference result as the original AIC postulate in expectation, thereby justifying the use of MDL approximations.
• The asymmetry between causal and anti-causal directions is preserved only when the model complexity is independent of the data, i.e., $ K(P_{Y|X} \mid x) \approx K(P_{Y|X}) $.
• For joint encodings, the model must be encoded independently of the data to maintain the necessary asymmetry; otherwise, the description length becomes symmetric and inference fails.
• The derived formulation provides a one-to-one mapping to standard two-part MDL encodings, validating their use in practical cause-effect inference.
• The results show that MDL-based methods are theoretically sound for causal discovery only when the model description is independent of the observed data.