[Paper Review] On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligence

Yi Ma, Doris Y. Tsao+1 more|arXiv (Cornell University)|Jul 11, 2022

Cell Image Analysis Techniques7 citations

TL;DR

This paper proposes Parsimony and Self-consistency as foundational principles for the emergence of intelligence in artificial and natural systems. It introduces compressive closed-loop transcription—a unified, computable framework that integrates discriminative and generative modeling via feedback, enabling efficient, stable, and autonomous learning with compact, interpretable representations.

ABSTRACT

Ten years into the revival of deep networks and artificial intelligence, we propose a theoretical framework that sheds light on understanding deep networks within a bigger picture of Intelligence in general. We introduce two fundamental principles, Parsimony and Self-consistency, that address two fundamental questions regarding Intelligence: what to learn and how to learn, respectively. We believe the two principles are the cornerstones for the emergence of Intelligence, artificial or natural. While these two principles have rich classical roots, we argue that they can be stated anew in entirely measurable and computable ways. More specifically, the two principles lead to an effective and efficient computational framework, compressive closed-loop transcription, that unifies and explains the evolution of modern deep networks and many artificial intelligence practices. While we mainly use modeling of visual data as an example, we believe the two principles will unify understanding of broad families of autonomous intelligent systems and provide a framework for understanding the brain.

Motivation & Objective

• To address the lack of principled, unified design in modern deep learning by identifying fundamental principles governing intelligence.
• To resolve inefficiencies in current AI practices—such as model bloat, poor robustness, and lack of interpretability—through a theoretical framework grounded in information and control theory.
• To unify discriminative and generative modeling into a single closed-loop system that enables autonomous, self-correcting learning without external supervision.
• To provide a computable, measurable, and implementable framework for building intelligent systems that emulate the efficiency and adaptability of biological brains.

Proposed method

• Formalize Parsimony as the principle of minimizing representation size while preserving essential information, grounded in information theory and compression.
• Formalize Self-consistency as a feedback mechanism where a model can regenerate its input from its internal representation, ensuring consistency and interpretability.
• Develop compressive closed-loop transcription: a two-stage system with an encoder $f(\cdot,\theta)$ mapping data $\bm{x}$ to compact representation $\bm{z}$, and a decoder $g(\cdot,\eta)$ reconstructing $\hat{\bm{x}}$ from $\bm{z}$, with $f$ and $g$ trained to be mutually consistent.
• Use feedback loops to enforce self-consistency: the encoder must not distinguish between original and reconstructed data, enabling error correction and stable training.
• Apply this framework to visual data modeling, demonstrating how structured, sparse, and modular representations emerge naturally from the two principles.
• Integrate insights from neuroscience (e.g., sparse and subspace coding) and control theory (e.g., closed-loop feedback) to unify artificial and biological intelligence.

Experimental results

Research questions

• RQ1 How can we define a principled objective for what to learn in intelligent systems, and how can it be measured computationally?
• RQ2 What computational mechanism enables efficient, stable, and adaptive learning in the absence of external supervision?
• RQ3 Can a unified framework integrate discriminative and generative modeling into a single, self-correcting system?
• RQ4 How do Parsimony and Self-consistency lead to structured, interpretable, and robust representations in deep networks?
• RQ5 To what extent can this framework explain the efficiency and adaptability of biological brains in perception and decision-making?

Key findings

• The framework of compressive closed-loop transcription unifies modern deep learning practices by integrating discriminative and generative modeling into a single, self-consistent system.
• Self-consistency ensures that the model’s internal representation is robust and interpretable, as the encoder cannot distinguish between real and reconstructed inputs.
• Parsimony leads to compact, structured representations that avoid neural collapse, preserving rich feature information even in the final layers.
• The closed-loop architecture improves training stability, mitigates mode collapse, and enhances robustness to deformations and adversarial attacks.
• The framework naturally explains functional modularity and sparse coding observed in the brain, suggesting a common principle across artificial and biological intelligence.
• Empirical results from recent research (e.g., Dai et al., 2022) demonstrate that this approach enables efficient, low-resource learning with performance comparable to large-scale end-to-end models, while remaining interpretable and adaptive.