[Paper Review] A General Framework for Interpretable Neural Learning based on Local Information-Theoretic Goal Functions

Abdullah Makkeh, Marcel Graetz+4 more|arXiv (Cornell University)|Jun 3, 2023

Neural Networks and Applications5 citations

TL;DR

This paper introduces infomorphic networks—neural networks built from artificial neurons that learn by optimizing local information-processing goals derived from partial information decomposition (PID). By leveraging a differentiable PID measure, these networks perform supervised, unsupervised, and memory-based learning with interpretable, information-theoretic objectives, demonstrating that PID-based learning rules enable self-organizing, functionally specialized neural dynamics.

ABSTRACT

Despite the impressive performance of biological and artificial networks, an intuitive understanding of how their local learning dynamics contribute to network-level task solutions remains a challenge to this date. Efforts to bring learning to a more local scale indeed lead to valuable insights, however, a general constructive approach to describe local learning goals that is both interpretable and adaptable across diverse tasks is still missing. We have previously formulated a local information processing goal that is highly adaptable and interpretable for a model neuron with compartmental structure. Building on recent advances in Partial Information Decomposition (PID), we here derive a corresponding parametric local learning rule, which allows us to introduce 'infomorphic' neural networks. We demonstrate the versatility of these networks to perform tasks from supervised, unsupervised and memory learning. By leveraging the interpretable nature of the PID framework, infomorphic networks represent a valuable tool to advance our understanding of the intricate structure of local learning.

Motivation & Objective

• To develop a biologically plausible, interpretable model of neural computation that captures local information processing without relying on global task semantics.
• To address the limitation of classical information theory in modeling multi-source input integration in neurons by employing partial information decomposition (PID).
• To demonstrate the feasibility of training artificial neurons using differentiable PID atoms as local learning objectives.
• To show that PID-based learning rules can support diverse learning paradigms, including classification, representation learning, and associative memory.
• To provide a framework for studying self-organization and functional specialization in neural circuits through information-theoretic principles.

Proposed method

• The paper derives analytical gradients for PID atoms—unique, redundant, synergistic, and total information—enabling backpropagation through the PID framework.
• A novel local learning rule is formulated based on a weighted combination of PID atoms, with hyperparameters (Γ) specifying the relative importance of each information component.
• Neurons integrate inputs from multiple sources (receptive and contextual) via learnable weights (wR, wC), with inputs discretized into bins for entropy and mutual information estimation.
• The learning process uses a two-phase training scheme: initial weight pullback to stabilize learning, followed by standard optimization with adaptive learning rates (η) and regularization (λ).
• Networks are trained using stochastic gradient descent on a loss function composed of PID-based information goals, with input data encoded as discrete vectors in {−1, 1}.
• The framework is implemented using a differentiable PID measure I^sx, enabling end-to-end training of neural networks grounded in information-theoretic principles.

Figure 1: Output information of an infomorphic neuron consists of 5 distinct information contributions. A The total output information $H(Y)$ of a neuron that consists of the mutual information with the inputs $I(Y:\mathbf{X}_{R},\mathbf{X}_{C})$ and the stochasticity $H(Y\mid\mathbf{X}_{R},\mathbf{

Experimental results

Research questions

• RQ1 Can artificial neurons be trained using local, interpretable information-processing goals derived from partial information decomposition (PID)?
• RQ2 How do PID-based learning rules support diverse learning paradigms such as supervised learning, unsupervised representation learning, and associative memory?
• RQ3 To what extent do PID atoms (unique, redundant, synergistic information) shape the functional specialization and self-organization of neural networks?
• RQ4 Does the use of differentiable PID enable stable and effective training of neural networks in the absence of backpropagation through the entire network?
• RQ5 Can PID-based learning rules produce interpretable, functionally meaningful neural dynamics that reflect cortical computation principles?

Key findings

• Infomorphic networks successfully perform supervised classification tasks using PID-based learning rules, achieving stable convergence with appropriate hyperparameter settings.
• In unsupervised learning, networks trained with I_unq(Y:R) and I_unq(Y:C) as objectives learned to represent independent bars in input patterns, demonstrating feature discovery.
• For associative memory tasks, networks with Γ = (0.1, 0.1, 1.0, 0.1, 0) successfully recalled stored patterns over 19 time steps after a single presentation, indicating robust memory retention.
• The use of weight pullback (λ > 0) during early training improved convergence and stability, especially in complex tasks like unsupervised representation learning.
• Smaller bin sizes (e.g., nR = 20) did not degrade performance in associative memory tasks, suggesting robustness to discretization choices.
• The framework supports diverse network architectures and learning objectives, confirming the versatility of PID-based local learning rules across multiple paradigms.

Figure 2: Illustration of the basic structure of an infomorphic neuron, inspired by the morphology of cortical pyramidal neurons. A Overlay of coherent infomax neural processor developed originally by Kay and Phillips ( 2011 ) on layer 5 pyramidal cells, highlighting potential parallels to existing