[Paper Review] Modeling Information Flow Through Deep Neural Networks

Ahmad Chaddad, Behnaz Naisiri+4 more|arXiv (Cornell University)|Nov 29, 2017

Domain Adaptation and Few-Shot Learning30 references3 citations

TL;DR

This paper introduces Conditional Entropy of Neural Activations (CENT), a novel information-theoretic framework that models information flow in deep neural networks by computing the conditional entropy H(Y|C,F) of filter responses Y given class C and filter bank F. CENT features—derived from trained CNNs—achieve state-of-the-art performance with minimal parameters, outperforming the original CNN's softmax output in both Alzheimer’s disease classification (AUC=94.6%) and transfer learning (AUC comparable to 1000-feature softmax).

ABSTRACT

This paper proposes a principled information theoretic analysis of classification for deep neural network structures, e.g. convolutional neural networks (CNN). The output of convolutional filters is modeled as a random variable Y conditioned on the object class C and network filter bank F. The conditional entropy (CENT) H(Y |C,F) is shown in theory and experiments to be a highly compact and class-informative code, that can be computed from the filter outputs throughout an existing CNN and used to obtain higher classification results than the original CNN itself. Experiments demonstrate the effectiveness of CENT feature analysis in two separate CNN classification contexts. 1) In the classification of neurodegeneration due to Alzheimer's disease (AD) and natural aging from 3D magnetic resonance image (MRI) volumes, 3 CENT features result in an AUC=94.6% for whole-brain AD classification, the highest reported accuracy on the public OASIS dataset used and 12% higher than the softmax output of the original CNN trained for the task. 2) In the context of visual object classification from 2D photographs, transfer learning based on a small set of CENT features identified throughout an existing CNN leads to AUC values comparable to the 1000-feature softmax output of the original network when classifying previously unseen object categories. The general information theoretical analysis explains various recent CNN design successes, e.g. densely connected CNN architectures, and provides insights for future research directions in deep learning.

Motivation & Objective

• To develop a principled information-theoretic framework for analyzing information flow in deep neural networks.
• To address the lack of a unified theoretical understanding behind heuristic CNN design improvements such as residual or densely connected networks.
• To identify compact, class-informative features from trained CNNs that outperform standard output representations like softmax.
• To enable effective transfer learning using a small number of features extracted from existing CNNs without retraining.

Proposed method

• Models the output of convolutional filters as a random variable Y conditioned on object class C and filter bank F.
• Computes the conditional entropy H(Y|C,F) as a measure of information content, termed CENT features.
• Uses CENT features as a compact, class-discriminative code for image classification, replacing or augmenting traditional CNN outputs.
• Applies random forest classifiers on CENT features to evaluate performance in both disease classification and transfer learning.
• Employs layer-wise computation of CENT across multiple convolutional layers to aggregate information from different network depths.
• Compares CENT performance against the original CNN’s softmax output in both ablation and transfer learning settings.

Experimental results

Research questions

• RQ1 Can conditional entropy H(Y|C,F) of filter responses serve as a highly informative and compact code for image classification in deep networks?
• RQ2 Does CENT feature extraction improve classification performance compared to standard CNN outputs like softmax?
• RQ3 Can a small set of CENT features enable effective transfer learning for previously unseen object categories?
• RQ4 How does the information-theoretic framework explain the success of modern CNN architectures like densely connected networks?
• RQ5 What is the impact of ReLU normalization on CENT feature quality and classification performance?

Key findings

• CENT features derived from just three convolutional layers achieved an AUC of 94.6% in classifying Alzheimer’s disease from 3D MRI scans on the OASIS dataset, surpassing the original CNN’s softmax output by 12%.
• In transfer learning, a small set of CENT features extracted from a pre-trained CNN achieved AUC values comparable to the full 1000-feature softmax layer when classifying 10 novel object categories.
• The performance of CENT features was significantly improved when computed after ReLU normalization, suggesting that normalized, non-bimodal responses enhance class-discriminative power.
• CENT features selected by random forest models corresponded to filters in deeper network layers, indicating that high-level representations are most informative for classification.
• The framework explains the success of modern architectures like densely connected networks by showing that information integration across layers enhances classification performance.
• The results suggest that a minimum of log₂N CENT features may be necessary to uniquely encode N object classes, offering a theoretical basis for CNN scaling.