[Paper Review] Avoiding pathologies in very deep networks

David Duvenaud, Oren Rippel+2 more|arXiv (Cornell University)|Feb 24, 2014

Gaussian Processes and Bayesian Inference28 references84 citations

TL;DR

This paper identifies a pathological collapse in very deep neural networks where representational capacity diminishes to a single degree of freedom as depth increases, proposing an input-connected architecture that preserves information across layers. Using deep Gaussian processes and kernel composition, it shows that standard architectures suffer from representational degeneracy, while the proposed architecture maintains rich, diverse feature representations in the infinite-depth limit.

ABSTRACT

Choosing appropriate architectures and regularization strategies for deep networks is crucial to good predictive performance. To shed light on this problem, we analyze the analogous problem of constructing useful priors on compositions of functions. Specifically, we study the deep Gaussian process, a type of infinitely-wide, deep neural network. We show that in standard architectures, the representational capacity of the network tends to capture fewer degrees of freedom as the number of layers increases, retaining only a single degree of freedom in the limit. We propose an alternate network architecture which does not suffer from this pathology. We also examine deep covariance functions, obtained by composing infinitely many feature transforms. Lastly, we characterize the class of models obtained by performing dropout on Gaussian processes.

Motivation & Objective

• To analyze the representational capacity of very deep neural networks by modeling them as priors on functions.
• To identify the pathological behavior in standard deep architectures where information is lost across layers.
• To propose a modified network architecture that avoids representational degeneracy by connecting the input to each layer.
• To study deep kernels and dropout-regularized Gaussian processes as alternative inductive biases for deep learning.
• To provide theoretical insights into weight initialization, regularization, and architecture design through prior analysis.

Proposed method

• Model deep neural networks as deep Gaussian processes (DGPs), where each layer's transformation is drawn from a GP prior.
• Analyze the limiting behavior of function compositions in DGPs to show that standard architectures lose representational degrees of freedom with depth.
• Propose an input-connected architecture where each layer receives both the original input and the previous layer's output, preserving input information.
• Derive closed-form expressions for deep kernels by composing fixed feature maps, such as those from arc-cosine or squared exponential kernels.
• Characterize dropout regularization on Gaussian processes, showing equivalence to existing models and enabling tractable inference.
• Use Mercer's theorem to connect kernel functions to implicit feature representations in neural networks.

Experimental results

Research questions

• RQ1 Why do very deep neural networks lose representational capacity as depth increases?
• RQ2 What architectural choices prevent the collapse of representational degrees of freedom in deep networks?
• RQ3 How do deep Gaussian processes model the inductive bias of deep neural networks in the infinite-width, infinite-depth limit?
• RQ4 What is the functional form and representational capacity of deep kernels formed by composing fixed feature maps?
• RQ5 How does dropout on Gaussian processes relate to standard dropout in neural networks, and what is its prior structure?

Key findings

• Standard deep architectures composed of independent GP priors collapse to a single degree of freedom in the infinite-depth limit, rendering them invariant to most input variations.
• The proposed input-connected architecture prevents this collapse by preserving the original input signal at every layer, maintaining rich representational capacity.
• Deep kernels formed by composing infinitely many fixed feature maps retain complex, hierarchical representations and can be analytically derived for common kernels like squared exponential and arc-cosine.
• Dropout applied to Gaussian processes yields a tractable prior that corresponds to a specific form of weight sharing and regularization, analogous to standard dropout in neural networks.
• The analysis reveals that many deep learning phenomena—such as performance degradation with depth—can be traced to the inductive bias encoded in the network's prior structure.
• The findings suggest that architectural choices, weight initialization, and regularization strategies should be guided by explicit analysis of the prior over functions.