[Paper Review] On the Expressive Power of Deep Learning: A Tensor Analysis

Nadav Cohen, Or Sharir+1 more|arXiv (Cornell University)|Sep 16, 2015

Tensor decomposition and applications40 references135 citations

TL;DR

This paper proposes a convolutional arithmetic circuit architecture that integrates locality, weight sharing, and pooling—key components of practical convolutional neural networks—by modeling them as hierarchical tensor decompositions. It proves that, except for a negligible set, functions expressible by deep networks of polynomial size require exponentially larger shallow networks to approximate, establishing a fundamental theoretical advantage of depth in expressive power.

ABSTRACT

It has long been conjectured that hypotheses spaces suitable for data that is compositional in nature, such as text or images, may be more efficiently represented with deep hierarchical networks than with shallow ones. Despite the vast empirical evidence supporting this belief, theoretical justifications to date are limited. In particular, they do not account for the locality, sharing and pooling constructs of convolutional networks, the most successful deep learning architecture to date. In this work we derive a deep network architecture based on arithmetic circuits that inherently employs locality, sharing and pooling. An equivalence between the networks and hierarchical tensor factorizations is established. We show that a shallow network corresponds to CP (rank-1) decomposition, whereas a deep network corresponds to Hierarchical Tucker decomposition. Using tools from measure theory and matrix algebra, we prove that besides a negligible set, all functions that can be implemented by a deep network of polynomial size, require exponential size in order to be realized (or even approximated) by a shallow network. Since log-space computation transforms our networks into SimNets, the result applies directly to a deep learning architecture demonstrating promising empirical performance. The construction and theory developed in this paper shed new light on various practices and ideas employed by the deep learning community.

Motivation & Objective

• To theoretically justify the superiority of deep networks over shallow ones in representing compositional data such as images and text.
• To develop a theoretical framework that accounts for key architectural elements of convolutional networks—locality, weight sharing, and pooling—previously absent in depth efficiency proofs.
• To establish a formal connection between deep neural networks and hierarchical tensor decompositions, specifically Hierarchical Tucker and CP decompositions.
• To prove that, except for a set of measure zero, functions realizable by deep networks of polynomial size require exponential size in shallow networks to approximate.
• To demonstrate that the theoretical findings apply to practical deep learning architectures, such as SimNets, via log-space computation and numerical stability.

Proposed method

• The authors design a convolutional arithmetic circuit architecture using sum nodes for convolutions (with locality and weight sharing) and product nodes for pooling operations.
• They establish an equivalence between deep networks and Hierarchical Tucker tensor decompositions, and between shallow networks and CP (rank-1) decompositions.
• Using tools from measure theory and matrix algebra, they analyze the expressive capacity of these decompositions to compare deep versus shallow network representational power.
• The network is transformed into log-space to ensure numerical stability, enabling implementation via SimNets with the MEX operator for stable log-sum-exp and sum-pooling operations.
• Theoretical analysis leverages the fact that non-negative tensor decompositions preserve universality and allow for efficient, numerically stable training.
• The construction is validated by showing equivalence to SimNets, a recently proposed architecture with strong empirical performance on image benchmarks.

Experimental results

Research questions

• RQ1 Can a theoretical framework be developed that explains the expressive power of deep learning while incorporating key architectural elements like locality, weight sharing, and pooling?
• RQ2 Is there a formal mathematical equivalence between deep neural networks and hierarchical tensor decompositions?
• RQ3 To what extent do deep networks outperform shallow networks in representing compositional functions?
• RQ4 How does the expressive power of deep networks compare to shallow networks in terms of size requirements for approximation?
• RQ5 Can theoretical depth efficiency results be transferred to practical deep learning architectures such as SimNets?

Key findings

• Except for a set of measure zero, all functions realizable by a deep network of polynomial size require exponentially larger shallow networks to approximate, establishing a strong depth efficiency result.
• Shallow networks correspond to CP (rank-1) tensor decompositions, while deep networks correspond to Hierarchical Tucker decompositions, providing a formal tensor-based characterization of depth.
• The proposed architecture, based on arithmetic circuits with sum and product nodes, naturally incorporates locality, weight sharing, and pooling, aligning with practical convolutional networks.
• Theoretical results are transferable to practical models: the construction maps directly to SimNets, which have demonstrated state-of-the-art performance on image recognition benchmarks under resource constraints.
• Numerical stability is achieved via log-space computation, with the MEX operator enabling stable implementation of log-sum-exp and sum-pooling operations.
• Non-negative tensor decompositions, though potentially less efficient than unconstrained ones, preserve universality and enable practical deployment with minimal performance degradation.