[Paper Review] On the Expressive Power of Deep Polynomial Neural Networks

Joe Kileel, Matthew Trager+1 more|arXiv (Cornell University)|May 29, 2019

Tensor decomposition and applications20 references34 citations

TL;DR

The paper treats deep polynomial networks as algebraic objects, defines a functional variety via the network map, and derives dimension formulas, filling/bottleneck conditions, and connections to tensor decompositions to quantify expressivity.

ABSTRACT

We study deep neural networks with polynomial activations, particularly their expressive power. For a fixed architecture and activation degree, a polynomial neural network defines an algebraic map from weights to polynomials. The image of this map is the functional space associated to the network, and it is an irreducible algebraic variety upon taking closure. This paper proposes the dimension of this variety as a precise measure of the expressive power of polynomial neural networks. We obtain several theoretical results regarding this dimension as a function of architecture, including an exact formula for high activation degrees, as well as upper and lower bounds on layer widths in order for deep polynomials networks to fill the ambient functional space. We also present computational evidence that it is profitable in terms of expressiveness for layer widths to increase monotonically and then decrease monotonically. Finally, we link our study to favorable optimization properties when training weights, and we draw intriguing connections with tensor and polynomial decompositions.

Motivation & Objective

• Motivate a rigorous measure of expressivity for deep nonlinear networks using polynomial activations.
• Model networks as algebraic maps from weights to polynomials and study the Zariski closure as a functional variety.
• Derive exact dimension formulas in high activation degree and provide bounds on layer widths for filling the ambient space.
• Explore implications for optimization, and relate network representations to tensor and polynomial decompositions.

Proposed method

• Formulate polynomial networks with activations rho_r(z) = z^r and fixed architecture d = (d0,...,dh).
• Define the algebraic map Phi_{d,r} from weights to a vector of homogeneous polynomials and study its image F_{d,r}.
• Consider the Zariski closure V_{d,r} of F_{d,r} as an irreducible algebraic variety and analyze its dimension.
• Derive a general dimension bound and an exact dimension result for high activation degree (Theorem 14).
• Establish a bottleneck (non-filling) criterion for widths (Theorem 19).
• Relate networks to tensor decompositions (CP, LORS-2019) and provide upper bounds on filling widths (Theorem 10).
• Use symmetry and multi-homogeneity to study the structure of Phi_{d,r} (Lemma 13).
• Provide computational methods (Jacobians, backpropagation over polynomials or finite fields) to estimate dim(V_{d,r}).

Experimental results

Research questions

• RQ1 What is the dimension of the functional variety V_{d,r} induced by a polynomial network architecture?
• RQ2 When does a network architecture fill the ambient space, and how do layer widths influence this?
• RQ3 How does the activation degree r affect the dimension and filling properties of the network’s functional space?
• RQ4 What are the connections between polynomial networks and tensor/polynomial decompositions, and what implications do these have for optimization?
• RQ5 What are the bottlenecks in architecture that limit expressivity regardless of depth or width?

Key findings

• For fixed r and architecture d, the network’s functional space forms an algebraic variety whose dimension can be exactly characterized in high activation degree (Theorem 14).
• A widening/filling criterion shows that if widths are not large enough (bottlenecks), the network cannot fill the ambient space (Theorem 19).
• The dimension is bounded above by a formula combining layer dimensions and combinatorial factors (equation (11)); under a conjecture, equality holds for high r.
• There is a tight link between deep polynomial networks and tensor/polynomial decompositions, enabling upper bounds on filling widths (Theorem 10).
• Computational experiments using Jacobians/backpropagation confirm unimodal minimal filling widths and stabilization of dimension with increasing r (Tables 1–2 discussions).