[Paper Review] Sharp bounds for the number of regions of maxout networks and vertices of Minkowski sums

Guido Montúfar, Yue Ren+1 more|arXiv (Cornell University)|Apr 16, 2021

Polynomial and algebraic computation52 references4 citations

TL;DR

This paper establishes sharp upper bounds on the number of linear regions in maxout neural networks by connecting them to the vertices of Minkowski sums of polytopes and tropical hypersurface arrangements. Using combinatorial geometry and tropical algebraic methods, it derives exact formulas and tight bounds for both shallow and deep maxout networks, significantly improving prior estimates and revealing connections to hyperplane arrangements and polytope theory.

ABSTRACT

We present results on the number of linear regions of the functions that can be represented by artificial feedforward neural networks with maxout units. A rank-k maxout unit is a function computing the maximum of $k$ linear functions. For networks with a single layer of maxout units, the linear regions correspond to the upper vertices of a Minkowski sum of polytopes. We obtain face counting formulas in terms of the intersection posets of tropical hypersurfaces or the number of upper faces of partial Minkowski sums, along with explicit sharp upper bounds for the number of regions for any input dimension, any number of units, and any ranks, in the cases with and without biases. Based on these results we also obtain asymptotically sharp upper bounds for networks with multiple layers.

Motivation & Objective

• To determine the maximum number of linear regions in shallow maxout networks with arbitrary input dimension, number of units, and ranks.
• To derive explicit upper bounds for the number of regions in maxout networks with and without biases.
• To establish connections between maxout network regions and the upper vertices of Minkowski sums of polytopes.
• To extend these bounds to deep maxout networks using recursive composition.
• To develop counting formulas for faces of maxout arrangements based on intersection posets and sub-arrangements.

Proposed method

• Mapping linear regions of maxout networks to upper vertices of Minkowski sums of polytopes via a geometric correspondence (Proposition 2.7).
• Using tropical geometry to model the nonlinear loci of maxout units as tropical hypersurfaces.
• Applying inclusion-exclusion principles on intersection posets of tropical arrangements to count regions.
• Deriving face counting formulas through sub-arrangements and Minkowski subsums in general position.
• Leveraging the Upper Bound Theorem for Minkowski sums to bound the number of vertices and upper vertices.
• Extending results from shallow to deep networks by composing layer-wise bounds asymptotically.

Experimental results

Research questions

• RQ1 What is the maximum number of linear regions that a shallow maxout network can represent, given its architecture?
• RQ2 How do biases affect the number of linear regions in maxout networks, and can sharp bounds be derived in both cases?
• RQ3 What is the relationship between the number of regions in maxout arrangements and the number of upper vertices in Minkowski sums of polytopes?
• RQ4 Can tight asymptotic bounds be established for deep maxout networks based on shallow network results?
• RQ5 What are the explicit formulas for the number of faces of any dimension in maxout arrangements?

Key findings

• The paper provides sharp upper bounds on the number of linear regions in shallow maxout networks, valid for any input dimension, number of units, and ranks, both with and without biases.
• For shallow maxout networks, the number of regions is bounded above by a sum over subsets of units, involving the difference in the number of regions of sub-arrangements and their generic counterparts.
• The maximum number of regions in a maxout layer corresponds exactly to the number of upper vertices of the Minkowski sum of the corresponding polytopes.
• The bounds are asymptotically tight for deep maxout networks, significantly improving upon previous estimates.
• A lower bound on the number of strict lower vertices (corresponding to bounded regions) is derived for Minkowski sums in general position.
• The results establish a precise connection between the combinatorics of maxout networks and the theory of tropical hypersurface arrangements and polytope Minkowski sums.