[Paper Review] An Improved Bound on the VC-Dimension of Neural Networks with Polynomial Activation Functions

J. Maurice Rojas, M. Vidyasagar|ArXiv.org|Dec 19, 2001

Neural Networks and Applications4 references3 citations

TL;DR

This paper presents a tighter upper bound on the VC-dimension of feedforward neural networks with polynomial activation functions by leveraging a refined semi-algebraic geometry result from Rojas (2000). By analyzing the degree distribution of weight parameters across network layers and using relative entropy to quantify improvement, the authors derive a bound that is strictly sharper than the classical Goldberg-Jerrum bound, reducing the estimate by up to 12.5% in numerical examples.

ABSTRACT

In this note, we derive an improved upper bound for the VC-dimension of neural networks with polynomial activation functions. This improved bound is based on a result of Rojas on the number of connected components of a semi-algebraic set.

Motivation & Objective

• To improve upon the classical VC-dimension upper bound for neural networks with polynomial activation functions.
• To address the looseness of prior bounds derived from Milnor's theorem, which relies only on total degree and number of variables.
• To incorporate layer-specific degree distributions of weight parameters into the VC-dimension analysis.
• To quantify the improvement in the bound using relative entropy between probability vectors derived from layer-wise parameter counts and degrees.
• To demonstrate that the new bound is both tighter and intuitively interpretable in terms of network architecture

Proposed method

• Utilizes a result by Rojas (2000) on the number of connected components of semi-algebraic sets, which depends on the degrees of polynomials and their variable groupings.
• Models the neural network as a Boolean combination of polynomial inequalities and equalities in the weight space.
• Defines layer-specific degree bounds $d_i$ for each weight vector $\mathbf{w}_i$, capturing the maximum total degree of monomials involving $\mathbf{w}_i$.
• Applies volume bounds on the set of critical values using the product of multinomial coefficients and factorials: $k! \prod_{i=1}^l \frac{d_i^{k_i}}{k_i!}$.
• Uses Stirling's approximation to derive a closed-form upper bound involving the relative entropy $H(\mathbf{v}|\mathbf{u})$ of two probability vectors $\mathbf{v}$ and $\mathbf{u}$.
• Combines the refined volume bound with Theorem 1 to derive the final VC-dimension upper bound: $2k(\lg(4ed) - H(\mathbf{v}|\mathbf{u}))$

Experimental results

Research questions

• RQ1 Can the VC-dimension of neural networks with polynomial activation functions be bounded more tightly by exploiting layer-wise degree distributions?
• RQ2 How does the relative entropy between the normalized layer-degree vector and the normalized layer-parameter vector affect the VC-dimension bound?
• RQ3 To what extent can the classical Milnor-type bound be improved when more refined information about polynomial degrees per layer is used?
• RQ4 Is the improvement in the bound quantitatively significant and interpretable in terms of network architecture?
• RQ5 Under what conditions does the new bound reduce the estimate by a non-trivial fraction compared to the Goldberg-Jerrum bound?

Key findings

• The proposed bound improves upon the classical Goldberg-Jerrum bound by incorporating layer-specific degree distributions of weight parameters.
• The improvement is quantified as $2k \cdot H(\mathbf{v}|\mathbf{u})$, where $H(\mathbf{v}|\mathbf{u})$ is the relative entropy between the normalized degree vector $\mathbf{v}$ and the normalized parameter count vector $\mathbf{u}$.
• In a numerical example with $k_1=50$, $k_2=20$, $d_1=3$, $d_2=1$, the bound is approximately 12.5% tighter than the prior estimate.
• The bound is strictly tighter than the Goldberg-Jerrum bound in all cases, with no improvement only when the degree and parameter distributions are perfectly aligned across layers.
• The method provides a principled way to relate network architecture (layer sizes and activation degrees) to generalization capacity via VC-dimension.
• The final bound is expressed as $\mathrm{VCDIM}(\Phi) \leq 2k(\lg(4ed) - H(\mathbf{v}|\mathbf{u}))$, which is tighter than $2k\lg(4ed)$ when $H(\mathbf{v}|\mathbf{u}) > 0$.