[Paper Review] Measuring Model Complexity of Neural Networks with Curve Activation Functions

Xia Hu, Weiqing Liu+2 more|arXiv (Cornell University)|Jun 16, 2020

Adversarial Robustness in Machine Learning35 references4 citations

TL;DR

This paper proposes a complexity measure for deep neural networks with curve activation functions by introducing a piecewise linear approximation framework called LANN, which minimizes linear regions while maintaining approximation accuracy. The method reveals a strong positive correlation between rising model complexity and overfitting, leading to effective regularization strategies like neuron pruning and customized L1 regularization that suppress complexity and improve generalization.

ABSTRACT

It is fundamental to measure model complexity of deep neural networks. The existing literature on model complexity mainly focuses on neural networks with piecewise linear activation functions. Model complexity of neural networks with general curve activation functions remains an open problem. To tackle the challenge, in this paper, we first propose the linear approximation neural network (LANN for short), a piecewise linear framework to approximate a given deep model with curve activation function. LANN constructs individual piecewise linear approximation for the activation function of each neuron, and minimizes the number of linear regions to satisfy a required approximation degree. Then, we analyze the upper bound of the number of linear regions formed by LANNs, and derive the complexity measure based on the upper bound. To examine the usefulness of the complexity measure, we experimentally explore the training process of neural networks and detect overfitting. Our results demonstrate that the occurrence of overfitting is positively correlated with the increase of model complexity during training. We find that the $L^1$ and $L^2$ regularizations suppress the increase of model complexity. Finally, we propose two approaches to prevent overfitting by directly constraining model complexity, namely neuron pruning and customized $L^1$ regularization.

Motivation & Objective

• To address the open problem of measuring model complexity in deep neural networks with curve activation functions, such as Sigmoid and Tanh.
• To develop a piecewise linear approximation framework that accurately represents models with curve activations while minimizing the number of linear regions.
• To establish a complexity measure based on the upper bound of linear regions in the approximation, enabling analysis of model nonlinearity.
• To empirically validate the link between increasing model complexity and overfitting during training.
• To propose and evaluate direct complexity-constraining regularization techniques—neuron pruning and customized L1 regularization—for improved generalization.

Proposed method

• Introduce Linear Approximation Neural Network (LANN), a framework that approximates each neuron's curve activation function with a piecewise linear function.
• Optimize the approximation to minimize the number of linear regions while satisfying a user-defined approximation error threshold.
• Derive an upper bound on the number of linear regions formed by LANNs, which serves as the model complexity measure.
• Use the complexity measure to analyze training dynamics and correlate complexity growth with overfitting.
• Propose neuron pruning based on the expected absolute value of the activation function's derivative to suppress nonlinear regions.
• Design customized L1 regularization where each neuron's weight penalty is proportional to the expected absolute value of its activation derivative.

Experimental results

Research questions

• RQ1 How can model complexity be meaningfully measured in deep neural networks with curve activation functions, given that existing methods focus only on piecewise linear activations?
• RQ2 To what extent does model complexity increase during training, and is this increase correlated with overfitting?
• RQ3 Can L1 and L2 regularization effectively suppress the growth of model complexity as measured by the proposed metric?
• RQ4 Can direct complexity control through neuron pruning and customized L1 regularization prevent overfitting while maintaining model performance?

Key findings

• The proposed complexity measure, based on the upper bound of linear regions in LANN approximations, effectively captures the nonlinearity of models with curve activations.
• A strong positive correlation was observed between increasing model complexity during training and the onset of overfitting, as demonstrated on CIFAR and MOON datasets.
• L1 and L2 regularization were found to suppress the rise in model complexity, with L1 regularization showing a more pronounced effect.
• Neuron pruning based on the expected absolute derivative of the activation function effectively limits complexity growth and prevents overfitting with minimal performance drop.
• Customized L1 regularization, which assigns higher penalty weights to neurons with more linear activation distributions, outperforms standard L1 regularization, especially under high penalty settings.
• On the MOON dataset, models trained with neuron pruning (PR) and customized L1 (C-L1) achieved complexity measures of 25.02 and 25.11, respectively, with only 182 and 356 linear regions, significantly lower than standard L1 (382 regions) and L2 (545 regions), while maintaining strong performance.