[Paper Review] Polynomial Regression As an Alternative to Neural Nets

Xi Cheng, Bohdan B. Khomtchouk+2 more|arXiv (Cornell University)|Jun 13, 2018

Neural Networks and ApplicationsComputer Science22 references73 citations

TL;DR

The paper argues that neural networks function essentially as polynomial regression with increasing polynomial degree per layer, and shows that polynomial regression (PR) can match or outperform NNs across many datasets, proposing polyreg as a practical alternative.

ABSTRACT

Despite the success of neural networks (NNs), there is still a concern among many over their "black box" nature. Why do they work? Here we present a simple analytic argument that NNs are in fact essentially polynomial regression models. This view will have various implications for NNs, e.g. providing an explanation for why convergence problems arise in NNs, and it gives rough guidance on avoiding overfitting. In addition, we use this phenomenon to predict and confirm a multicollinearity property of NNs not previously reported in the literature. Most importantly, given this loose correspondence, one may choose to routinely use polynomial models instead of NNs, thus avoiding some major problems of the latter, such as having to set many tuning parameters and dealing with convergence issues. We present a number of empirical results; in each case, the accuracy of the polynomial approach matches or exceeds that of NN approaches. A many-featured, open-source software package, polyreg, is available.

Motivation & Objective

• Demonstrate a theoretical NN ↔ PR correspondence and explain why NNs behave like higher-degree polynomials across layers.
• Show that increasing polynomial degree across layers explains convergence and overfitting issues in NNs.
• Provide empirical evidence that PR can match or outperform NNs on diverse datasets.
• Offer practical guidance and open-source tooling (polyreg) for applying PR as an alternative to NNs.
• Discuss implications for multicollinearity and regularization in neural network contexts.

Proposed method

• Provide informal mathematical argument that NN activations approximate polynomial regression, with degree increasing by layer.
• Invoke Stone–Weierstrass theorem to argue activation functions can be approximated by polynomials, yielding PR behavior for NNs.
• Analyze ReLU as piecewise polynomial, leading to piecewise polynomial regression (PPR) view.
• Empirically compare PR (via polyreg) to NNs across multiple datasets, reporting mean absolute prediction error (MAPE) or proportion correct (PCC).
• Examine multicollinearity across NN layers using variance inflation factors (VIF) and interpret implications for convergence and regularization.

Experimental results

Research questions

• RQ1 Is there a concrete NN ↔ PR correspondence that holds across typical feedforward networks?
• RQ2 Does the effective polynomial degree increase from layer to layer in NNs, and how does this affect learning and convergence?
• RQ3 Can PR with appropriate polynomial features match or exceed NN performance on a variety of datasets?
• RQ4 What implications does the NN ↔ PR view have for multicollinearity, convergence, and regularization in neural networks?

Key findings

• NNs can be viewed as a form of polynomial regression, with the degree of the approximating polynomial growing with each hidden layer.
• Activation functions can be approximated by polynomials, implying NNs effectively implement PR; ReLU is a piecewise polynomial, leading to piecewise polynomial regression interpretations.
• PR often matches or exceeds NN performance across diverse datasets, including census wages, song year prediction, concrete strength, letter recognition, NYC taxi duration, forest cover, MOOCs certification, Crossfit rankings, cancer genomics, MNIST, and political election data.
• Multicollinearity in NN layers increases with depth, similar to high-degree PR, suggesting layer-by-layer checks and regularization to mitigate convergence issues.
• The polyreg open-source package provides practical tools for implementing PR and comparing with NN approaches.