[Paper Review] $\Pi-$nets: Deep Polynomial Neural Networks

Grigorios G. Chrysos, Stylianos Moschoglou+4 more|arXiv (Cornell University)|Mar 8, 2020

Advanced Neural Network Applications62 references4 citations

TL;DR

This paper introduces $Π$-Nets, a novel class of deep convolutional neural networks that express high-order polynomial functions of the input through specialized skip connections, enabling powerful representation learning without relying on non-linear activation functions. The method achieves state-of-the-art performance in image generation and classification, even with fewer parameters, and explains the success of models like StyleGAN through polynomial expressivity.

ABSTRACT

Deep Convolutional Neural Networks (DCNNs) is currently the method of choice both for generative, as well as for discriminative learning in computer vision and machine learning. The success of DCNNs can be attributed to the careful selection of their building blocks (e.g., residual blocks, rectifiers, sophisticated normalization schemes, to mention but a few). In this paper, we propose $\\Pi$-Nets, a new class of DCNNs. $\\Pi$-Nets are polynomial neural networks, i.e., the output is a high-order polynomial of the input. $\\Pi$-Nets can be implemented using special kind of skip connections and their parameters can be represented via high-order tensors. We empirically demonstrate that $\\Pi$-Nets have better representation power than standard DCNNs and they even produce good results without the use of non-linear activation functions in a large battery of tasks and signals, i.e., images, graphs, and audio. When used in conjunction with activation functions, $\\Pi$-Nets produce state-of-the-art results in challenging tasks, such as image generation. Lastly, our framework elucidates why recent generative models, such as StyleGAN, improve upon their predecessors, e.g., ProGAN.

Motivation & Objective

• To develop a new class of deep neural networks with enhanced representation power through polynomial function approximation.
• To explain the success of recent generative models like StyleGAN by framing their skip connections as inducing higher-order polynomial behavior.
• To demonstrate that polynomial networks can achieve strong performance in both generative and discriminative tasks without non-linear activation functions.
• To enable efficient parameterization of high-order polynomials using tensor decompositions via specialized skip connections.
• To show that existing architectures can be significantly improved by replacing standard layers with $Π$-Net components, even with minimal architectural changes.

Proposed method

• The core mechanism uses a special type of skip connection that enables the network output to be a high-order polynomial of the input, avoiding the parameter explosion of direct polynomial parameterization.
• The architecture is built using tensorial factors derived from structured tensor decompositions, which efficiently represent the polynomial terms.
• The method generalizes the standard deep network paradigm by replacing activation functions with a compositional polynomial expansion through skip connections.
• The framework is applied to both discriminative tasks (e.g., image classification) and generative tasks (e.g., GANs), with minimal architectural modifications.
• The approach is extended to graph-based tasks using spiral convolutions, where polynomial layers are applied to mesh and graph signals.
• A plug-and-play operator called ProdPoly is introduced, which can be used to upgrade existing models like ResNet or GNNs with polynomial expressivity.

Experimental results

Research questions

• RQ1 Can deep neural networks achieve strong performance without non-linear activation functions by leveraging polynomial function approximation?
• RQ2 How do skip connections in models like StyleGAN contribute to improved generative performance, and can this be formalized as polynomial expansion?
• RQ3 Can high-order polynomial representations be efficiently parameterized in deep networks without excessive parameter growth?
• RQ4 To what extent can existing state-of-the-art models be improved by replacing standard layers with polynomial counterparts?
• RQ5 Does polynomial expressivity enhance performance across diverse signals such as images, audio, and 3D meshes?

Key findings

• Prodpoly-ResNet50 achieves a Top-5 validation error of 6.358%, outperforming standard ResNet50 (6.838%) with only a small increase in model size and computation.
• On the 3D mesh autoencoding task, the ProdPoly (full) model achieves a reconstruction error of 0.474 mm, outperforming prior methods like SpiralGNN (0.635 mm) and MoNet (0.583 mm).
• Even without activation functions, ProdPoly-based models significantly outperform state-of-the-art graph learning operators in mesh representation learning, with lower reconstruction error and comparable inference speed.
• The proposed $Π$-Net framework improves performance across image generation, classification, and audio tasks, demonstrating consistent gains with minimal architectural changes.
• The method explains the success of StyleGAN over ProGAN as stemming from the induction of higher-order polynomial behavior via style injection layers.
• The framework enables plug-and-play upgrades to existing models, such as converting ResNet or GNNs into polynomial networks with measurable performance gains.