[Paper Review] Multilayer Perceptron Algebra

Peng Zhao|arXiv (Cornell University)|Jan 18, 2017

Neural Networks and Applications4 references3 citations

TL;DR

This paper introduces MLP algebra, a mathematical framework that enables systematic design and composition of Multilayer Perceptron (MLP) neural networks through algebraic operations such as sum, product, complement, and component decomposition. By treating MLPs as algebraic objects, the theory allows constructing complex networks from simpler, task-specific subnetworks trained on decomposed data, preserving accuracy while providing a principled approach to network architecture design.

ABSTRACT

Artificial Neural Networks(ANN) has been phenomenally successful on various pattern recognition tasks. However, the design of neural networks rely heavily on the experience and intuitions of individual developers. In this article, the author introduces a mathematical structure called MLP algebra on the set of all Multilayer Perceptron Neural Networks(MLP), which can serve as a guiding principle to build MLPs accommodating to the particular data sets, and to build complex MLPs from simpler ones.

Motivation & Objective

• To address the lack of systematic principles in designing neural networks, which currently rely heavily on intuition and experience.
• To develop a formal mathematical structure—MLP algebra—that enables the construction of complex MLPs from simpler, task-specific networks.
• To provide explicit algebraic operations for combining or decomposing neural networks while maintaining classification performance on original data subsets.
• To offer a theoretical foundation for integrating networks trained on disjoint or labeled subsets into a unified, accurate classifier.
• To explore how geometric and structural properties of data can guide network composition via algebraic operations.

Proposed method

• Defining the Complementary Net operation to invert network outputs, enabling logical negation of classification results.
• Introducing the Sum Net operation to combine two MLPs with identical input and output layers by concatenating hidden layers via Cartesian product.
• Proposing the I-Product Net for combining networks trained on disjoint, non-overlapping data subsets, using block-diagonal weight matrices.
• Introducing the O-Product Net to integrate multiple single-label classifiers into a multi-label network using direct sum of output layers.
• Defining Component Net to decompose a multi-output network into individual single-output classifiers for each label.
• Introducing Identical Extension (β) to align networks with different depths by adding identity layers with ReLU activation, enabling consistent composition across varying architectures.

Experimental results

Research questions

• RQ1 How can neural networks be systematically composed from simpler, task-specific subnetworks trained on decomposed data?
• RQ2 What algebraic operations on MLPs preserve classification accuracy when combining networks trained on disjoint or labeled data subsets?
• RQ3 Can a formal mathematical structure be defined for MLPs that supports operations like addition, multiplication, and complementation similar to algebraic fields?
• RQ4 How can networks with different architectures (e.g., different depths) be composed consistently using algebraic rules?
• RQ5 What is the role of data decomposition (e.g., by connected components or labels) in enabling effective network composition via MLP algebra?

Key findings

• The MLP algebra framework enables the construction of a single, unified neural network that maintains high classification accuracy on a combined dataset by integrating subnetworks trained on disjoint or labeled subsets.
• The Sum Net operation successfully combines two networks trained on different connected components of a dataset, preserving performance on each component.
• The O-Product Net operation allows integration of multiple single-label classifiers into a single multi-label network, with the output layer representing the direct sum of individual label outputs.
• The Component Net operation enables decomposition of a multi-output network into individual single-output networks, each corresponding to one label, facilitating modular training and analysis.
• The Identical Extension (β) operation enables consistent composition of networks with different depths by extending shorter networks with identity layers using ReLU activation, ensuring compatibility in layer structure.
• The theory demonstrates that complex networks can be systematically built from simpler ones through algebraic operations, offering a principled alternative to heuristic design.