[Paper Review] From Two-Class Linear Discriminant Analysis to Interpretable Multilayer Perceptron Design

Ruiyuan Lin, Zhiruo Zhou+3 more|arXiv (Cornell University)|Sep 9, 2020

Blind Source Separation Techniques43 references9 citations

TL;DR

This paper proposes a feedforward, interpretable multilayer perceptron (FF-MLP) design that generalizes two-class linear discriminant analysis (LDA) to multi-class, multi-modal Gaussian data. By structuring the network in three stages—half-space partitioning via parallel LDA, region isolation, and class-specific merging—the method determines architecture and weights in a single forward pass without backpropagation, achieving faster training and improved interpretability compared to traditional BP-MLPs.

ABSTRACT

A closed-form solution exists in two-class linear discriminant analysis (LDA), which discriminates two Gaussian-distributed classes in a multi-dimensional feature space. In this work, we interpret the multilayer perceptron (MLP) as a generalization of a two-class LDA system so that it can handle an input composed by multiple Gaussian modalities belonging to multiple classes. Besides input layer $l_{in}$ and output layer $l_{out}$, the MLP of interest consists of two intermediate layers, $l_1$ and $l_2$. We propose a feedforward design that has three stages: 1) from $l_{in}$ to $l_1$: half-space partitionings accomplished by multiple parallel LDAs, 2) from $l_1$ to $l_2$: subspace isolation where one Gaussian modality is represented by one neuron, 3) from $l_2$ to $l_{out}$: class-wise subspace mergence, where each Gaussian modality is connected to its target class. Through this process, we present an automatic MLP design that can specify the network architecture (i.e., the layer number and the neuron number at a layer) and all filter weights in a feedforward one-pass fashion. This design can be generalized to an arbitrary distribution by leveraging the Gaussian mixture model (GMM). Experiments are conducted to compare the performance of the traditional backpropagation-based MLP (BP-MLP) and the new feedforward MLP (FF-MLP).

Motivation & Objective

• To address the lack of theoretical guidance in multilayer perceptron (MLP) architecture and weight design, which currently relies on time-consuming trial-and-error.
• To establish a formal connection between two-class linear discriminant analysis (LDA) and multi-class MLPs for improved interpretability and design efficiency.
• To develop a one-pass, feedforward method that determines network depth, neuron count, and all filter weights without backpropagation or hyperparameter tuning.
• To demonstrate that the proposed feedforward MLP (FF-MLP) outperforms traditional backpropagation-based MLPs (BP-MLP) in design speed, training time, and interpretability.
• To extend the framework to general data distributions using Gaussian mixture models (GMMs) while maintaining analytical tractability.

Proposed method

• The method generalizes two-class LDA to multi-class, multi-modal data by modeling each Gaussian modality as a distinct class region in the input space.
• Stage 1 partitions the input space into multiple half-subspaces using parallel two-class LDA systems, forming intersecting regions of interest.
• Stage 2 isolates each region using a second hidden layer where each neuron corresponds to one Gaussian modality, activated only within its designated region.
• Stage 3 maps each isolated region to its target class via class-wise connections in the output layer, enabling direct classification.
• The entire network is constructed in a one-pass feedforward manner using closed-form LDA solutions and ReLU activation to resolve sign confusion in cascaded layers.
• Gaussian Mixture Models (GMMs) are used to approximate arbitrary data distributions, allowing the method to generalize beyond strictly Gaussian data.

Experimental results

Research questions

• RQ1 Can two-class linear discriminant analysis be systematically generalized to serve as a theoretical foundation for multilayer perceptron design?
• RQ2 Can a multilayer perceptron be constructed in a one-pass, feedforward manner that determines both architecture and weights without backpropagation?
• RQ3 Does the proposed feedforward MLP (FF-MLP) achieve better interpretability and faster training than traditional backpropagation-based MLPs (BP-MLPs)?
• RQ4 How does the performance of FF-MLP compare to BP-MLP in terms of accuracy, training time, and design efficiency across different data distributions?
• RQ5 Can the FF-MLP framework be extended to non-Gaussian data using Gaussian mixture models while preserving analytical tractability and interpretability?

Key findings

• The proposed FF-MLP design determines the number of neurons in each layer and all filter weights in a single forward pass, eliminating the need for iterative backpropagation or hyperparameter tuning.
• Experiments show that FF-MLP achieves comparable or better classification accuracy than BP-MLP while significantly reducing training and design time.
• The method provides explicit, interpretable intermediate layers that correspond to distinct data regions and class associations, enhancing model transparency.
• The use of ReLU activation resolves sign confusion in cascaded layers, enabling stable and interpretable signal propagation across stages.
• The framework generalizes to arbitrary distributions via Gaussian Mixture Models (GMMs), allowing modeling of complex, multi-modal data with varying covariance structures.
• Despite the absence of backpropagation, the FF-MLP achieves strong performance, suggesting that for compact networks, end-to-end optimization may not be necessary.