[Paper Review] Constructing the Matrix Multilayer Perceptron and its Application to the VAE

Jalil Taghia, Maria Bånkestad+2 more|arXiv (Cornell University)|Feb 4, 2019

Generative Adversarial Networks and Image Synthesis31 references4 citations

TL;DR

This paper introduces the matrix multilayer perceptron (mMLP), a deep neural network architecture that learns symmetric positive definite (SPD) matrices by embedding the non-Euclidean geometry of SPD manifolds via the von Neumann divergence and a novel α-derivative-based backpropagation. The mMLP enables flexible variational autoencoders (VAEs) with dense covariance matrices in both recognition and generative networks, significantly improving likelihood and disentanglement without overfitting.

ABSTRACT

Like most learning algorithms, the multilayer perceptrons (MLP) is designed to learn a vector of parameters from data. However, in certain scenarios we are interested in learning structured parameters (predictions) in the form of symmetric positive definite matrices. Here, we introduce a variant of the MLP, referred to as the matrix MLP, that is specialized at learning symmetric positive definite matrices. We also present an application of the model within the context of the variational autoencoder (VAE). Our formulation of the VAE extends the vanilla formulation to the cases where the recognition and the generative networks can be from the parametric family of distributions with dense covariance matrices. Two specific examples are discussed in more detail: the dense covariance Gaussian and its generalization, the power exponential distribution. Our new developments are illustrated using both synthetic and real data.

Motivation & Objective

• To address the challenge of learning structured, symmetric positive definite (SPD) matrices in deep learning, where standard MLPs fail due to lack of constraint enforcement.
• To develop a neural network architecture that inherently respects the SPD constraint across all layers, avoiding invalid outputs.
• To extend the variational autoencoder (VAE) framework to support parametric families with dense covariance matrices, enhancing model flexibility.
• To enable end-to-end training of VAEs with both recognition and generative networks using flexible, SPD-structured distributions like multivariate Gaussian and power exponential.
• To demonstrate the effectiveness of the mMLP in both synthetic and real-world data, showing improved likelihood and meaningful latent space disentanglement.

Proposed method

• The mMLP uses positive-definite kernel functions in each layer to ensure output matrices remain symmetric and positive definite.
• It employs the von Neumann divergence as a loss function to respect the Riemannian geometry of the SPD manifold.
• A new backpropagation algorithm is derived using α-derivatives (Magnus, 2010), enabling gradient computation that preserves the SPD structure through all layers.
• The mMLP is integrated into the VAE framework, replacing standard MLPs in both the recognition and generative networks.
• The VAE objective is optimized using the ELBO, with the generative and inference models parameterized by mMLP to model dense covariance matrices.
• The model supports two key distributions: multivariate Gaussian with full covariance and multivariate power exponential (mPE), increasing flexibility beyond diagonal or low-rank assumptions.

Experimental results

Research questions

• RQ1 Can a deep neural network be designed to learn symmetric positive definite matrices while preserving the non-Euclidean geometry of the SPD manifold?
• RQ2 How can backpropagation be adapted to maintain SPD constraints across all layers in a deep network?
• RQ3 Does allowing flexible, dense covariance matrices in both the recognition and generative networks of a VAE improve model likelihood and disentanglement?
• RQ4 What is the impact of increasing flexibility in the recognition network versus the generative network on VAE performance?
• RQ5 Can the mMLP be effectively applied to real-world data while maintaining computational feasibility and avoiding overfitting?

Key findings

• The mMLP successfully enforces the SPD constraint across all layers using PD kernels and α-derivative-based backpropagation, ensuring valid matrix outputs at every stage.
• The most flexible model variant, with full covariance in both recognition and generative networks (mPE-mPE), achieved the highest likelihood and the largest KLD, indicating effective information coding in the latent space.
• The model with mPE-based recognition and generative networks outperformed Gaussian-based variants in likelihood and disentanglement, demonstrating the benefit of flexible parametric families.
• No signs of overfitting were observed despite increased model flexibility, suggesting robust generalization under complex covariance structures.
• The higher variance estimator (14) used in the mPE-mPE model may explain its slightly worse performance compared to the lower-variance alternative, highlighting the sensitivity of estimation in flexible models.
• The mMLP enables effective learning of SPD matrices in both synthetic and real-world data, with sample generation results showing meaningful structure in the latent space.