[Paper Review] A Novel Representation of Neural Networks

Anthony L. Caterini, Dong Eui Chang|arXiv (Cornell University)|Oct 5, 2016

Gaussian Processes and Bayesian Inference2 references3 citations

TL;DR

This paper introduces a coordinate-free, inner product space-based mathematical framework for deep neural networks (DNNs) that treats parameters as whole entities rather than scalar components. By leveraging adjoints, bilinear maps, and chain rule in abstract inner product spaces, it enables a unified, compact derivation of backpropagation and gradient descent for Multilayer Perceptrons and Deep Autoencoders, significantly simplifying the theoretical foundation of DNN optimization.

ABSTRACT

Deep Neural Networks (DNNs) have become very popular for prediction in many areas. Their strength is in representation with a high number of parameters that are commonly learned via gradient descent or similar optimization methods. However, the representation is non-standardized, and the gradient calculation methods are often performed using component-based approaches that break parameters down into scalar units, instead of considering the parameters as whole entities. In this work, these problems are addressed. Standard notation is used to represent DNNs in a compact framework. Gradients of DNN loss functions are calculated directly over the inner product space on which the parameters are defined. This framework is general and is applied to two common network types: the Multilayer Perceptron and the Deep Autoencoder.

Motivation & Objective

• To address the lack of standardized mathematical notation in deep learning that hinders theoretical consistency and composability.
• To overcome the limitations of component-wise gradient computation in backpropagation by treating network parameters as whole entities in inner product spaces.
• To develop a general, compact, and coordinate-free mathematical framework applicable to diverse DNN architectures like MLPs and autoencoders.
• To enable cleaner, more interpretable derivations of gradients and optimization algorithms by avoiding vectorization and component decomposition.
• To lay a foundation for extending the framework to other architectures such as CNNs and RNNs.

Proposed method

• Formalizing DNNs using abstract inner product spaces and linear maps, with parameters defined as elements of these spaces.
• Defining derivatives via the standard directional derivative in inner product spaces, avoiding component-wise decomposition.
• Using adjoint operators and hook operations (left/right) to express derivatives and their transposes cleanly.
• Applying the chain rule in a coordinate-free manner to derive backpropagation without explicit matrix flattening.
• Deriving gradient descent updates directly on the parameter space using adjoint-based gradient computation.
• Applying the framework to Multilayer Perceptrons and Deep Autoencoders, including higher-order loss functions with explicit gradient computation.

Experimental results

Research questions

• RQ1 How can deep neural networks be formally represented using a unified, coordinate-free mathematical framework based on inner product spaces?
• RQ2 What is the role of adjoint operators and hook operations in simplifying the derivation of backpropagation?
• RQ3 How can gradient descent be formulated directly on the parameter space without decomposing matrices into vectors?
• RQ4 Can this framework be extended to handle higher-order loss functions and regularization terms?
• RQ5 How does this approach improve the theoretical clarity and generality of DNN optimization compared to traditional component-based methods?

Key findings

• The framework enables a coordinate-free derivation of backpropagation by treating parameters as elements of inner product spaces, eliminating the need for vectorization.
• Gradients of loss functions are computed directly over the parameter space using adjoint operators, leading to a more natural and compact formulation.
• The method successfully derives gradient descent updates for both Multilayer Perceptrons and Deep Autoencoders using the same abstract formalism.
• Higher-order loss functions, including those with second-order terms, are handled consistently through the use of second derivatives and adjoint-based gradient computation.
• The approach provides a clear path to extending to other architectures like CNNs and RNNs by leveraging the same underlying mathematical structure.
• The framework enhances theoretical transparency and reduces notational ambiguity in deep learning research.