[Paper Review] ManifoldNet: A Deep Network Framework for Manifold-valued Data

Rudrasis Chakraborty, Jose Bouza+2 more|arXiv (Cornell University)|Sep 11, 2018

Human Pose and Action Recognition9 references16 citations

TL;DR

ManifoldNet introduces a deep learning framework for manifold-valued data by replacing standard convolutions with a provably convergent recursive weighted Fréchet Mean (wFM) layer, enabling equivariance to isometry group actions and eliminating the need for ReLU or pooling layers. The method achieves state-of-the-art performance in video classification and image reconstruction with reduced overfitting and parameter efficiency.

ABSTRACT

Deep neural networks have become the main work horse for many tasks involving learning from data in a variety of applications in Science and Engineering. Traditionally, the input to these networks lie in a vector space and the operations employed within the network are well defined on vector-spaces. In the recent past, due to technological advances in sensing, it has become possible to acquire manifold-valued data sets either directly or indirectly. Examples include but are not limited to data from omnidirectional cameras on automobiles, drones etc., synthetic aperture radar imaging, diffusion magnetic resonance imaging, elastography and conductance imaging in the Medical Imaging domain and others. Thus, there is need to generalize the deep neural networks to cope with input data that reside on curved manifolds where vector space operations are not naturally admissible. In this paper, we present a novel theoretical framework to generalize the widely popular convolutional neural networks (CNNs) to high dimensional manifold-valued data inputs. We call these networks, ManifoldNets. In ManifoldNets, convolution operation on data residing on Riemannian manifolds is achieved via a provably convergent recursive computation of the weighted Fréchet Mean (wFM) of the given data, where the weights makeup the convolution mask, to be learned. Further, we prove that the proposed wFM layer achieves a contraction mapping and hence ManifoldNet does not need the non-linear ReLU unit used in standard CNNs. We present experiments, using the ManifoldNet framework, to achieve dimensionality reduction by computing the principal linear subspaces that naturally reside on a Grassmannian. The experimental results demonstrate the efficacy of ManifoldNets in the context of classification and reconstruction accuracy.

Motivation & Objective

• To address the lack of deep learning frameworks for data that reside on curved Riemannian manifolds, such as symmetric positive definite matrices, spheres, and Grassmannians.
• To generalize convolutional neural networks (CNNs) by replacing vector-space convolutions with intrinsic operations suitable for non-Euclidean data.
• To develop a layer that performs weighted Fréchet Mean computation on manifolds, ensuring convergence and enabling weight sharing via group equivariance.
• To eliminate the need for ReLU and pooling layers by leveraging the inherent pooling property of the wFM operation.
• To demonstrate the framework's efficacy in video classification and image reconstruction tasks using autoencoder architectures.

Proposed method

• Proposes a novel network layer based on the weighted Fréchet Mean (wFM) as a manifold analog to convolution, replacing standard linear and ReLU operations in CNNs.
• Introduces a provably convergent recursive algorithm for computing the wFM, ensuring stable and efficient optimization on Riemannian manifolds.
• Proves that the wFM layer is equivariant to the action of the isometry group of the underlying Riemannian manifold, enabling parameter sharing across spatial locations.
• Designs a manifold-aware autoencoder+decoder architecture using the wFM layer for dimensionality reduction and reconstruction, replacing fully connected layers.
• Employs Grassmann averaging and iFME (iterative Fréchet Mean Estimator) layers to compute low-dimensional representations of manifold-valued data.
• Uses intrinsic Riemannian geometry to ensure all operations—such as mean computation and layer propagation—remain within the manifold structure without embedding into Euclidean space.

Experimental results

Research questions

• RQ1 Can a deep neural network be generalized to handle manifold-valued data using only intrinsic Riemannian operations?
• RQ2 Does a weighted Fréchet Mean layer provide a stable, convergent, and equivariant alternative to standard convolution and ReLU in deep networks?
• RQ3 Can the wFM layer serve as a natural pooling operation, eliminating the need for additional pooling or non-linear activation layers?
• RQ4 How does the ManifoldNet framework compare to standard autoencoders and PCA in terms of reconstruction quality and generalization on manifold-valued data?
• RQ5 To what extent does the intrinsic geometry of the data manifold improve generalization and reduce overfitting in downstream tasks?

Key findings

• The proposed ManifoldNet framework achieves a 46% reduction in parameters compared to a standard fully connected autoencoder, primarily due to the replacement of large fully connected layers with Grassmann averaging layers.
• The iFME+autoencoder architecture converges faster than a standard autoencoder, achieving lower reconstruction error in less computation time, as shown in the time-vs-error plot (Fig. 4).
• On the MNIST dataset, the ManifoldNet-based autoencoder+iFME achieved a test set classification accuracy of 46.00% when using a classifier on reconstructed data, outperforming both the standard autoencoder (40.00%) and PCA (26.00%).
• The iFME layer acts as a regularizer, improving generalization: while the standard autoencoder slightly outperforms ManifoldNet on the training set, ManifoldNet significantly improves performance on the test set.
• Visual reconstructions from the iFME+autoencoder on a 1280×720 video sequence show higher visual quality than PCA and standard autoencoders, with full video reconstruction available for comparison.
• The framework demonstrates state-of-the-art performance in video classification and reconstruction, with the wFM layer achieving contraction mapping behavior and eliminating the need for ReLU units.