[Paper Review] Matrix Neural Networks

Junbin Gao, Yi Guo+1 more|arXiv (Cornell University)|Jan 15, 2016

Neural Networks and Applications24 references4 citations

TL;DR

This paper proposes Matrix Neural Networks (MatNet), a novel neural network architecture that processes matrix-shaped inputs directly without vectorization, preserving spatial structure and reducing model complexity. By using bilinear mappings between matrix layers and enabling standard backpropagation, MatNet achieves performance comparable to state-of-the-art methods—such as CNNs—in MNIST classification and image super-resolution—while significantly lowering computational cost and parameter complexity.

ABSTRACT

Traditional neural networks assume vectorial inputs as the network is arranged as layers of single line of computing units called neurons. This special structure requires the non-vectorial inputs such as matrices to be converted into vectors. This process can be problematic. Firstly, the spatial information among elements of the data may be lost during vectorisation. Secondly, the solution space becomes very large which demands very special treatments to the network parameters and high computational cost. To address these issues, we propose matrix neural networks (MatNet), which takes matrices directly as inputs. Each neuron senses summarised information through bilinear mapping from lower layer units in exactly the same way as the classic feed forward neural networks. Under this structure, back prorogation and gradient descent combination can be utilised to obtain network parameters efficiently. Furthermore, it can be conveniently extended for multimodal inputs. We apply MatNet to MNIST handwritten digits classification and image super resolution tasks to show its effectiveness. Without too much tweaking MatNet achieves comparable performance as the state-of-the-art methods in both tasks with considerably reduced complexity.

Motivation & Objective

• Address the loss of spatial correlation and increased model complexity caused by vectorizing matrix inputs (e.g., images) in traditional neural networks.
• Reduce the solution space and computational burden associated with high-dimensional parameter spaces in vectorized deep networks.
• Preserve interpretability and enable efficient training by maintaining matrix structure throughout the network.
• Demonstrate that direct matrix processing can achieve competitive performance in vision tasks without extensive hyperparameter tuning.
• Extend the framework to multimodal inputs for complex tasks like image super-resolution.

Proposed method

• Propose a matrix neural network (MatNet) where input and hidden layers are organized as matrices, with each neuron receiving bilinear mappings from the previous layer’s outputs.
• Use a bilinear transformation to compute neuron activations: $ z = \mathbf{W}^T \cdot \mathbf{x} \cdot \mathbf{v} + b $, where $ \mathbf{x} $ is the input matrix and $ \mathbf{v} $ is a learnable vector, preserving matrix structure.
• Apply standard backpropagation and gradient descent for efficient parameter optimization, enabled by the differentiability of the bilinear mapping.
• Extend MatNet to multimodal inputs by fusing multiple data streams (e.g., image patches, gradients) through shared or parallel matrix layers.
• Incorporate regularization techniques such as weight decay ($ \lambda = 0.001 $) and sparsity constraints ($ \rho = 0.05 $) to improve generalization.
• Train MatNet end-to-end on image datasets using stochastic gradient descent with standard activation functions like ReLU and sigmoid.

Experimental results

Research questions

• RQ1 Can a neural network that processes matrix inputs directly outperform vectorized networks in tasks like image classification and super-resolution?
• RQ2 To what extent does preserving matrix structure reduce model complexity and improve training efficiency compared to standard vector-based networks?
• RQ3 How does MatNet perform in comparison to state-of-the-art methods like CNNs and SR (super-resolution) in terms of accuracy and computational cost?
• RQ4 Can MatNet be effectively extended to multimodal inputs, such as combining image patches and their derivatives, for complex vision tasks?
• RQ5 Does MatNet maintain interpretability and avoid the 'black box' problem by preserving spatial correlations in data?

Key findings

• MatNet achieved 98.48% test accuracy on MNIST with a 4-layer architecture (two hidden layers of $160 \times 160$ neurons and 316,160 parameters), approaching CNN performance despite no hyperparameter optimization.
• In image super-resolution with $ s = 2 $, MatNet achieved PSNR of 33.966 dB on the Lena image and 36.056 dB on the kingfisher image, closely matching the state-of-the-art SR method (35.037 dB and 36.541 dB respectively).
• For 12 test images of size $ 256 \times 256 $, MatNet’s PSNR results were consistently near those of the SR method, especially for images 5 and 8, indicating strong generalization.
• The learned weights in MatNet’s super-resolution model revealed scaling filters that adaptively capture high-frequency details, suggesting effective feature learning.
• MatNet’s performance was achieved with minimal architectural tuning, indicating strong potential for further improvement with optimization techniques like ReLU and max-pooling.
• The model’s reduced parameter count and direct matrix processing significantly lower computational cost compared to standard deep networks, especially for large-scale data.