[Paper Review] Learning Deep Matrix Representations

Kien Do, Truyen Tran+1 more|arXiv (Cornell University)|Mar 4, 2017

Advanced Graph Neural Networks22 references9 citations

TL;DR

This paper introduces matrix-based deep neural networks where all representations—input, hidden, and output—are matrices rather than vectors, using a novel matrix-to-matrix transformation $ Y = \sigma(U^T X V + B) $ that reduces parameters and preserves structural information. The approach enables more compact models with improved performance in tasks like sequence modeling, EEG classification, and graph learning, especially through multi-attention mechanisms and efficient graph convolutions.

ABSTRACT

We present a new distributed representation in deep neural nets wherein the information is represented in native form as a matrix. This differs from current neural architectures that rely on vector representations. We consider matrices as central to the architecture and they compose the input, hidden and output layers. The model representation is more compact and elegant -- the number of parameters grows only with the largest dimension of the incoming layer rather than the number of hidden units. We derive several new deep networks: (i) feed-forward nets that map an input matrix into an output matrix, (ii) recurrent nets which map a sequence of input matrices into a sequence of output matrices. We also reinterpret existing models for (iii) memory-augmented networks and (iv) graphs using matrix notations. For graphs we demonstrate how the new notations lead to simple but effective extensions with multiple attentions. Extensive experiments on handwritten digits recognition, face reconstruction, sequence to sequence learning, EEG classification, and graph-based node classification demonstrate the efficacy and compactness of the matrix architectures.

Motivation & Objective

• To address the limitations of vector-based representations in deep learning, which lose structural information and require large parameter counts for two-way or associative data.
• To develop a unified framework where all neural network components—input, hidden states, and output—are naturally represented as matrices.
• To demonstrate that matrix representations enable more efficient parameterization and better generalization, especially in tasks with inherent matrix structure such as EEG, graphs, and sequences.
• To show how matrix networks naturally support multi-attention mechanisms and can reinterpret existing models like memory-augmented networks and graph convolutions.

Proposed method

• Proposes a core transformation $ Y = \sigma(U^T X V + B) $, where input $ X $, output $ Y $, and parameters $ U, V, B $ are matrices, enabling structured, parameter-efficient learning.
• Uses row- and column-mapping matrices $ U $ and $ V $ to enable soft attention over rows and columns of the input matrix, supporting multi-head attention mechanisms.
• Derives matrix-based feed-forward and recurrent networks by replacing vector operations with matrix operations, preserving data structure throughout the network.
• Reinterprets memory-augmented networks and graph neural networks as special cases of matrix networks, showing that existing models like GCNs can be expressed via the proposed mat2mat layer.
• Introduces a parameter-efficient graph convolution mechanism by expressing spectral GCNs in matrix form, enabling scalable and interpretable message passing.
• Employs a renormalization trick in graph convolutions to stabilize training, similar to Kipf & Welling (2016), but within a matrix representation framework.

Experimental results

Research questions

• RQ1 Can matrix representations in deep networks preserve structural information better than vectorized representations in two-way or bipartite data?
• RQ2 How does the proposed matrix-to-matrix transformation $ Y = \sigma(U^T X V + B) $ compare to standard vector-based fully connected layers in terms of parameter efficiency and performance?
• RQ3 Can matrix recurrent networks outperform vector-based RNNs in sequence modeling and temporal data tasks like EEG classification with fewer parameters?
• RQ4 How can matrix representations naturally support multi-head attention mechanisms in sequence and graph tasks?
• RQ5 Can existing graph neural networks be systematically reinterpreted and extended using the proposed matrix network framework?

Key findings

• Matrix feed-forward networks achieve better reconstruction performance on MNIST and face data with fewer parameters, demonstrating structural regularization not present in vector-based models.
• Matrix recurrent networks outperform standard RNNs in sequence-to-sequence learning and EEG signal classification, achieving higher accuracy with significantly reduced parameter counts.
• The matrix-based graph model improves node classification on citation networks (e.g., Cora, PubMed), with multi-attention mechanisms further boosting performance over single-head attention.
• The proposed matrix representation allows for a more compact and interpretable formulation of graph convolutions, closely aligning with spectral GCN formulations while enabling efficient implementation.
• The parameter count in matrix networks grows only with the largest dimension of the input matrix, not with the number of hidden units, leading to substantial efficiency gains in high-dimensional settings.
• Experiments confirm that the matrix representation enables effective attention mechanisms and memory access, particularly in tasks involving sequential or relational data.