[Paper Review] ProdSumNet: reducing model parameters in deep neural networks via product-of-sums matrix decompositions

Chai Wah Wu|arXiv (Cornell University)|Sep 6, 2018

Model Reduction and Neural Networks23 references15 citations

TL;DR

ProdSumNet proposes a novel matrix decomposition framework that reduces deep neural network parameters by expressing weight matrices as products of sums of simpler, fixed matrices. By training only the scalar coefficients of these sums, the method achieves high accuracy with drastically fewer parameters—e.g., 98.44% accuracy on MNIST using only 3,554 trainable parameters instead of over 3 million—offering a flexible, trainable parameter tradeoff across architectures.

ABSTRACT

We consider a general framework for reducing the number of trainable model parameters in deep learning networks by decomposing linear operators as a product of sums of simpler linear operators. Recently proposed deep learning architectures such as CNN, KFC, Dilated CNN, etc. are all subsumed in this framework and we illustrate other types of neural network architectures within this framework. We show that good accuracy on MNIST and Fashion MNIST can be obtained using a relatively small number of trainable parameters. In addition, since implementation of the convolutional layer is resource-heavy, we consider an approach in the transform domain that obviates the need for convolutional layers. One of the advantages of this general framework over prior approaches is that the number of trainable parameters is not fixed and can be varied arbitrarily. In particular, we illustrate the tradeoff of varying the number of trainable variables and the corresponding error rate. As an example, by using this decomposition on a reference CNN architecture for MNIST with over 3x10^6 trainable parameters, we are able to obtain an accuracy of 98.44% using only 3554 trainable parameters.

Motivation & Objective

• Reduce the number of trainable parameters in deep neural networks without sacrificing classification accuracy.
• Develop a general framework that unifies existing parameter-efficient architectures like CNNs and KFC as special cases of product-of-sums decompositions.
• Enable flexible, dynamic control over the number of trainable parameters by varying the number of sum terms in the decomposition.
• Demonstrate that model complexity can be significantly reduced while maintaining high performance on image classification benchmarks like MNIST and Fashion MNIST.
• Explore the feasibility of replacing computationally heavy convolutional layers with efficient transform-domain implementations using this decomposition framework.

Proposed method

• Represent a weight matrix $ W $ as a product of sums: $ W = \prod_{j=1}^{p} \sum_{k=1}^{s_j} a_{jk} M_{jk} $, where $ a_{jk} $ are trainable parameters and $ M_{jk} $ are fixed matrices.
• Use linear activation functions $ g_{jk}(x) = x $ to simplify the decomposition, enabling differentiable training of the scalar coefficients $ a_{jk} $.
• Apply the decomposition to fully connected and convolutional layers, replacing standard dense or conv layers with parameter-efficient alternatives.
• Implement the decomposition in the transform domain (e.g., via FFT or other transforms) to avoid expensive convolution operations.
• Use iterative refinement: train in stages with increasing $ s_j $, initializing higher-stage parameters using least-squares approximation of lower-stage results.
• Leverage low-rank or structured matrices (e.g., circulant, Toeplitz) for $ M_{jk} $ to reduce storage and computation costs.

Experimental results

Research questions

• RQ1 Can a product-of-sums matrix decomposition effectively reduce the number of trainable parameters in deep neural networks while maintaining high accuracy?
• RQ2 How does the number of trainable parameters in the decomposition affect the final test error and model performance on image classification tasks?
• RQ3 To what extent can this framework generalize beyond CNNs and KFC, and can it be applied to arbitrary neural network layers?
• RQ4 Is it possible to achieve high accuracy with significantly fewer parameters by using structured, fixed matrices $ M_{jk} $ and learning only the scalar coefficients $ a_{jk} $?
• RQ5 How does the performance of the decomposition vary across different datasets, such as MNIST versus Fashion MNIST, which may differ in intrinsic complexity?

Key findings

• ProdSumNet achieves 98.44% test accuracy on MNIST using only 3,554 trainable parameters, a reduction of over 99.8% compared to a standard CNN with ~3.2 million parameters.
• The method demonstrates a flexible tradeoff between model complexity and accuracy, with performance decreasing gradually as the number of trainable parameters is reduced.
• Fashion MNIST requires more parameters than MNIST to achieve comparable accuracy, indicating higher intrinsic complexity and more degrees of freedom in the data.
• The decomposition framework successfully generalizes known architectures like CNNs and KFC as special cases, showing its broad applicability.
• Replacing convolutional layers with transform-domain implementations using the decomposition reduces computational cost and avoids expensive convolution operations.
• The use of structured matrices (e.g., circulant, low-rank) for $ M_{jk} $ enables efficient storage and computation, making the approach practical for deployment.