[Paper Review] Nonparametric Bayesian Deep Networks with Local Competition

Konstantinos P. Panousis, Sotirios Chatzis+1 more|arXiv (Cornell University)|May 19, 2018

Gaussian Processes and Bayesian Inference29 references3 citations

TL;DR

This paper proposes SB-LWTA networks, a nonparametric Bayesian deep learning framework that uses local winner-take-all (LWTA) nonlinearities and stick-breaking priors to infer minimal network complexity and optimal floating-point precision during training. By modeling component utility via discrete latent variables and performing Bayesian inference, the method achieves state-of-the-art accuracy with significantly reduced computational footprint and prediction time, outperforming existing approaches on MNIST and LeNet-5-Caffe benchmarks.

ABSTRACT

The aim of this work is to enable inference of deep networks that retain high accuracy for the least possible model complexity, with the latter deduced from the data during inference. To this end, we revisit deep networks that comprise competing linear units, as opposed to nonlinear units that do not entail any form of (local) competition. In this context, our main technical innovation consists in an inferential setup that leverages solid arguments from Bayesian nonparametrics. We infer both the needed set of connections or locally competing sets of units, as well as the required floating-point precision for storing the network parameters. Specifically, we introduce auxiliary discrete latent variables representing which initial network components are actually needed for modeling the data at hand, and perform Bayesian inference over them by imposing appropriate stick-breaking priors. As we experimentally show using benchmark datasets, our approach yields networks with less computational footprint than the state-of-the-art, and with no compromises in predictive accuracy.

Motivation & Objective

• To address overparameterization and high computational costs in deep neural networks by enabling data-driven model complexity inference.
• To reduce model redundancy and improve scalability to resource-constrained devices through automatic network pruning and precision compression.
• To develop a principled Bayesian framework that jointly infers network structure and parameter precision using nonparametric priors.
• To leverage local winner-take-all (LWTA) mechanisms for biologically plausible, sparse, and discriminative representations.
• To outperform existing regularization, distillation, and pruning methods in both predictive accuracy and computational efficiency.

Proposed method

• Introduces a deep network architecture based on local winner-take-all (LWTA) units, where only one unit per block is active via lateral inhibition.
• Employs auxiliary discrete latent variables to represent which network components (units or connections) are actually needed for modeling the data.
• Applies stick-breaking priors over these latent variables to enable nonparametric Bayesian inference of component utility and model complexity.
• Uses stochastic gradient variational Bayes (SGVB) for efficient training and posterior inference over network components and weight precision.
• Infers optimal floating-point precision by analyzing the variance of the inferred weight posterior, enabling compression without accuracy loss.
• Derives the SB-LWTA model as a principled, data-driven alternative to heuristic distillation and regularization techniques.

Experimental results

Research questions

• RQ1 Can a Bayesian nonparametric framework effectively infer the minimal network complexity required for a given dataset?
• RQ2 Does the use of local competition (LWTA) in deep networks lead to more efficient and accurate models compared to standard nonlinearities?
• RQ3 Can stick-breaking priors be effectively adapted to infer both network structure and parameter precision in deep networks?
• RQ4 To what extent does the proposed method reduce computational footprint while maintaining or improving predictive accuracy?
• RQ5 How do winner selection patterns in LWTA blocks reflect discriminative, generalizable features across different classes?

Key findings

• SB-LWTA networks achieve the highest predictive accuracy among all compared methods on the MNIST dataset, outperforming even the original LeNet-5-Caffe architecture.
• The method reduces the number of feature maps to the lowest among all baselines, indicating superior structural compression.
• Prediction time is reduced by one order of magnitude compared to the original network, demonstrating significant inference efficiency gains.
• The average training time per epoch is only 10% higher than the original network, indicating minimal training overhead.
• Winner selection probabilities vary meaningfully across MNIST digits, showing that the model learns discriminative, class-specific patterns.
• Overlaps in winning units between digit pairs are consistently below 50%, confirming that the winner-take-all mechanism encodes distinct, generalizable representations.