[Paper Review] Gated XNOR Networks: Deep Neural Networks with Ternary Weights and Activations under a Unified Discretization Framework.

Lei Deng, Peng Jiao+3 more|arXiv (Cornell University)|May 25, 2017

Advanced Memory and Neural Computing17 citations

TL;DR

This paper proposes a unified framework for deep neural networks with ternary weights and activations, enabling efficient backpropagation via multistep discretization and derivative approximation, while eliminating full-precision hidden weights through discrete state transition. The method yields gated XNOR networks that support event-driven hardware design and achieve state-of-the-art performance with flexible sparsity and discrete state control.

ABSTRACT

There is a pressing need to build an architecture that could subsume these networks undera unified framework that achieves both higher performance and less overhead. To this end, two fundamental issues are yet to be addressed. The first one is how to implement the back propagation when neuronal activations are discrete. The second one is how to remove the full-precision hidden weights in the training phase to break the bottlenecks of memory/computation consumption. To address the first issue, we present a multistep neuronal activation discretization method and a derivative approximation technique that enable the implementing the back propagation algorithm on discrete DNNs. While for the second issue, we propose a discrete state transition (DST) methodology to constrain the weights in a discrete space without saving the hidden weights. In this way, we build a unified framework that subsumes the binary or ternary networks as its special cases.More particularly, we find that when both the weights and activations become ternary values, the DNNs can be reduced to gated XNOR networks (or sparse binary networks) since only the event of non-zero weight and non-zero activation enables the control gate to start the XNOR logic operations in the original binary networks. This promises the event-driven hardware design for efficient mobile intelligence. We achieve advanced performance compared with state-of-the-art algorithms. Furthermore,the computational sparsity and the number of states in the discrete space can be flexibly modified to make it suitable for various hardware platforms.

Motivation & Objective

• Address the challenge of enabling backpropagation in discrete neural networks where activations are quantized to discrete values.
• Eliminate the need to store full-precision hidden weights during training to reduce memory and computational overhead.
• Unify binary and ternary networks under a single framework to support flexible hardware mapping.
• Enable event-driven, sparse computation through gated XNOR logic operations in ternary networks.
• Design a system adaptable to diverse hardware platforms by tuning computational sparsity and discrete state count.

Proposed method

• Propose a multistep neuronal activation discretization method to map continuous activations into discrete states while preserving gradient flow.
• Introduce a derivative approximation technique to enable backpropagation through discrete activation functions.
• Develop a discrete state transition (DST) methodology that constrains network weights to discrete values without storing full-precision weights during training.
• Formulate the network such that only non-zero weights and non-zero activations trigger XNOR logic operations, enabling sparse, event-driven inference.
• Integrate ternary weights and activations into a unified framework that generalizes binary and ternary networks.
• Allow flexible adjustment of sparsity and discrete state space size to suit various hardware constraints.

Experimental results

Research questions

• RQ1 How can backpropagation be effectively implemented in deep neural networks with discrete (ternary) activations?
• RQ2 What mechanism can eliminate the need for full-precision hidden weights during training without compromising learning?
• RQ3 Can a unified framework be established to subsume both binary and ternary networks under a single architecture?
• RQ4 How does the proposed method enable efficient, event-driven hardware design for mobile inference?
• RQ5 To what extent can computational sparsity and discrete state space be tuned for hardware-specific optimization?

Key findings

• The proposed method successfully enables backpropagation in discrete networks through multistep discretization and derivative approximation.
• The discrete state transition (DST) technique eliminates the need to store full-precision hidden weights, reducing memory and computation overhead.
• When both weights and activations are ternary, the network reduces to a gated XNOR network, enabling sparse, event-driven computation.
• The framework achieves advanced performance compared to state-of-the-art algorithms in the evaluated benchmarks.
• Computational sparsity and the number of discrete states can be flexibly adjusted to match the constraints of different hardware platforms.
• The unified framework generalizes binary and ternary networks, demonstrating compatibility and scalability across network types.