[Paper Review] Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network

Lixin Fan|arXiv (Cornell University)|Oct 27, 2017

Neural Networks and Applications18 references15 citations

TL;DR

This paper proposes a generalized hamming network (GHN) that reinterprets deep learning techniques like batch normalization and ReLU through the lens of fuzzy logic using generalized Hamming distance (GHD). By analytically enforcing the correct bias term derived from GHD, the method eliminates the need for batch normalization and shows ReLU is non-essential for simple tasks, achieving state-of-the-art performance with faster convergence and well-controlled behavior across MNIST, CIFAR, and VAE tasks.

ABSTRACT

We revisit fuzzy neural network with a cornerstone notion of generalized hamming distance, which provides a novel and theoretically justified framework to re-interpret many useful neural network techniques in terms of fuzzy logic. In particular, we conjecture and empirically illustrate that, the celebrated batch normalization (BN) technique actually adapts the normalized bias such that it approximates the rightful bias induced by the generalized hamming distance. Once the due bias is enforced analytically, neither the optimization of bias terms nor the sophisticated batch normalization is needed. Also in the light of generalized hamming distance, the popular rectified linear units (ReLU) can be treated as setting a minimal hamming distance threshold between network inputs and weights. This thresholding scheme, on the one hand, can be improved by introducing double thresholding on both extremes of neuron outputs. On the other hand, ReLUs turn out to be non-essential and can be removed from networks trained for simple tasks like MNIST classification. The proposed generalized hamming network (GHN) as such not only lends itself to rigorous analysis and interpretation within the fuzzy logic theory but also demonstrates fast learning speed, well-controlled behaviour and state-of-the-art performances on a variety of learning tasks.

Motivation & Objective

• To reframe deep learning techniques like batch normalization and ReLU using fuzzy logic and generalized Hamming distance.
• To demonstrate that batch normalization is unnecessary when the correct bias is analytically enforced via GHD.
• To show that ReLU is non-essential for simple tasks like MNIST classification.
• To develop a generalized hamming network (GHN) that enables rigorous theoretical analysis and superior training dynamics.
• To provide a unified, interpretable framework for deep learning by grounding it in fuzzy logic principles.

Proposed method

• Introduces generalized Hamming distance (GHD) as h(x, w) = x + w - 2xw for real-valued inputs and weights, linking it to neuron output computation.
• Defines a generalized hamming network (GHN) where neuron outputs satisfy the bias condition derived from GHD, eliminating the need for learned bias optimization.
• Proposes rectifying neuron outputs using the analytical bias term h(x, w), which removes the need for batch normalization by correcting internal covariate shift at the source.
• Introduces a double-thresholding scheme that suppresses fuzzy elements by setting minimal and maximal thresholds on neuron outputs, improving learning efficiency.
• Applies the rectified GHN to various tasks—MNIST, CIFAR, VAE, and sentence classification—without batch normalization or ReLU in some cases.
• Uses direct minimization of reconstruction error in VAEs and classification loss in CNNs, omitting weight regularization due to GHN’s inherent stability.

Experimental results

Research questions

• RQ1 Can batch normalization be theoretically justified and replaced by an analytical bias correction derived from generalized Hamming distance?
• RQ2 Is the ReLU activation function essential for deep learning, or can it be removed without performance degradation in simple tasks?
• RQ3 How does enforcing the correct bias term via GHD affect training speed, convergence, and model performance across diverse learning tasks?
• RQ4 Can a fuzzy logic-based neural network architecture achieve state-of-the-art performance while remaining interpretable and well-controlled?
• RQ5 What is the impact of double-thresholding on neuron outputs in suppressing fuzzy elements and improving learning efficiency?

Key findings

• The rectified GHN achieved a 28% reduction in minimal reconstruction error on VAEs trained on MNIST compared to the baseline, with faster convergence.
• On CIFAR-10, the GHN with optimized double-thresholding reached 89.26% test accuracy, outperforming the baseline without thresholding (84.63%).
• On CIFAR-100, the GHN with double-thresholding achieved 60.05% accuracy, significantly improving over the non-thresholded version (51.71%).
• For MNIST classification, removing ReLU entirely did not degrade performance, demonstrating that ReLU is non-essential for simple tasks.
• The rectified GHN achieved faster learning speed and well-controlled behavior across all evaluated tasks, including sentence classification and VAEs, without batch normalization or weight regularization.
• The proposed framework enables rigorous theoretical analysis within fuzzy logic, providing a new interpretive lens for deep learning.