[Paper Review] Tensor Normalization and Full Distribution Training

Wolfgang Fuhl|arXiv (Cornell University)|Sep 6, 2021

Model Reduction and Neural NetworksPhysics and Astronomy102 references31 citations

TL;DR

Introduces pixel-wise tensor normalization after ReLU and a full distribution training approach that reformulates multi-class tasks into multi-label problems, improving accuracy and robustness across several datasets and architectures.

ABSTRACT

In this work, we introduce pixel wise tensor normalization, which is inserted after rectifier linear units and, together with batch normalization, provides a significant improvement in the accuracy of modern deep neural networks. In addition, this work deals with the robustness of networks. We show that the factorized superposition of images from the training set and the reformulation of the multi class problem into a multi-label problem yields significantly more robust networks. The reformulation and the adjustment of the multi class log loss also improves the results compared to the overlay with only one class as label. https://atreus.informatik.uni-tuebingen.de/seafile/d/8e2ab8c3fdd444e1a135/?p=%2FTNandFDT&mode=list

Motivation & Objective

• Improve generalization by introducing a parameter-free pixel-wise tensor normalization layer applied after activations.
• Enhance model robustness by factorized superposition of training images and a multi-label loss formulation.
• Reformulate multi-class classification into a multi-label setting with an adapted softmax loss.
• Evaluate the proposed methods across multiple datasets and network architectures to demonstrate robustness and accuracy gains.
• Assess robustness under adversarial-like perturbations using PGD and analyze implications.

Proposed method

• Propose a 2D tensor normalization (TN) that computes a per-(x,y) mean across the depth Z and subtracts it from the activation tensor.
• Normalization is performed online after ReLU with no extra memory burden and gradients simply backpropagate through the subtraction.
• Introduce Full Distribution Training (FDT) by composing multiple images into a single multi-label example using a harmonic-series-based weighting scheme and random selection of classes.
• Adapt the cross-entropy loss to a multi-label setting (Algorithm 4) by applying softmax to predictions and using ground-truth distribution GT to compute gradients for all relevant labels.
• Provide algorithms for constructing multi-label inputs (Eq. 2–4) and for multi-label loss computation (Alg. 4) to justify robust learning.
• Evaluate robustness via PGD attacks and compare against baseline, overlaid (OV), and TN+FDT configurations.

Experimental results

Research questions

• RQ1 Does pixel-wise tensor normalization after ReLU improve generalization across common vision datasets?
• RQ2 Does full distribution training (FDT) increase robustness to adversarial-like perturbations without extra training time or parameters?
• RQ3 How does reformulating multi-class classification into a multi-label problem influence accuracy and robustness, compared to standard cross-entropy losses?
• RQ4 What is the combined impact of TN and FDT across different architectures and datasets under PGD attacks?

Key findings

• Tensor normalization (TN) outperforms other combinations in several datasets in terms of robustness and accuracy.
• Full distribution training (FDT) increases robustness to PGD attacks, with notable gains when combined with TN.
• On SVHN, FDT shows less robustness improvement due to intrinsic multi-label gradients already present in the data, illustrating dataset-dependent effects.
• The TN and FDT combination yields strong performance improvements across CIFAR-10, CIFAR-100, Fashion-MNIST, and SVHN with small ResNet-34 and larger models.
• Results indicate that no additional parameters or training time are required for TN and FDT to be effective, though TN adds some compute for the normalization step.