[Paper Review] Generalizing MLPs With Dropouts, Batch Normalization, and Skip Connections

Taewoon Kim|arXiv (Cornell University)|Aug 18, 2021

Neural Networks and Applications25 references13 citations

TL;DR

This paper proposes a generalized MLP architecture enhanced with Independent-Component (IC) layers for input whitening, batch normalization, dropout, and skip connections. By whitening inputs before each fully connected layer using a learnable IC layer with dropout, the model achieves faster convergence, better generalization, and improved uncertainty estimation via Monte Carlo dropout, outperforming standard MLPs on age and gender classification benchmarks with pretrained initialization.

ABSTRACT

A multilayer perceptron (MLP) is typically made of multiple fully connected layers with nonlinear activation functions. There have been several approaches to make them better (e.g. faster convergence, better convergence limit, etc.). But the researches lack structured ways to test them. We test different MLP architectures by carrying out the experiments on the age and gender datasets. We empirically show that by whitening inputs before every linear layer and adding skip connections, our proposed MLP architecture can result in better performance. Since the whitening process includes dropouts, it can also be used to approximate Bayesian inference. We have open sourced our code, and released models and docker images at https://github.com/tae898/age-gender/

Motivation & Objective

• To address the lack of systematic methods for generalizing multilayer perceptrons (MLPs) beyond standard architectures.
• To investigate whether input whitening via an IC layer with dropout improves training stability, convergence speed, and final performance.
• To evaluate the impact of skip connections and pretrained weight initialization on MLP generalization.
• To explore uncertainty estimation in MLPs using Monte Carlo dropout within the proposed architecture.

Proposed method

• Introduce an Independent-Component (IC) layer that whitens inputs before each fully connected layer using batch normalization and dropout.
• Use dropout within the IC layer to decorrelate features and reduce mutual information between neurons by a factor of p².
• Integrate skip connections after each nonlinear transformation to stabilize training and improve gradient flow.
• Apply ReLU activation after each IC layer and use stochastic weight scaling during inference to approximate Bayesian inference.
• Train models with and without IC layers, skip connections, and dropout, using both random and pretrained weight initialization.
• Estimate model uncertainty via Monte Carlo dropout by performing 512 forward passes and computing entropy of predictions.

Experimental results

Research questions

• RQ1 Does incorporating an IC layer with dropout before each fully connected layer improve the convergence limit and generalization of MLPs?
• RQ2 How do skip connections affect the training dynamics and final performance of MLPs with IC layers?
• RQ3 Can the proposed MLP architecture with dropout enable effective uncertainty estimation through Monte Carlo dropout?
• RQ4 Does using pretrained weights in combination with IC layers and skip connections lead to better transfer performance on downstream tasks?

Key findings

• The proposed MLP with IC layers and skip connections achieved the lowest validation loss (0.0444) and highest accuracy (0.9918) on the 8-class age classification task with pretrained initialization.
• Models with both IC layers and skip connections showed the most consistent improvement in validation loss across all tasks, especially under pretrained initialization.
• The model with IC layers and dropout enabled effective uncertainty estimation, with the highest entropy values (close to 0.6931) observed for baby and kid faces—rare classes in the Adience dataset.
• The ablation study showed that removing either IC layers or skip connections led to a significant performance drop, with validation loss increasing to 0.1100 and 0.1311, respectively, under random initialization.
• The model with both IC layers and skip connections achieved a test loss of 0.0455 and accuracy of 0.9913 on the 8-class age classification task with pretrained weights, outperforming all ablated variants.
• The authors observed that the IC layer with dropout effectively reduced feature correlation and mutual information, enabling better generalization and stable training.