[Paper Review] Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

Xiao Zhang, Dongrui Wu|arXiv (Cornell University)|Jan 4, 2020

Neural Networks and Applications38 references21 citations

TL;DR

This paper introduces a novel empirical framework to analyze the geometric and structural properties of linear regions in deep neural networks (DNNs), using polyhedral computation to study how optimization techniques like batch normalization (BN) and dropout shape these regions. It reveals that even with similar accuracy, BN increases region count and uniformity, while dropout concentrates regions near decision boundaries with shared normal directions, highlighting distinct implicit inductive biases.

ABSTRACT

A deep neural network (DNN) with piecewise linear activations can partition the input space into numerous small linear regions, where different linear functions are fitted. It is believed that the number of these regions represents the expressivity of the DNN. This paper provides a novel and meticulous perspective to look into DNNs: Instead of just counting the number of the linear regions, we study their local properties, such as the inspheres, the directions of the corresponding hyperplanes, the decision boundaries, and the relevance of the surrounding regions. We empirically observed that different optimization techniques lead to completely different linear regions, even though they result in similar classification accuracies. We hope our study can inspire the design of novel optimization techniques, and help discover and analyze the behaviors of DNNs.

Motivation & Objective

• To investigate how different optimization techniques (e.g., BN, dropout) influence the geometric and topological properties of linear regions in DNNs, beyond just classification accuracy.
• To develop and apply polyhedral computation tools to analyze the local structure of linear regions, including hyperplane directions, inspheres, and region relevance.
• To uncover implicit inductive biases introduced by optimization methods by examining their effects on the partitioning of the input space into linear regions.
• To explore the relationship between linear region properties and DNN behaviors such as adversarial robustness and gradient dynamics.

Proposed method

• The study uses polyhedral computation to identify and characterize linear regions in ReLU-based DNNs by analyzing activation patterns across the input space.
• For each test point, the method samples L2-normalized directions within the null space of the active neuron Jacobian to explore transitions between regions.
• It computes the number of unique linear regions along these directions using a small perturbation (ε = 0.2), measuring region count and spatial distribution.
• The relevance of neighboring regions is quantified via cosine similarity of gradient vectors (decision directions) between adjacent regions.
• The analysis is applied to models trained with different optimization techniques (vanilla, BN, dropout) on MNIST, using 1000 test points and 100 sampled directions per point.
• The method enables comparison of hyperplane normal directions, insphere sizes, and clustering patterns of regions, particularly near decision boundaries.

Experimental results

Research questions

• RQ1 How do different optimization techniques such as batch normalization and dropout alter the number, size, and spatial distribution of linear regions in DNNs?
• RQ2 What is the relationship between the normal directions of hyperplanes defining linear regions and the optimization method used?
• RQ3 How does the relevance of neighboring linear regions—measured via gradient similarity—affect DNN behavior such as adversarial robustness or optimization stability?
• RQ4 To what extent do linear region properties correlate with model generalization or robustness, independent of accuracy?

Key findings

• Batch normalization increases the number of linear regions and promotes more uniform region sizes compared to vanilla networks.
• Dropout leads to a higher concentration of linear regions near decision boundaries, with many sharing similar normal directions.
• The gradient direction in a linear region is highly correlated with those of its immediate neighbors, indicating strong local linearity, though this relevance decreases with depth.
• Batch normalization slightly reduces the gradient relevance between neighboring regions, potentially mitigating the shattered gradients problem.
• Models trained with smaller learning rates partition the input space into more linear regions, suggesting a link between training dynamics and region complexity.
• Despite similar test accuracy, BN and dropout produce fundamentally different linear region structures, implying distinct inductive biases in model behavior.