[Paper Review] Learning Parities with Neural Networks

Amit Daniely, Eran Malach|arXiv (Cornell University)|Feb 18, 2020

Domain Adaptation and Few-Shot Learning26 references11 citations

TL;DR

This paper demonstrates that gradient-descent-trained two-layer neural networks can efficiently learn sparse parities—nonlinear functions that are provably intractable for linear models—under specific data distributions. The key contribution is an exponential separation: neural networks achieve low error, while linear methods require exponentially large representations or suffer from exponentially large norms.

ABSTRACT

In recent years we see a rapidly growing line of research which shows learnability of various models via common neural network algorithms. Yet, besides a very few outliers, these results show learnability of models that can be learned using linear methods. Namely, such results show that learning neural-networks with gradient-descent is competitive with learning a linear classifier on top of a data-independent representation of the examples. This leaves much to be desired, as neural networks are far more successful than linear methods. Furthermore, on the more conceptual level, linear models don't seem to capture the "deepness" of deep networks. In this paper we make a step towards showing leanability of models that are inherently non-linear. We show that under certain distributions, sparse parities are learnable via gradient decent on depth-two network. On the other hand, under the same distributions, these parities cannot be learned efficiently by linear methods.

Motivation & Objective

• To demonstrate that neural networks trained with gradient descent can learn inherently nonlinear functions, such as sparse parities, beyond the reach of linear methods.
• To establish a theoretical separation between the learning power of neural networks and linear models, including kernel methods and random features.
• To show that certain distributions over input data allow efficient learning of parities via neural networks, while rendering such learning infeasible for linear classifiers on fixed representations.
• To challenge the prevailing view that neural network generalization is reducible to linear models via random features or kernel approximations.
• To open new theoretical pathways for understanding the representational power of deep learning beyond linearization and kernel-based approximations.

Proposed method

• Define a family of distributions 𝒟_A over {±1/√n}^n × {±1} where labels are parities over k-bit subsets of inputs.
• Use a two-layer ReLU network with trainable weights and biases, trained via stochastic gradient descent on the hinge loss.
• Analyze the generalization error of the network under the specified data distributions, showing convergence to low error.
• Prove that no linear classifier on any fixed N-dimensional embedding can achieve low error unless N grows exponentially or the classifier norm is exponentially large.
• Use the neural tangent kernel (NTK) regime as a baseline, comparing it to standard ReLU networks to isolate the role of nonlinearity.
• Conduct experiments on a modified MNIST dataset (MNIST-parity), where inputs are strips of digits and the label is the parity of their sum, to validate theoretical claims empirically.

Experimental results

Research questions

• RQ1 Can gradient-descent-trained neural networks learn sparse parities that are provably hard for linear models?
• RQ2 Is there a distribution family for which neural networks achieve low generalization error while linear models require exponentially large representations?
• RQ3 Does the nonlinearity of ReLU networks provide a fundamental advantage over linear methods in learning non-linear target functions?
• RQ4 Can we establish an exponential separation between the sample or representation complexity of neural networks and linear models for a specific learning task?
• RQ5 To what extent does the structure of the data distribution enable or hinder the learning of non-linear functions by neural networks?

Key findings

• Neural networks trained with gradient descent achieve test accuracy of nearly 80% on the MNIST-parity task with k=3, where the label is the parity of the sum of three digits.
• In contrast, linear models—including Gaussian features, ReLU features, and NTK-regime networks—achieve performance barely above random chance (50%) on the same k=3 task.
• Theoretical analysis shows that any linear classifier on a fixed N-dimensional embedding requires N to grow exponentially in k to achieve low error on the same distribution family.
• Equivalently, if the representation size is polynomial in k, then any linear classifier must have exponentially large norm to approximate the parity function.
• The exponential separation is established under a specific distribution family where the parity bits are randomly set with equal probability, while the remaining bits are fixed or uniformly distributed.
• The results demonstrate that standard ReLU networks with gradient descent can learn non-linear functions that are fundamentally beyond the reach of linear models, even with optimal fixed representations.