[Paper Review] Limits on representing Boolean functions by linear combinations of simple functions: thresholds, ReLUs, and low-degree polynomials

Ryan Williams|arXiv (Cornell University)|Feb 25, 2018

Machine Learning and Algorithms4 citations

TL;DR

This paper establishes new super-polynomial lower bounds on the sparsity of linear combinations of simple functions—specifically linear threshold functions, ReLU gates, and low-degree polynomials over finite fields—for representing Boolean functions. By leveraging a novel 'easy witness' lemma and advanced algebraic techniques, it proves that certain functions in NTIME[n^α(n)] and E^NP require exponentially or super-polynomially many such components, advancing circuit complexity and neural network expressivity limits.

ABSTRACT

We consider the problem of representing Boolean functions exactly by "sparse" linear combinations (over $\\mathbb{R}$) of functions from some "simple" class ${\\cal C}$. In particular, given ${\\cal C}$ we are interested in finding low-complexity functions lacking sparse representations. When ${\\cal C}$ is the set of PARITY functions or the set of conjunctions, this sort of problem has a well-understood answer, the problem becomes interesting when ${\\cal C}$ is "overcomplete" and the set of functions is not linearly independent. We focus on the cases where ${\\cal C}$ is the set of linear threshold functions, the set of rectified linear units (ReLUs), and the set of low-degree polynomials over a finite field, all of which are well-studied in different contexts. We provide generic tools for proving lower bounds on representations of this kind. Applying these, we give several new lower bounds for "semi-explicit" Boolean functions. For example, we show there are functions in nondeterministic quasi-polynomial time that require super-polynomial size: $\\bullet$ Depth-two neural networks with sign activation function, a special case of depth-two threshold circuit lower bounds. $\\bullet$ Depth-two neural networks with ReLU activation function. $\\bullet$ $\\mathbb{R}$-linear combinations of $O(1)$-degree $\\mathbb{F}_p$-polynomials, for every prime $p$ (related to problems regarding Higher-Order "Uncertainty Principles"). We also obtain a function in $E^{NP}$ requiring $2^{\\Omega(n)}$ linear combinations. $\\bullet$ $\\mathbb{R}$-linear combinations of $ACC \\circ THR$ circuits of polynomial size (further generalizing the recent lower bounds of Murray and the author). (The above is a shortened abstract. For the full abstract, see the paper.)

Motivation & Objective

• To establish strong lower bounds on the sparsity of linear combinations of simple functions (thresholds, ReLUs, low-degree polynomials) needed to represent Boolean functions.
• To resolve longstanding open problems regarding the expressivity limits of depth-two neural networks and threshold circuits.
• To extend correlation-based lower bound techniques to settings with arbitrary real weights, overcoming prior limitations in weight size.
• To provide new tools for proving lower bounds in circuit complexity, particularly for functions in NP and E^NP.
• To investigate the connection between representation complexity and higher-order uncertainty principles in Fourier analysis.

Proposed method

• Leverages a new 'easy witness' lemma from Cody Murray and the author to identify hard instances for representation.
• Applies algebraic techniques involving polynomial interpolation and modulus amplification over finite fields to simulate Boolean function counting.
• Uses the polynomial $ Q $, constructed as a sum of $ P_{\delta n}(1 - q(x)^{p-1}) $, to count roots of a given polynomial $ q $ over $ \mathbb{F}_p $.
• Employs fast polynomial evaluation algorithms (Theorem 2.4) to compute $ Q $ over all $ 2^{n - \delta n} $ assignments in $ 2^{n - \varepsilon n} \cdot \text{poly}(n) $ time.
• Reduces the Sum-Product problem to evaluating $ Q $, enabling sub-exponential time algorithms for fixed-degree polynomials.
• Translates these algorithms into lower bounds by showing that if a function had a sparse representation, it would contradict the time complexity of the Sum-Product problem.

Experimental results

Research questions

• RQ1 Can functions in NTIME[n^α(n)] be shown to require super-polynomially many linear threshold functions for exact representation?
• RQ2 Do ReLU-based depth-two neural networks require super-polynomial size to represent certain functions in NTIME[n^α(n)]?
• RQ3 Can lower bounds be established for representing AND or other NP functions as linear combinations of low-degree polynomials over finite fields?
• RQ4 Can the techniques of this paper resolve the Constant Degree Hypothesis for degree-2 polynomials?
• RQ5 What is the minimal size of linear combinations of quadratic $ \mathbb{F}_2 $-polynomials needed to represent the AND function?

Key findings

• Functions in NTIME[n^α(n)] for unbounded α(n) such that n^α(n) is time-constructible require super-polynomially many linear threshold functions for exact representation.
• Functions in NTIME[n^α(n)] require super-polynomially many ReLU gates for exact representation, even with arbitrary real weights.
• Functions in NTIME[n^poly(log n)] require super-polynomially many ACC∘THR circuits of polynomial size for exact representation.
• A function in E^NP requires 2^Ω(n) size linear combinations of threshold functions, establishing an exponential lower bound.
• For every fixed degree d and prime p, there exists ε > 0 such that the Sum-Product problem for MODp∘ANDd circuits can be solved in 2^{n - εn} · poly(n) time, implying sub-exponential algorithms.
• The paper provides the first fixed-polynomial lower bounds for NP functions in the representation classes of thresholds, ReLUs, and low-degree polynomials over finite fields.