[Paper Review] Training Neural Networks is NP-Hard in Fixed Dimension

Vincent Froese, Christoph Hertrich|arXiv (Cornell University)|Mar 29, 2023

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper proves that training two-layer ReLU and linear threshold neural networks is NP-hard even in fixed dimension d=2, resolving long-standing open questions. It establishes W[1]-hardness for four ReLUs with zero training error and shows fixed-parameter tractability for convex maps when parameterized by dimension and width.

ABSTRACT

We study the parameterized complexity of training two-layer neural networks with respect to the dimension of the input data and the number of hidden neurons, considering ReLU and linear threshold activation functions. Albeit the computational complexity of these problems has been studied numerous times in recent years, several questions are still open. We answer questions by Arora et al. [ICLR '18] and Khalife and Basu [IPCO '22] showing that both problems are NP-hard for two dimensions, which excludes any polynomial-time algorithm for constant dimension. We also answer a question by Froese et al. [JAIR '22] proving W[1]-hardness for four ReLUs (or two linear threshold neurons) with zero training error. Finally, in the ReLU case, we show fixed-parameter tractability for the combined parameter number of dimensions and number of ReLUs if the network is assumed to compute a convex map. Our results settle the complexity status regarding these parameters almost completely.

Motivation & Objective

• To resolve open questions on the parameterized complexity of training two-layer ReLU and linear threshold neural networks.
• To determine whether training is fixed-parameter tractable (FPT) when parameterized by input dimension d or number of neurons k.
• To close gaps in the computational complexity landscape of neural network training, particularly in low-dimensional settings.
• To clarify the boundary between tractable and intractable cases for exact training with zero error.
• To provide a comprehensive complexity classification for two-layer networks under various parameterizations.

Proposed method

• Reduces from the NP-complete problem of subset sum to prove NP-hardness for d=2 with ReLU activations.
• Uses linear programming to compute lower bounds on the sum of ReLU outputs, enabling forced point detection in the search space.
• Employs a recursive search tree algorithm that dynamically enforces constraints based on forced points to bound recursion depth.
• Analyzes the dimension of the feasible polyhedron defined by weight and bias constraints to bound recursion depth to O(kd).
• Applies the Exponential Time Hypothesis (ETH) to rule out sub-exponential algorithms for k=4 ReLUs with zero error.
• Extends results to linear threshold neurons using analogous reductions and constraint analysis.

Experimental results

Research questions

• RQ1 Is 2L-ReLU-NN-Train(L) in XP when parameterized by input dimension d, i.e., solvable in (nk)^f(d) poly(L) time?
• RQ2 Is 2L-ReLU-NN-Train(L) fixed-parameter tractable with respect to k for any constant d, i.e., solvable in n^f(d) g(k,d) poly(L) time?
• RQ3 Is the problem with zero training error and k=4 ReLUs W[1]-hard when parameterized by d?
• RQ4 Can the search tree approach be extended to networks with mixed positive and negative coefficients (aj ∈ {−1,1})?
• RQ5 Is the case with d=2 and k∈{2,3} in FPT when parameterized by k for ReLU networks?

Key findings

• Training two-layer ReLU networks is NP-hard even when the input dimension d is fixed at 2, resolving Question 1 from Arora et al. [ICLR ’18].
• The problem is W[1]-hard with respect to d when k=4 ReLUs and training error is zero, answering an open question by Froese et al. [JAIR ’22].
• For ReLU networks computing convex maps, the problem is fixed-parameter tractable when parameterized by both d and k, with a running time of 2^{O(k^2 d)} poly(k, L).
• The same hardness results extend to linear threshold networks, showing NP-hardness and W[1]-hardness under similar parameterizations.
• The recursive search algorithm with forced point detection ensures bounded recursion depth of at most k(d+1)+1, leading to a 2^{O(k^2 d)} time algorithm.
• The approach fails for mixed-sign coefficients (aj ∈ {−1,1}) due to inability to detect forced points, indicating inherent limitations in the search tree method for general cases.