[Paper Review] DNN Verification, Reachability, and the Exponential Function Problem

Omri Isac, Yoni Zohar+2 more|arXiv (Cornell University)|Jan 1, 2023

Adversarial Robustness in Machine Learning3 citations

TL;DR

This paper establishes that verifying deep neural networks (DNNs) with smooth and piecewise-smooth activation functions—such as Sigmoid and tanh—is equivalent to solving Tarski’s exponential function problem, a long-standing open problem in model theory. It further proves that DNN verification with quantifier-free linear arithmetic specifications reduces to the NP-complete DNN reachability problem under ϵ-error tolerance, revealing a fundamental complexity divide between piecewise-linear and smooth DNNs.

ABSTRACT

Deep neural networks (DNNs) are increasingly being deployed to perform safety-critical tasks. The opacity of DNNs, which prevents humans from reasoning about them, presents new safety and security challenges. To address these challenges, the verification community has begun developing techniques for rigorously analyzing DNNs, with numerous verification algorithms proposed in recent years. While a significant amount of work has gone into developing these verification algorithms, little work has been devoted to rigorously studying the computability and complexity of the underlying theoretical problems. Here, we seek to contribute to the bridging of this gap. We focus on two kinds of DNNs: those that employ piecewise-linear activation functions (e.g., ReLU), and those that employ piecewise-smooth activation functions (e.g., Sigmoids). We prove the two following theorems: 1) The decidability of verifying DNNs with a particular set of piecewise-smooth activation functions is equivalent to a well-known, open problem formulated by Tarski; and 2) The DNN verification problem for any quantifier-free linear arithmetic specification can be reduced to the DNN reachability problem, whose approximation is NP-complete. These results answer two fundamental questions about the computability and complexity of DNN verification, and the ways it is affected by the network's activation functions and error tolerance; and could help guide future efforts in developing DNN verification tools.

Motivation & Objective

• To investigate the computability and complexity of DNN verification, particularly for networks with non-piecewise-linear activation functions.
• To determine whether verification of DNNs with smooth and piecewise-smooth activations is decidable.
• To analyze the computational complexity of DNN verification under ϵ-error tolerance and quantifier-free linear arithmetic specifications.
• To establish formal connections between DNN verification, reachability, and known decidable theories.
• To guide future development of verification tools by identifying theoretical boundaries and complexity classes.

Proposed method

• Constructing a formal bijection between DNN verification queries involving smooth and piecewise-smooth activation functions and instances of Tarski’s exponential function problem.
• Proving that the DNN verification problem for such networks is logically equivalent to Tarski’s open problem, implying undecidability unless Tarski’s problem is resolved.
• Reducing any DNN verification query with a quantifier-free linear arithmetic specification to the DNN reachability problem via a constructive transformation.
• Demonstrating that the DNN reachability problem with ϵ-error tolerance is NP-complete, using known complexity results from computational logic.
• Using the Nelson-Oppen method as a theoretical foundation for combining decision procedures, though adapting it to non-disjoint theories due to shared signatures.
• Extending the reachability reduction to multi-network verification tasks, such as DNN equivalence, by constructing auxiliary networks that encode equality constraints.

Experimental results

Research questions

• RQ1 Is the verification problem for DNNs with smooth and piecewise-smooth activation functions decidable?
• RQ2 What is the computational complexity of DNN verification when ϵ-error tolerance is introduced?
• RQ3 Can DNN verification with quantifier-free linear arithmetic specifications be reduced to the DNN reachability problem?
• RQ4 How do the theoretical properties of DNNs with ReLU-like activations differ from those with Sigmoid or tanh in terms of verification complexity?
• RQ5 Can decidable fragments of DNN verification be identified by restricting network architecture or specification language?

Key findings

• The DNN verification problem for networks with smooth and piecewise-smooth activation functions is logically equivalent to Tarski’s exponential function problem, a well-known open problem in model theory.
• Verification of DNNs with quantifier-free linear arithmetic specifications reduces to the DNN reachability problem, which is NP-complete under ϵ-error tolerance.
• This reduction establishes that DNN reachability with ϵ-tolerance is a complete problem for NP, providing a complexity benchmark for verification tools.
• The equivalence between verification and reachability holds even for multi-network queries, such as DNN equivalence, via a construction that encodes equality constraints using additional ReLU neurons.
• The results reveal a fundamental theoretical divide: verification of piecewise-linear DNNs is NP-complete, while verification of smooth DNNs is tied to an unresolved problem in mathematical logic.
• The study suggests that exact verification of smooth DNNs may be inherently harder than for piecewise-linear ones, unless Tarski’s problem is resolved.