[Paper Review] Semialgebraic Representation of Monotone Deep Equilibrium Models and Applications to Certification

Tong Chen, Jean B. Lasserre+2 more|arXiv (Cornell University)|Jun 2, 2021

Adversarial Robustness in Machine Learning35 references4 citations

TL;DR

This paper introduces a semialgebraic representation of ReLU-based monotone deep equilibrium models (monDEQs) that enables robustness certification, Lipschitz constant estimation, and ellipsoidal uncertainty propagation via semidefinite programming (SDP). The proposed SDP-based models outperform prior methods, particularly for $L_2$ robustness, and reveal that monDEQs are significantly more vulnerable to $L_∞$ perturbations than $L_2$ perturbations.

ABSTRACT

Deep equilibrium models are based on implicitly defined functional relations and have shown competitive performance compared with the traditional deep networks. Monotone operator equilibrium networks (monDEQ) retain interesting performance with additional theoretical guaranties. Existing certification tools for classical deep networks cannot directly be applied to monDEQs for which much fewer tools exist. We introduce a semialgebraic representation for ReLU based monDEQs which allows to approximate the corresponding input output relation by semidefinite programming (SDP). We present several applications to network certification and obtain SDP models for the following problems : robustness certification, Lipschitz constant estimation, ellipsoidal uncertainty propagation. We use these models to certify robustness of monDEQs w.r.t. a general $L_q$ norm. Experimental results show that the proposed models outperform existing approaches for monDEQ certification. Furthermore, our investigations suggest that monDEQs are much more robust to $L_2$ perturbations than $L_{\infty}$ perturbations.

Motivation & Objective

• To address the lack of robustness certification tools for monotone deep equilibrium models (monDEQs), which are theoretically sound but under-explored in certification.
• To develop a general framework for certifying robustness, estimating Lipschitz constants, and computing outer-approximations of network outputs using semialgebraic representations.
• To enable efficient certification of monDEQs under general $L_p$ norms, especially $L_2$ and $L_\infty$, via semidefinite programming relaxations.
• To demonstrate that monDEQs exhibit asymmetric robustness, being more sensitive to $L_\infty$ than $L_2$ perturbations.
• To extend algebraic geometry-based certification techniques—previously limited to classical DNNs—to implicitly defined monDEQ architectures.

Proposed method

• Formulate monDEQs with ReLU activations as semialgebraic sets using fixed-point equations and monotonicity constraints.
• Apply Shor's relaxation to transform the semialgebraic representation into a hierarchy of semidefinite programs (SDPs).
• Construct three distinct SDP models: (1) robustness certification under $L_p$ norms, (2) Lipschitz constant estimation, and (3) ellipsoidal outer-approximation of image sets.
• Use the order-1 relaxation of the SDP hierarchy to compute tight upper bounds on Lipschitz constants and validate robustness.
• Leverage the geometry of the outer-approximating ellipsoid to certify robustness by checking whether the ellipsoid lies below the decision boundary in label space.
• Apply the models to MNIST with fully connected monDEQs, comparing results against the state-of-the-art method from [25].

Experimental results

Research questions

• RQ1 Can semialgebraic representations of monDEQs be constructed to enable algebraic certification via semidefinite programming?
• RQ2 How do the proposed SDP-based models compare to existing methods in terms of robustness certification performance for monDEQs?
• RQ3 Is the robustness of monDEQs symmetric across different $L_p$ norms, particularly between $L_2$ and $L_\infty$?
• RQ4 Can the proposed models estimate tight Lipschitz constants for monDEQs, improving upon existing bounds?
• RQ5 Can outer-approximating ellipsoids be effectively used to certify robustness in monDEQs, and how do they compare to direct robustness checks?

Key findings

• The proposed Robustness Model outperforms the method in [25], certifying robustness for more MNIST examples under $L_2$ norm.
• The Robustness Model achieves tighter certification bounds than the state-of-the-art method, even with only order-1 SDP relaxation.
• The Lipschitz Model provides tighter upper bounds on the Lipschitz constant than [25], demonstrating improved estimation accuracy.
• The Ellipsoid Model enables effective outer-approximation of network outputs, with visualization showing that $L_2$-based ellipsoids lie below the decision threshold while $L_\infty$-based ones do not.
• Experimental results confirm that monDEQs are substantially less robust to $L_\infty$ perturbations than to $L_2$ perturbations, contrary to classical DNNs.
• The SDP-based models are effective for certification tasks, but scalability is limited by current SDP solvers' ability to handle matrices larger than 5000x5000.