[Paper Review] Controlling Neural Level Sets

Matan Atzmon, Niv Haim+4 more|arXiv (Cornell University)|May 28, 2019

Adversarial Robustness in Machine Learning37 references44 citations

TL;DR

The paper introduces a scalable method to directly sample and incorporate neural network level sets into loss functions, enabling control over decision boundaries and manifold surfaces, with applications to generalization, adversarial robustness, and 3D surface/curve reconstruction.

ABSTRACT

The level sets of neural networks represent fundamental properties such as decision boundaries of classifiers and are used to model non-linear manifold data such as curves and surfaces. Thus, methods for controlling the neural level sets could find many applications in machine learning. In this paper we present a simple and scalable approach to directly control level sets of a deep neural network. Our method consists of two parts: (i) sampling of the neural level sets, and (ii) relating the samples' positions to the network parameters. The latter is achieved by a sample network that is constructed by adding a single fixed linear layer to the original network. In turn, the sample network can be used to incorporate the level set samples into a loss function of interest. We have tested our method on three different learning tasks: improving generalization to unseen data, training networks robust to adversarial attacks, and curve and surface reconstruction from point clouds. For surface reconstruction, we produce high fidelity surfaces directly from raw 3D point clouds. When training small to medium networks to be robust to adversarial attacks we obtain robust accuracy comparable to state-of-the-art methods.

Motivation & Objective

• Motivate and formalize the manipulation of neural network level sets as a practical tool.
• Develop a sample network that links level-set samples to network parameters in a differentiable way.
• Demonstrate how level-set-based losses improve generalization and adversarial robustness.
• Showcase surface and curve reconstruction from point clouds using implicit level-set representations.
• Provide theoretical grounding via a universality result for neural level sets modeled by ReLU networks.

Proposed method

• Define the neural level set S(θ) = {x | F(x; θ) = 0} for a neural network F with parameters θ.
• Sample points p on S(θ) using generalized Newton iterations to project onto the level set.
• Construct a sample network p(θ) by adding a fixed linear layer to F, enabling differentiable dependence of p on θ.
• Formulate loss functions that incorporate the sample network as a proxy for S(θ).
• Propose a geometric margin-based loss (geometric SVM) to increase input-space margin rather than output-space margin.
• Develop a robust training loss for adversarial perturbations operating in input space, leveraging distances to level-set samples.
• Apply a reconstruction loss to encourage level-set surfaces to approximate and pass near given point clouds, enabling surface and curve reconstruction.
• Prove a geometric universality result: any watertight piecewise-linear hypersurface in R^d can be exactly represented as a neural level set of an MLP with ReLU activations.

Experimental results

Research questions

• RQ1 Can neural level sets be sampled efficiently and incorporated into differentiable losses?
• RQ2 Does directly controlling the margin via level-set samples improve generalization on small data regimes?
• RQ3 Can level-set based losses enhance robustness to adversarial perturbations?
• RQ4 Can neural level sets be used to reconstruct curves and surfaces from point clouds, yielding high-fidelity implicit representations?
• RQ5 Is there a geometric universality guarantee for representing piecewise-linear manifolds as neural level sets?

Key findings

• The sample-network construction provides a differentiable link between level-set samples and network parameters, enabling direct loss-based control of S(θ).
• Geometric SVM loss improves generalization on MNIST, Fashion-MNIST, and CIFAR-10 when training with small data fractions compared to cross-entropy and hinge losses.
• Robust training using level-set based margins achieves competitive robustness to PGD attacks on MNIST and CIFAR-10 across several architectures, with some trade-offs for margin-based attacks.
• The method enables direct surface reconstruction from raw 3D point clouds, achieving higher Chamfer distances (lower is better) than a competing method (AtlasNet) on FAUST data, and effectively reconstructs curves by intersecting scalar level-set surfaces.
• A geometric universality theorem shows any watertight piecewise-linear hypersurface in R^d can be exactly represented as the neural level set of an MLP with ReLU activations.