[论文解读] On Robustness of Neural Ordinary Differential Equations

Hanshu Yan, Jiawei Du+2 more|arXiv (Cornell University)|Oct 12, 2019

Model Reduction and Neural Networks参考文献 34被引用 39

一句话总结

经验性地表明 ODENets 相对于 CNNs 对高斯噪声和对抗扰动更具有鲁棒性，并引入 Time-invariant Steady Neural ODEs (TisODE) 以进一步提升鲁棒性，在 MNIST、SVHN 和 ImgNet10 数据集上给出结果。

ABSTRACT

Neural ordinary differential equations (ODEs) have been attracting increasing attention in various research domains recently. There have been some works studying optimization issues and approximation capabilities of neural ODEs, but their robustness is still yet unclear. In this work, we fill this important gap by exploring robustness properties of neural ODEs both empirically and theoretically. We first present an empirical study on the robustness of the neural ODE-based networks (ODENets) by exposing them to inputs with various types of perturbations and subsequently investigating the changes of the corresponding outputs. In contrast to conventional convolutional neural networks (CNNs), we find that the ODENets are more robust against both random Gaussian perturbations and adversarial attack examples. We then provide an insightful understanding of this phenomenon by exploiting a certain desirable property of the flow of a continuous-time ODE, namely that integral curves are non-intersecting. Our work suggests that, due to their intrinsic robustness, it is promising to use neural ODEs as a basic block for building robust deep network models. To further enhance the robustness of vanilla neural ODEs, we propose the time-invariant steady neural ODE (TisODE), which regularizes the flow on perturbed data via the time-invariant property and the imposition of a steady-state constraint. We show that the TisODE method outperforms vanilla neural ODEs and also can work in conjunction with other state-of-the-art architectural methods to build more robust deep networks.

研究动机与目标

• Assess robustness of neural ODE-based networks (ODENets) under Gaussian noise and adversarial perturbations.
• Compare ODENets to conventional CNNs across multiple datasets (MNIST, SVHN, ImgNet10).
• Understand intrinsic robustness properties of ODENets via flow characteristics of continuous-time ODEs.
• Propose and evaluate a robustness-enhancing extension (TisODE) based on time-invariance and steady-state constraints.

提出的方法

• Model architecture: ODENet comprises a feature extractor, a neural ODE mapper, and a fully-connected classifier, trained with standard or perturbed data.
• Perturb robustness evaluation using Gaussian noise, FGSM adversarial examples, and PGD adversarial examples; compare against CNNs with similar parameter counts.
• Empirical analysis leveraging the non-intersecting integral curves property of ODE flows to explain ODENet robustness (Theorem: ODENet integral curves do not intersect).
• Introduce Time-invariant Steady Neural ODE (TisODE) by removing time dependence in dynamics and adding a steady-state regularization term L_ss to constrain output drift over time.
• Evaluate TisODE robustness against perturbations and test its compatibility with other robust methods (FDn, IR) as a drop-in module.

实验结果

研究问题

• RQ1 Do ODENets exhibit greater robustness than CNNs to Gaussian perturbations and adversarial attacks across standard vision datasets?
• RQ2 Can the intrinsic properties of neural ODE flows (non-intersecting integral curves) explain observed robustness in ODENets?
• RQ3 Does time-invariant steady neural ODE (TisODE) provide additional robustness gains over vanilla ODENets, and can it complement other robustness techniques?

主要发现

• ODENets outperform CNNs in robustness across MNIST, SVHN, and ImgNet10 for Gaussian noise and FGSM/PGD adversaries.
• ODENets maintain higher accuracy under strong perturbations (e.g., MNIST with Gaussian σ=100: 73.2% vs 56.4% for CNN; FGSM-0.3: 42.1% vs 14.3% on MNIST).
• The non-intersecting integral curves property of ODE flows offers an intrinsic robustness mechanism for ODENets not present in CNNs.
• TisODE further boosts robustness over vanilla ODENets across perturbations, with notable gains on MNIST FGSM/PGD attacks and SVHN/ImgNet10 tests.
• TisODE serves as a versatile drop-in to enhance robustness and can work with feature denoising and input randomization techniques to yield further improvements.
• Combining TisODE with FDn or IRd yields substantial robustness gains compared to using CNN or ODENet alone.