[论文解读] Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noise

Xuanqing Liu, Tesi Xiao+4 more|arXiv (Cornell University)|Jun 5, 2019

Adversarial Robustness in Machine Learning参考文献 21被引用 89

一句话总结

本文提出 neural SDE，这是基于随机微分方程的 Neural ODE 的扩展，通过引入噪声（如 dropout、高斯噪声）来正则化和稳定连续网络，从而提高泛化能力以及对对抗和非对抗扰动的鲁棒性。

ABSTRACT

Neural Ordinary Differential Equation (Neural ODE) has been proposed as a continuous approximation to the ResNet architecture. Some commonly used regularization mechanisms in discrete neural networks (e.g. dropout, Gaussian noise) are missing in current Neural ODE networks. In this paper, we propose a new continuous neural network framework called Neural Stochastic Differential Equation (Neural SDE) network, which naturally incorporates various commonly used regularization mechanisms based on random noise injection. Our framework can model various types of noise injection frequently used in discrete networks for regularization purpose, such as dropout and additive/multiplicative noise in each block. We provide theoretical analysis explaining the improved robustness of Neural SDE models against input perturbations/adversarial attacks. Furthermore, we demonstrate that the Neural SDE network can achieve better generalization than the Neural ODE and is more resistant to adversarial and non-adversarial input perturbations.

研究动机与目标

• Motivate and address the lack of regularization in Neural ODEs by introducing stochastic noise in a continuous framework.
• Develop a Neural SDE model that can implement common regularization techniques (dropout, Gaussian noise) within a continuous dynamics setting.
• Derive a scalable backpropagation method for training Neural SDEs, leveraging path-wise gradient and stochastic control concepts.
• Provide theoretical analysis showing that stochasticity can stabilize dynamical systems and improve robustness against perturbations.
• Empirically validate that Neural SDE improves generalization and robustness on CIFAR-10, STL-10, and Tiny-ImageNet datasets.

提出的方法

• 将神经动力学表述为随机微分方程：dh_t = f(h_t,t;w) dt + G(h_t,t;v) dB_t，其中 B_t 为 Brownian motion。
• 通过对扩散项 G(h_t,t;v) 的合适选择来对各种噪声类型（加性、乘性、类似 dropout）进行建模。
• 开发基于路径梯度的反向传播方法，引入一个跟随 SDE 的辅助变量 β_t，以高效计算 ∂h_{t1}/∂w。
• 利用随机 Lyapunov 方法进行稳定性分析，表明适当选择的扩散可以稳定扰动。
• 概述将 Neural SDE 模块置于特征提取器和分类器之间的实际架构，作为可直接替换的正则化组件。

实验结果

研究问题

• RQ1 Can injecting stochastic noise into Neural ODEs improve generalization similarly to discrete regularization techniques (dropout, Gaussian noise)?
• RQ2 How can standard regularization forms (dropout, additive/multiplicative noise) be mapped into continuous SDE dynamics for neural networks?
• RQ3 Does Neural SDE theory provide stability guarantees that explain observed improvements in robustness to perturbations and adversarial attacks?
• RQ4 Can we devise a scalable, memory-efficient gradient method to train Neural SDEs comparable to Neural ODE training?
• RQ5 Do Neural SDEs yield measurable gains in accuracy and robustness across standard vision benchmarks?

主要发现

• Neural SDEs can reproduce common regularization strategies (dropout, additive/multiplicative Gaussian noise) within a continuous-time framework.
• A path-wise gradient method yields unbiased estimators for training Neural SDEs with reduced memory (comparable to adjoint methods for Neural ODEs).
• Theoretical stability analysis shows that appropriately designed diffusion terms can make the system robust to input perturbations, and in some cases lead to almost surely exponentially stable perturbations.
• Empirical results demonstrate improved generalization over Neural ODEs across CIFAR-10, STL-10, and Tiny-ImageNet, with gains such as CIFAR-10 accuracy rising from 81.63% (ODE) to up to 84.55% (Neural SDE with TTN at testing).
• Neural SDEs also exhibit enhanced robustness to non-adversarial corruptions and adversarial perturbations, outperforming Neural ODEs under several attack and corruption settings.