[论文解读] Universal Approximation in Dropout Neural Networks

Oxana A. Manita, Mark A. Peletier+3 more|arXiv (Cornell University)|Dec 18, 2020

Neural Networks and Applications被引用 11

一句话总结

本文建立了dropout神经网络的通用逼近定理，证明了无论采用随机模式还是确定性模式，dropout神经网络均可对任意可测函数实现任意精度的逼近。通过利用期望的代数性质与递归网络展开，作者表明在对激活函数和滤波器分布的假设尽可能弱的情况下，即使输入层的连接被随机丢弃，dropout神经网络仍能保持通用逼近能力。

ABSTRACT

We prove two universal approximation theorems for a range of dropout neural networks. These are feed-forward neural networks in which each edge is given a random $\{0,1\}$-valued filter, that have two modes of operation: in the first each edge output is multiplied by its random filter, resulting in a random output, while in the second each edge output is multiplied by the expectation of its filter, leading to a deterministic output. It is common to use the random mode during training and the deterministic mode during testing and prediction. Both theorems are of the following form: Given a function to approximate and a threshold $\varepsilon>0$, there exists a dropout network that is $\varepsilon$-close in probability and in $L^q$. The first theorem applies to dropout networks in the random mode. It assumes little on the activation function, applies to a wide class of networks, and can even be applied to approximation schemes other than neural networks. The core is an algebraic property that shows that deterministic networks can be exactly matched in expectation by random networks. The second theorem makes stronger assumptions and gives a stronger result. Given a function to approximate, it provides existence of a network that approximates in both modes simultaneously. Proof components are a recursive replacement of edges by independent copies, and a special first-layer replacement that couples the resulting larger network to the input. The functions to be approximated are assumed to be elements of general normed spaces, and the approximations are measured in the corresponding norms. The networks are constructed explicitly. Because of the different methods of proof, the two results give independent insight into the approximation properties of random dropout networks. With this, we establish that dropout neural networks broadly satisfy a universal-approximation property.

研究动机与目标

• 建立dropout神经网络满足通用逼近性质的理论基础，即使存在随机连接过滤。
• 分析dropout的随机性是否干扰网络逼近任意函数的能力。
• 证明随机模式与确定性模式下的dropout网络均可实现任意逼近精度。
• 将通用逼近结果从ReLU网络扩展至广泛的激活函数与滤波器分布类别。
• 提供dropout网络的显式构造方法，使其在Lq范数与概率范数下逼近目标函数。

提出的方法

• 利用一种匹配确定性网络在期望下的代数性质，证明随机模式下dropout网络的通用逼近定理。
• 通过递归地将边替换为独立副本以控制方差，并确保概率收敛。
• 引入一种专门设计的一层替换方法，将扩展后的网络与输入耦合，从而控制输出分布。
• 利用期望-方差分解，在Lq范数与概率范数下界 approximation 误差。
• 将结果推广至一般的赋范函数空间，确保逼近误差在适当的半范数下进行度量。
• 通过利用网络输出的置换对称性，证明输入层dropout不会阻碍通用逼近能力。

实验结果

研究问题

• RQ1 dropout神经网络是否能在随机模式与确定性模式下，对任意可测函数实现任意精度的逼近？
• RQ2 dropout引入的随机性是否会削弱标准神经网络的通用逼近性质？
• RQ3 激活函数的性质与滤波器分布的假设在多大程度上影响dropout网络的逼近能力？
• RQ4 当输入层的连接被随机丢弃时，是否仍能实现通用逼近？
• RQ5 在推理阶段，期望替换dropout方法为何表现出强大性能，其理论基础是什么？

主要发现

• 在随机模式下，dropout网络满足通用逼近定理：对任意ε > 0及目标函数ζ，存在一个dropout网络，使得逼近误差超过ε的概率小于ε。
• 逼近误差在Lq范数下有界：对任意ε > 0，存在一个dropout网络，使得误差的Lq范数小于ε。
• 第一个定理适用于广泛的激活函数与滤波器分布类别，包括基于边的DropConnect与基于节点的Dropout，且假设条件极为宽松。
• 第二个定理构造出一个单一网络，使其在随机模式与确定性模式下均能同时良好逼近目标函数。
• 证明依赖于将第一层通过并行复制进行“放大”，并利用置换对称性控制输出分布。
• 即使存在输入层dropout，网络仍能实现通用逼近，原因在于不同随机实现之间具有很高的概率为彼此的置换。