[论文解读] On the Expressive Power of Overlapping Operations of Deep Networks.

Or Sharir, Amnon Shashua|arXiv (Cornell University)|Mar 6, 2017

Ferroelectric and Negative Capacitance Devices被引用 3

一句话总结

本文研究了重叠卷积（卷积步长小于滤波器大小）如何增强深度神经网络的表达能力。通过将卷积算术电路（ConvACs）作为理论模型，本文证明重叠操作可使表达能力呈指数级增长，使现代架构在无需全连接层的情况下，天然地比浅层模型更强大。

ABSTRACT

Expressive efficiency refers to the relation between two architectures A and B, whereby any function realized by B could be replicated by A, but there exists functions realized by A, which cannot be replicated by B unless its size grows significantly larger. For example, it is known that deep networks are exponentially efficient with respect to shallow networks, in the sense that a shallow network must grow exponentially large in order to approximate the functions represented by a deep network of polynomial size. In this work, we extend the study of expressive efficiency to the attribute of network connectivity and in particular to the effect of overlaps in the convolutional process, i.e., when the stride of the convolution is smaller than its filter size (receptive field). To theoretically analyze this aspect of network's design, we focus on a well-established surrogate for ConvNets called Convolutional Arithmetic Circuits (ConvACs), and then demonstrate empirically that our results hold for standard ConvNets as well. Specifically, our analysis shows that having overlapping local receptive fields, and more broadly denser connectivity, results in an exponential increase in the expressive capacity of neural networks. Moreover, while denser connectivity can increase the expressive capacity, we show that the most common types of modern architectures already exhibit exponential increase in expressivity, without relying on fully-connected layers.

研究动机与目标

• 理解重叠卷积如何影响深度网络的表达效率。
• 分析密集连接性在提升神经架构表征能力方面的作用。
• 确定现代卷积架构是否能在不依赖全连接层的情况下实现指数级表达增益。

提出的方法

• 本研究使用卷积算术电路（ConvACs）作为标准ConvNets的理论替代模型，以分析表达效率。
• 从另一架构可表示函数所需的网络规模角度，形式化表达效率。
• 分析聚焦于重叠局部感受野（即步长 < 滤波器大小）对表达能力的影响。
• 理论推导表明，重叠卷积可使可表示函数的数量呈指数级增长。
• 在标准ConvNets上进行实证验证，以确认理论发现可在实际中成立。
• 通过比较具有不同连接度的架构，隔离重叠对表达能力的影响。

实验结果

研究问题

• RQ1 重叠卷积操作如何影响深度网络的表达能力？
• RQ2 通过重叠感受野实现的更密集连接性，在多大程度上增强了表征能力？
• RQ3 现代卷积架构是否能在不使用全连接层的情况下实现指数级表达效率？
• RQ4 重叠卷积的表达优势是否可数学量化，并可迁移至标准ConvNets？

主要发现

• 与非重叠版本相比，重叠卷积使神经网络的表达能力呈指数级增长。
• 通过重叠感受野实现的更密集连接性，显著提升了可表示函数的数量。
• 基于ConvACs的理论分析表明，重叠操作可实现指数级表达效率。
• 即使没有全连接层，现代卷积架构也已展现出指数级的表达增益。
• 实证结果证实，ConvACs上的理论发现可推广至标准ConvNets。