[论文解读] Boosting Dilated Convolutional Networks with Mixed Tensor Decompositions

Nadav Cohen, Ronen Tamari+1 more|arXiv (Cornell University)|Mar 20, 2017

Tensor decomposition and applications参考文献 21被引用 5

一句话总结

本文引入混合张量分解以增强空洞卷积神经网络，表明中间层之间的战略性连接可实现表达效率——将模型大小近乎以平方因子减少。实证结果表明，这种由连接性驱动的效率可提升准确率，其增益效果与单纯增加深度相当。

ABSTRACT

The driving force behind deep networks is their ability to compactly represent rich classes of functions. The primary notion for formally reasoning about this phenomenon is expressive efficiency, which refers to a situation where one network must grow unfeasibly large in order to realize (or approximate) functions of another. To date, expressive efficiency analyses focused on the architectural feature of depth, showing that deep networks are representationally superior to shallow ones. In this paper we study the expressive efficiency brought forth by connectivity, motivated by the observation that modern networks interconnect their layers in elaborate ways. We focus on dilated convolutional networks, a family of deep models delivering state of the art performance in sequence processing tasks. By introducing and analyzing the concept of mixed tensor decompositions, we prove that interconnecting dilated convolutional networks can lead to expressive efficiency. In particular, we show that even a single connection between intermediate layers can already lead to an almost quadratic gap, which in large-scale settings typically makes the difference between a model that is practical and one that is not. Empirical evaluation demonstrates how the expressive efficiency of connectivity, similarly to that of depth, translates into gains in accuracy. This leads us to believe that expressive efficiency may serve a key role in the development of new tools for deep network design.

研究动机与目标

• 探究在空洞卷积神经网络中连接中间层是否能带来超越深度的表达效率。
• 在层连接的背景下，形式化并分析混合张量分解的概念。
• 评估由连接性引发的表达效率如何转化为序列建模中的实际性能提升。
• 探索连接性是否能够匹配或超越传统上归因于深度的表达效率优势。

提出的方法

• 引入混合张量分解作为数学框架，以分析和利用空洞卷积神经网络中的层间连接性。
• 通过连接性实现的模型大小缩减，形式化定义表达效率，类比于基于深度的效率。
• 分析单个层间连接对表达效率的影响，表明所需模型大小近乎呈平方减少。
• 设计并训练具有受控连接模式的空洞卷积神经网络，以实证验证理论假设。
• 使用张量分解技术表示并压缩空洞卷积的权重张量，实现在层间高效共享参数。
• 对比具有和不具有中间连接的模型，以隔离连接性对表达效率的贡献。

实验结果

研究问题

• RQ1 在空洞卷积神经网络中，连接中间层是否能带来表达效率？
• RQ2 由连接性引发的表达效率与源自深度的表达效率相比如何？
• RQ3 单个层间连接对模型大小和性能的定量影响是什么？
• RQ4 由连接性带来的表达效率在多大程度上转化为准确率的提升？

主要发现

• 在空洞卷积神经网络中，单个层间连接即可实现模型大小近乎平方级的减少，展现出显著的表达效率。
• 理论分析表明，混合张量分解可通过战略性连接实现复杂函数的紧凑表示。
• 实证结果证实，由连接性驱动的表达效率可提升测试准确率，其效果与增加深度带来的增益相似。
• 所提出的方法使原本在实际中不可行的超大网络规模变为可实现的实用模型大小。