[論文レビュー] Mathematics of Deep Learning

Renè Vidal, Joan Bruna+2 more|arXiv (Cornell University)|Dec 13, 2017

Sparse and Compressive Sensing Techniques参考文献 64被引用数 79

ひとこと要約

深層ネットワークの数学的正当化を要約するチュートリアルで、global optimality、stability、invariance、structure、および representations の情報理論的側面をカバーします。

ABSTRACT

Recently there has been a dramatic increase in the performance of recognition systems due to the introduction of deep architectures for representation learning and classification. However, the mathematical reasons for this success remain elusive. This tutorial will review recent work that aims to provide a mathematical justification for several properties of deep networks, such as global optimality, geometric stability, and invariance of the learned representations.

研究の動機と目的

• Explain how deep architectures can approximate arbitrary functions and why depth aids invariance and generalization.
• Survey optimization landscapes, global optimality results, and connections to regularization and SGD behavior.
• Discuss geometric stability and invariance properties of CNNs and scattering networks.
• Outline structure-based theory linking data geometry, generalization, and representation learning.
• Introduce information-theoretic perspectives on representations and regularization.

提案手法

• Model the deep network as a composition of linear transforms and nonlinear activations; formalize the input–output map Phi(X, W).
• Review global optimality results for networks, including positively homogeneous architectures and conditions ensuring global minimizers.
• Discuss optimization landscapes and SGD behavior in high dimensions, including saddle points and flat minima.
• Present geometric stability via invariance to translations and deformations, with CNNs and scattering networks as key examples.
• Analyze data geometry propagation through random-weight networks and implications for metric preservation and generalization.
• Discuss information-theoretic representations and regularizers, including the information bottleneck and information dropout.

実験結果

リサーチクエスチョン

• RQ1 Under what conditions do deep networks admit global minima despite non-convexity?
• RQ2 How do depth, invariances, and geometric priors contribute to generalization and stability?
• RQ3 What is the role of random weights and structured representations in preserving data geometry across layers?
• RQ4 How can information-theoretic principles inform regularization and robustness of learned representations?

主な発見

• For large enough networks with positively homogeneous components, certain critical points are global minima or saddles, explaining empirical success of ReLU networks.
• Convolutional architectures inherently provide stationarity and deformation stability, enabling scalable sample complexity.
• Random-weight networks preserve data geometry across layers, supporting stable recovery and angle-based separation of classes.
• Information-theoretic approaches (e.g., information bottleneck) offer regularization that yields disentangled representations and potential robustness benefits.
• SGD in non-convex, high-dimensional landscapes tends to find flat minima and may avoid poor local minima, with connections to PDE-based interpretations.