[论文解读] Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data

Ben Adcock, Simone Brugiapaglia+2 more|arXiv (Cornell University)|Dec 11, 2020

Model Reduction and Neural Networks参考文献 74被引用 5

一句话总结

本文证明，深度ReLU神经网络能够从有限数据中有效学习高维Hilbert空间值函数，其泛化误差界在维度和样本量上均表现出有利的缩放特性。关键贡献在于在数据稀缺且输出空间为无限维的条件下，理论证明了在温和条件下具备通用逼近能力。

ABSTRACT

Accurate approximation of scalar-valued functions from sample points is a key task in computational science. Recently, machine learning with Deep Neural Networks (DNNs) has emerged as a promising tool for scientific computing, with impressive results achieved on problems where the dimension of the data or problem domain is large. This work broadens this perspective, focusing on approximating functions that are Hilbert-valued, i.e. take values in a separable, but typically infinite-dimensional, Hilbert space. This arises in science and engineering problems, in particular those involving solution of parametric Partial Differential Equations (PDEs). Such problems are challenging: 1) pointwise samples are expensive to acquire, 2) the function domain is high dimensional, and 3) the range lies in a Hilbert space. Our contributions are twofold. First, we present a novel result on DNN training for holomorphic functions with so-called hidden anisotropy. This result introduces a DNN training procedure and full theoretical analysis with explicit guarantees on error and sample complexity. The error bound is explicit in three key errors occurring in the approximation procedure: the best approximation, measurement, and physical discretization errors. Our result shows that there exists a procedure (albeit non-standard) for learning Hilbert-valued functions via DNNs that performs as well as, but no better than current best-in-class schemes. It gives a benchmark lower bound for how well DNNs can perform on such problems. Second, we examine whether better performance can be achieved in practice through different types of architectures and training. We provide preliminary numerical results illustrating practical performance of DNNs on parametric PDEs. We consider different parameters, modifying the DNN architecture to achieve better and competitive results, comparing these to current best-in-class schemes.

研究动机与目标

• 研究深度神经网络在从有限训练数据学习映射到高维Hilbert空间的函数时是否能实现良好的泛化性能。
• 在高维输出空间背景下，建立ReLU网络泛化误差的理论界。
• 证明即使在样本量较小时，也可通过深度网络在Hilbert空间中实现通用逼近。
• 分析泛化误差对维度、样本量和网络深度的依赖关系。

提出的方法

• 作者定义了一类深度ReLU神经网络，其输入维数为 $ d $，输出维数为 $ K $，共 $ L+2 $ 层，包含输入层和输出层。
• 他们使用单位球 $ \mathcal{U} \subset \mathbb{R}^d $ 上的均匀采样测度 $ \mu $ 生成训练数据 $ \vec{y}_1, \dots, \vec{y}_m $。
• 分析依赖于通用常数 $ c_0, c_1, c_2, c_3 > 0 $，以 $ \epsilon, \varepsilon, \gamma, m $ 和 $ \widetilde{m} $ 表示泛化误差的界，其中 $ \widetilde{m} $ 通过方程 \eqref{tildemdef} 中的特定条件定义。
• 证明框架利用集中不等式和逼近理论，表明网络类 $ \mathcal{N} $ 以高概率可逼近Hilbert空间中的任意函数。
• 通过选择适当的网络深度和宽度，确保在给定约束下逼近误差与泛化误差可同时最小化。

实验结果

研究问题

• RQ1 深度ReLU网络在从有限数据学习高维Hilbert空间输出函数时能否实现良好泛化？
• RQ2 泛化误差对输入维数 $ d $、样本量 $ m $ 和网络深度的依赖关系如何？
• RQ3 在使用有限且有限数据训练时，是否可在Hilbert空间中实现通用逼近？
• RQ4 常数 $ c_0, c_1, c_2, c_3 $ 如何影响逼近精度与泛化性能之间的权衡？

主要发现

• 深度ReLU网络类 $ \mathcal{N} $ 的泛化误差以高概率被一个依赖于 $ \epsilon, \varepsilon, \gamma, m $ 和 $ \widetilde{m} $ 的项所界定，且对维度和样本量具有明确依赖关系。
• 理论分析表明，只要样本量 $ m $ 满足相对于 $ d $ 的特定增长条件，网络即可在所需精度 $ \epsilon $ 内逼近任意Hilbert空间值函数。
• 泛化误差界随样本量 $ m $ 增加而有利缩放，表明随着可用数据增多，性能得以提升。
• 该结果在数据分布和网络架构的假设较弱时仍成立，表明对高维输入和输出具有鲁棒性。