[论文解读] On the expressiveness and spectral bias of KANs

Yixuan Wang, Jonathan W. Siegel+2 more|arXiv (Cornell University)|Oct 2, 2024

User Authentication and Security Systems被引用 4

一句话总结

本文从理论和实证两方面比较了Kolmogorov-Arnold网络（KANs）与多层感知机（MLPs），表明KANs在表达能力上与MLPs相当或更优，同时显著降低了谱偏差。由于其基于样条的可学习激活函数，KANs在高频函数和偏微分方程（PDEs）上表现出更优的性能，实现了更均衡的频率学习。

ABSTRACT

Kolmogorov-Arnold Networks (KAN) \cite{liu2024kan} were very recently proposed as a potential alternative to the prevalent architectural backbone of many deep learning models, the multi-layer perceptron (MLP). KANs have seen success in various tasks of AI for science, with their empirical efficiency and accuracy demostrated in function regression, PDE solving, and many more scientific problems. In this article, we revisit the comparison of KANs and MLPs, with emphasis on a theoretical perspective. On the one hand, we compare the representation and approximation capabilities of KANs and MLPs. We establish that MLPs can be represented using KANs of a comparable size. This shows that the approximation and representation capabilities of KANs are at least as good as MLPs. Conversely, we show that KANs can be represented using MLPs, but that in this representation the number of parameters increases by a factor of the KAN grid size. This suggests that KANs with a large grid size may be more efficient than MLPs at approximating certain functions. On the other hand, from the perspective of learning and optimization, we study the spectral bias of KANs compared with MLPs. We demonstrate that KANs are less biased toward low frequencies than MLPs. We highlight that the multi-level learning feature specific to KANs, i.e. grid extension of splines, improves the learning process for high-frequency components. Detailed comparisons with different choices of depth, width, and grid sizes of KANs are made, shedding some light on how to choose the hyperparameters in practice.

研究动机与目标

• 从理论上分析KANs相对于MLPs的近似与表征能力。
• 研究KANs与MLPs在科学机器学习背景下的谱偏差现象。
• 为KANs在求解PDE和拟合高频函数方面观察到的成功提供理论与实证依据。
• 通过研究深度、宽度和网格大小对谱偏差与泛化能力的影响，指导KANs的超参数选择。
• 解释KANs为何更容易过拟合，并探讨通过网格粗化缓解该问题的方法。

提出的方法

• 证明任意ReLU激活的MLP均可重参数化为参数量相当的KAN，从而确立KANs的表达能力至少与MLPs相当或更优。
• 构建KAN的MLP表示形式，表明参数量增加的比例与KAN的网格大小成正比。
• 分析单个KAN层上的梯度下降动力学，论证KANs不会像MLPs那样表现出对低频的强烈谱偏差。
• 通过在1D频率拟合、高斯随机场以及具有高频解的1D泊松方程上的数值实验，比较谱偏差与泛化能力。
• 在KANs中应用网格扩展，以细化样条近似并提升对高频分量的学习能力。
• 使用LBFGS优化算法，结合均方误差损失，在合成数据集和PDE数据集上评估不同架构与超参数下的训练与测试性能。

Figure 1 : 1D wave dataset, where the target function has equal amplitudes of different frequency modes. Under various hyperparameters, MLPs manifest strong spectral biases (top), while KANs do not (bottom). Note that the y axis (training steps) of MLP is 10 times that of KAN.

实验结果

研究问题

• RQ1 KANs能否以相近的参数效率近似任何MLP可近似的函数？
• RQ2 在训练过程中，KANs的谱偏差与标准MLPs相比如何？
• RQ3 KANs中的网格扩展在多大程度上提升了对高频分量的学习能力？
• RQ4 为何KANs表现出更高的过拟合倾向，以及如何通过超参数调优缓解此问题？
• RQ5 KANs中降低的谱偏差是否导致其在求解高频PDE等科学计算任务中表现更优？

主要发现

• KANs能够以相近的参数量表示任意ReLU激活的MLP，表明KANs的表达能力至少与MLPs相当或更优。
• 反之，若用MLP表示一个KAN，其参数量将按与KAN网格大小成正比的因子增加，表明当KAN网格较大时，KANs可能具有更高的参数效率。
• 对单个KAN层梯度下降动力学的理论分析表明，KANs不会像MLPs那样表现出对低频的强烈谱偏差。
• 实证结果表明，KANs在高频函数回归和PDE求解任务中始终优于MLPs，且随着频率增加，优势更加显著。
• KANs在噪声较大或结构复杂的数据上更容易过拟合，但通过网格粗化可有效降低过拟合，该方法增加了谱偏差。
• 增加训练数据量可显著减少KANs的过拟合现象，且网格扩展能提升其在粗糙或低尺度函数上的性能。

Figure 2 : 1D wave dataset, where the target function has increasing amplitudes of different frequency modes. Under various hyperparameters, MLPs manifest severe spectral biases (top), while KANs do not (bottom). Note that the y axis (training steps) of MLP is 10 times that of KAN.