[Paper Review] KAN: Kolmogorov-Arnold Networks

Ziming Liu, Yixuan Wang+6 more|arXiv (Cornell University)|Apr 30, 2024

Computability, Logic, AI Algorithms481 citations

TL;DR

KANs replace traditional neural weights with learnable edge activations (splines), enabling smaller, more accurate networks with improved interpretability compared to MLPs, demonstrated on data fitting and PDE solving and with interactive symbolic insights.

ABSTRACT

Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs.

Motivation & Objective

• Introduce Kolmogorov-Arnold Networks (KANs) as an alternative to MLPs by learning activation functions on edges via splines.
• Demonstrate superior accuracy and parameter efficiency in data fitting and PDE solving, and establish neural scaling laws.
• Develop grid extension and simplification techniques to improve accuracy and interpretability.
• Showcase interpretability tools (visualization, pruning, symbolification) and collaborative use in science (math/physics).

Proposed method

• Represent f via the Kolmogorov-Arnold form with univariate functions and sums, implemented as KAN layers where activations are edge functions.
• Parameterize 1D functions with B-splines; implement a matrix of function-activations between layers (Φ_l) to replace weight matrices.
• Use residual activation design and careful initialization; update spline grids during training to adapt to activation magnitudes.
• Extend depth and width to form deeper KANs; provide a theoretical approximation bound (Theorem 2.1) showing grid-size-based error decay.
• Introduce grid-extension to refine accuracy without retraining from scratch; 2) sparsification/entropy regularization and pruning for interpretability; 3) symbolification interfaces for interactive symbolic discovery.
• Provide an interpretable framework with visualization tools and an interactive workflow for symbolic regression-inspired insights.

Figure 0.1: Multi-Layer Perceptrons (MLPs) vs. Kolmogorov-Arnold Networks (KANs)

Experimental results

Research questions

• RQ1 Can KANs achieve higher accuracy with fewer parameters than MLPs across representative tasks (data fitting and PDE solving)?
• RQ2 Do KANs exhibit favorable neural scaling laws under smooth Kolmogorov-Arnold representations?
• RQ3 How can grid extension and sparsification improve KAN accuracy and interpretability without excessive retraining?
• RQ4 How can KANs be used as collaborative tools for scientific discovery through visualization, pruning, and symbolic interaction?

Key findings

• A 2-layer, width-10 KAN achieves substantially higher accuracy than a 4-layer, width-100 MLP in PDE solving (MSE 10^-7 vs 10^-5) with 100x fewer parameters (100 vs 10^4).
• KANs exhibit faster neural scaling laws, with an effective exponent alpha up to 4 (k+1 for cubic splines with k=3), and empirical results approaching this bound.
• Grid extension enables systematic improvements in accuracy by refining spline grids, with training-time considerations and interpolation-threshold behavior discussed.
• Simplification techniques (sparsification with L1-like norms plus entropy regularization, pruning by node importance, and symbolification) yield more interpretable networks and enable interactive symbolic discovery in examples like exp(sin(pi x)+y^2).
• A theoretical approximation bound (Theorem 2.1) shows that smooth KAN representations with finite grids can approximate smooth functions with error decaying as CG^{-k-1+m}, indicating potential to beat the curse of dimensionality under suitable representations.
• KANs emphasize external vs internal degrees of freedom, combining compositional learning (external) with learnable univariate functions (internal) to achieve both structure and accuracy.

Figure 2.1: Our proposed Kolmogorov-Arnold networks are in honor of two great late mathematicians, Andrey Kolmogorov and Vladimir Arnold. KANs are mathematically sound, accurate and interpretable.