[Paper Review] Exploring the Limitations of Kolmogorov-Arnold Networks in Classification: Insights to Software Training and Hardware Implementation

Van Duy Tran, Tran Xuan Hieu Le+6 more|arXiv (Cornell University)|Jul 25, 2024

Neural Networks and Applications4 citations

TL;DR

This paper evaluates Kolmogorov-Arnold Networks (KANs) against Multi-Layer Perceptrons (MLPs) in classification tasks across four datasets, combining software training and FPGA-based hardware implementation. Despite KANs' theoretical advantages in interpretability and accuracy, the results show KANs underperform MLPs in both accuracy and hardware efficiency, requiring significantly more resources and higher power consumption while achieving no consistent accuracy gain.

ABSTRACT

Kolmogorov-Arnold Networks (KANs), a novel type of neural network, have recently gained popularity and attention due to the ability to substitute multi-layer perceptions (MLPs) in artificial intelligence (AI) with higher accuracy and interoperability. However, KAN assessment is still limited and cannot provide an in-depth analysis of a specific domain. Furthermore, no study has been conducted on the implementation of KANs in hardware design, which would directly demonstrate whether KANs are truly superior to MLPs in practical applications. As a result, in this paper, we focus on verifying KANs for classification issues, which are a common but significant topic in AI using four different types of datasets. Furthermore, the corresponding hardware implementation is considered using the Vitis high-level synthesis (HLS) tool. To the best of our knowledge, this is the first article to implement hardware for KAN. The results indicate that KANs cannot achieve more accuracy than MLPs in high complex datasets while utilizing substantially higher hardware resources. Therefore, MLP remains an effective approach for achieving accuracy and efficiency in software and hardware implementation.

Motivation & Objective

• To evaluate the practical effectiveness of KANs in classification tasks compared to MLPs across diverse datasets.
• To investigate the hardware resource overhead of implementing KANs on FPGAs using Vitis High-Level Synthesis (HLS).
• To assess the trade-off between accuracy and hardware efficiency in KANs versus MLPs for real-world deployment.
• To determine whether KANs' theoretical benefits in interpretability and function approximation translate into practical advantages in classification.
• To provide the first hardware implementation of KANs, enabling direct comparison with MLPs in resource utilization and latency.

Proposed method

• Training KANs and MLPs on four benchmark datasets: moons (binary), wine (three-class), dry bean (seven-class), and mushroom (binary) classification.
• Using the Kolmogorov-Arnold representation theorem to structure KANs as compositions of univariate functions, replacing weight matrices with learnable spline functions.
• Implementing both KAN and MLP models in hardware using Vitis HLS on FPGA platforms to measure resource usage, latency, and power consumption.
• Converting trained KAN models into symbolic formulas for hardware mapping, which involves parameterizing and synthesizing univariate spline functions.
• Measuring and comparing key hardware metrics: BRAMs, DSPs, LUTs, FFs, latency, and Power Delay Product (PDP).
• Analyzing the correlation between model complexity, hardware resource usage, and classification accuracy across all datasets.

Figure 1: The generalized KAN architecture

Experimental results

Research questions

• RQ1 Do KANs achieve higher classification accuracy than MLPs on diverse, real-world datasets?
• RQ2 How does the hardware resource consumption of KANs compare to that of MLPs when implemented on FPGAs?
• RQ3 Does the symbolic formula representation of KANs lead to better hardware efficiency or increased resource overhead?
• RQ4 What is the trade-off between accuracy and hardware efficiency (latency, power, area) in KANs versus MLPs?
• RQ5 Can KANs be practically deployed in hardware-constrained environments, given their high resource demands?

Key findings

• KANs failed to outperform MLPs in classification accuracy across all four datasets, with MLPs achieving consistently higher or comparable performance.
• KANs required significantly more hardware resources than MLPs: DSPs increased by up to 958.76 times (e.g., 16,299 vs. 17 in the mushroom dataset), and LUTs by up to 284.7 times.
• Power consumption of KANs was substantially higher, reaching 14.802W in the dry bean dataset—over 20 times higher than MLPs—due to excessive resource usage.
• Latency in KANs was higher than in MLPs, with cycle counts ranging from 1.1× to 2.27× longer (e.g., 3,434 vs. 3,128 cycles in the mushroom dataset).
• The symbolic formula conversion process for KANs was time-consuming and error-prone, requiring extensive manual tuning to minimize accuracy loss.
• Despite lower parameter counts, KANs consumed more hardware resources than MLPs, indicating that symbolic function representation is far more resource-intensive than matrix multiplication.

Figure 2: The training time between MLPs and KANs