[Paper Review] Artificial to Spiking Neural Networks Conversion for Scientific Machine Learning

Qian Zhang, Chenxi Wu+5 more|arXiv (Cornell University)|Aug 31, 2023

Neural Networks and Reservoir Computing4 citations

TL;DR

This paper proposes a novel method to convert Physics-Informed Neural Networks (PINNs) and Separable PINNs (SPINNs) into Spiking Neural Networks (SNNs) for energy-efficient scientific machine learning. By extending SNN calibration to arbitrary activation functions—including tanh and sine—and proving its convergence, the approach enables accurate, low-spike-rate SNN inference on differential equations like Navier-Stokes, achieving up to 17.5% spike rate and relative L² errors below 5% on complex fluid dynamics problems.

ABSTRACT

We introduce a method to convert Physics-Informed Neural Networks (PINNs), commonly used in scientific machine learning, to Spiking Neural Networks (SNNs), which are expected to have higher energy efficiency compared to traditional Artificial Neural Networks (ANNs). We first extend the calibration technique of SNNs to arbitrary activation functions beyond ReLU, making it more versatile, and we prove a theorem that ensures the effectiveness of the calibration. We successfully convert PINNs to SNNs, enabling computational efficiency for diverse regression tasks in solving multiple differential equations, including the unsteady Navier-Stokes equations. We demonstrate great gains in terms of overall efficiency, including Separable PINNs (SPINNs), which accelerate the training process. Overall, this is the first work of this kind and the proposed method achieves relatively good accuracy with low spike rates.

Motivation & Objective

• To enable energy-efficient inference in scientific machine learning by converting trained PINNs to Spiking Neural Networks (SNNs).
• To extend SNN calibration techniques beyond ReLU to arbitrary nonlinear activation functions, such as tanh and sine, used in PINNs.
• To prove a convergence theorem for the generalized calibration method, ensuring accuracy preservation during conversion.
• To demonstrate the method’s effectiveness on regression tasks involving partial differential equations, including unsteady Navier-Stokes flows.
• To achieve low spike rates in SNNs, enabling efficient deployment on neuromorphic hardware.

Proposed method

• The method generalizes SNN calibration by incorporating the training loss as an additional term in the error bound, relaxing the prior reliance on ReLU’s second-order derivative being zero.
• A convergence theorem is proven for the generalized calibration, ensuring that the SNN output approximates the original ANN within a bounded error.
• The conversion process applies the calibrated SNN transformation to both standard PINNs and SPINNs, preserving network architecture and activation functions.
• The approach supports smooth, non-ReLU activations common in PINNs, such as tanh and sine, enabling broader applicability.
• SNN inference is evaluated via spike rate and L² error metrics on multiple PDEs, including Poisson, diffusion-reaction, wave, Burgers, and Navier-Stokes equations.
• The method is applied to SPINNs, which accelerate training by processing variables independently, further enhancing efficiency.

Figure 1: Illustration of $\operatorname{ClipFloor}$ function and its relationship to $\operatorname{ReLU}$ (left) and $\tanh$ (right).

Experimental results

Research questions

• RQ1 Can SNN calibration be generalized beyond ReLU to support smooth activation functions like tanh and sine used in PINNs?
• RQ2 Does the proposed calibration method preserve network accuracy when converting PINNs to SNNs for solving PDEs?
• RQ3 Can the conversion achieve low spike rates, indicating energy efficiency, on scientific machine learning tasks?
• RQ4 How does the performance of the converted SNN compare to the original PINN or SPINN in terms of error and inference speed?
• RQ5 Can the method be effectively applied to complex PDEs such as the unsteady Navier-Stokes equations?

Key findings

• The generalized SNN calibration method achieves a relative L² error of 4.33% on the unsteady Navier-Stokes Beltrami flow problem, demonstrating high accuracy.
• The SNN converted from a SPINN achieves a spike rate of only 17.54% on the Navier-Stokes problem, indicating significant computational efficiency.
• The method maintains low spike rates across all tested equations, with the lowest being 28.79% for the diffusion-reaction equation.
• The SNN conversion preserves accuracy, with L² errors below 8.3×10⁻³ across all test problems, including the most complex Navier-Stokes case.
• The approach enables energy-efficient inference on neuromorphic hardware by ensuring sparse, low-activity spiking patterns.
• The theoretical convergence guarantee ensures reliable performance across diverse activation functions and PDEs.

(a) T=8