[Paper Review] Inference with Hybrid Bio-hardware Neural Networks

Yuan Zeng, Zubayer Ibne Ferdous+7 more|arXiv (Cornell University)|May 28, 2019

Advanced Memory and Neural Computing31 references4 citations

TL;DR

This paper proposes a two-layer hybrid bio-hardware neural network that uses an in vitro living neural network as the biological layer and a computational fully-connected layer for optimization, achieving 98.3% testing accuracy on the full MNIST dataset through adaptive pre-processing, gradient estimation, and parameter tuning—demonstrating high-accuracy inference with biologically plausible dynamics.

ABSTRACT

To understand the learning process in brains, biologically plausible algorithms have been explored by modeling the detailed neuron properties and dynamics. On the other hand, simplified multi-layer models of neural networks have shown great success on computational tasks such as image classification and speech recognition. However, the computational models that can achieve good accuracy for these learning applications are very different from the bio-plausible models. This paper studies whether a bio-plausible model of a in vitro living neural network can be used to perform machine learning tasks and achieve good inference accuracy. A novel two-layer bio-hardware hybrid neural network is proposed. The biological layer faithfully models variations of synapses, neurons, and network sparsity in in vitro living neural networks. The hardware layer is a computational fully-connected layer that tunes parameters to optimize for accuracy. Several techniques are proposed to improve the inference accuracy of the proposed hybrid neural network. For instance, an adaptive pre-processing technique helps the proposed neural network to achieve good learning accuracy for different living neural network sparsity. The proposed hybrid neural network with realistic neuron parameters and variations achieves a 98.3% testing accuracy for the handwritten digit recognition task on the full MNIST dataset.

Motivation & Objective

• To investigate whether in vitro living neural networks with biological variability can perform machine learning tasks with high inference accuracy.
• To develop a bio-plausible inference mechanism that leverages both biological and computational layers for improved performance.
• To bridge the gap between biologically realistic neural dynamics and high-accuracy machine learning by integrating experimental neural parameters into a hybrid architecture.
• To validate the feasibility of using living neural networks as a platform for testing and implementing computational learning algorithms.

Proposed method

• A two-layer hybrid neural network is designed: the biological layer models real in vitro rat cortical networks with experimentally derived parameters for neurons, synapses, and sparsity.
• The computational layer uses a fully-connected network with tunable weights and hyperparameters to optimize classification accuracy.
• An adaptive pre-processing technique adjusts for varying network sparsity in the biological layer to maintain consistent performance.
• A gradient estimator with a bounded range (0, 0.0075) is used to improve training by selectively passing gradients when input current is below a threshold.
• An adaptive learning rate with exponential decay is applied to the computational layer to enhance convergence and accuracy.
• Parameter tuning via automated search identifies optimal learning rates and weight distributions for both biological and computational layers.

Experimental results

Research questions

• RQ1 Can a hybrid bio-hardware neural network with biologically realistic neuron and synapse dynamics achieve high inference accuracy on standard machine learning benchmarks?
• RQ2 How does biological variability, such as synaptic strength and neuron threshold distribution, affect the performance of a hybrid neural network?
• RQ3 What techniques can effectively compensate for the performance gap between biophysical models and computational models in hybrid learning systems?
• RQ4 Can gradient estimation strategies that respect biological constraints improve training stability and accuracy in hybrid networks?

Key findings

• The hybrid bio-hardware neural network achieved 98.3% testing accuracy on the full MNIST dataset using 2000 hidden neurons, approaching state-of-the-art performance.
• With 2000 hidden neurons, the network reached 97.94% accuracy on the full MNIST dataset after optimization, demonstrating scalability and robustness.
• The adaptive pre-processing technique enabled consistent performance across different levels of network sparsity in the biological layer.
• The use of a bounded gradient estimator (0, 0.0075) improved training by selectively updating weights based on input current, enhancing convergence.
• An adaptive learning rate with exponential decay improved accuracy by 2% over baseline optimization, confirming its effectiveness in the hybrid framework.
• The biophysical model with realistic neuron variations reduced hidden layer firing rates (to 60%) compared to the computational model (80%), yet improved accuracy, validating the model's biological fidelity.