[Paper Review] Unsupervised Regenerative Learning of Hierarchical Features in Spiking Deep Networks for Object Recognition

Priyadarshini Panda, Kaushik Roy|arXiv (Cornell University)|Feb 3, 2016

Advanced Memory and Neural Computing23 references4 citations

TL;DR

This paper proposes an unsupervised regenerative learning framework for spiking deep networks that leverages spike-timing and latency to train hierarchical features layer-by-layer using an autoencoder-decoder principle. It achieves 0.92% and 29.84% classification error on MNIST and CIFAR-10, respectively, with sparse, event-based representations enabling efficient computation.

ABSTRACT

We present a spike-based unsupervised regenerative learning scheme to train Spiking Deep Networks (SpikeCNN) for object recognition problems using biologically realistic leaky integrate-and-fire neurons. The training methodology is based on the Auto-Encoder learning model wherein the hierarchical network is trained layer wise using the encoder-decoder principle. Regenerative learning uses spike-timing information and inherent latencies to update the weights and learn representative levels for each convolutional layer in an unsupervised manner. The features learnt from the final layer in the hierarchy are then fed to an output layer. The output layer is trained with supervision by showing a fraction of the labeled training dataset and performs the overall classification of the input. Our proposed methodology yields 0.92%/29.84% classification error on MNIST/CIFAR10 datasets which is comparable with state-of-the-art results. The proposed methodology also introduces sparsity in the hierarchical feature representations on account of event-based coding resulting in computationally efficient learning.

Motivation & Objective

• To develop a biologically plausible, unsupervised learning method for spiking neural networks that captures hierarchical features in object recognition.
• To enable efficient, sparse feature representation through event-based coding in deep spiking networks.
• To integrate regenerative learning using spike-timing and intrinsic latencies for weight updates without requiring backpropagation.
• To combine unsupervised pre-training with a small amount of supervised fine-tuning for improved classification accuracy.
• To achieve competitive performance on standard benchmarks like MNIST and CIFAR-10 using only biologically realistic neurons.

Proposed method

• The network uses leaky integrate-and-fire (LIF) neurons to simulate biological spiking behavior.
• A layer-wise autoencoder-decoder training scheme is applied, where each convolutional layer learns to reconstruct its input from encoded features.
• Regenerative learning updates synaptic weights based on spike-timing and inherent neuronal latencies, enabling unsupervised feature learning.
• Spike-based coding introduces sparsity in feature representations, reducing computational load.
• The final hierarchical representation is fed to a supervised output layer trained on a small fraction of labeled data.
• The method combines unsupervised pre-training with minimal supervised fine-tuning to achieve classification.

Experimental results

Research questions

• RQ1 Can spike-timing and latency be effectively used to train deep spiking networks in an unsupervised manner?
• RQ2 How well can hierarchical features be learned through a regenerative autoencoder-decoder framework in spiking networks?
• RQ3 To what extent does event-based coding and sparsity improve computational efficiency in deep spiking networks?
• RQ4 What classification accuracy can be achieved on MNIST and CIFAR-10 using only a small fraction of labeled data after unsupervised pre-training?
• RQ5 How does the performance of this biologically inspired method compare to state-of-the-art deep learning models on standard benchmarks?

Key findings

• The proposed method achieves a classification error of 0.92% on the MNIST dataset, approaching state-of-the-art performance.
• On the CIFAR-10 dataset, the method attains a classification error of 29.84%, demonstrating strong generalization with minimal supervision.
• The network learns sparse, hierarchical representations due to event-based spike coding, which enhances computational efficiency.
• The use of regenerative learning based on spike-timing and latency enables effective unsupervised weight updates without backpropagation.
• The combination of unsupervised pre-training and supervised fine-tuning on a small labeled subset yields high accuracy with low labeling cost.
• The model demonstrates that biologically plausible learning mechanisms can achieve competitive performance on standard object recognition tasks.