[Paper Review] Hardware based Spatio-Temporal Neural Processing Backend for Imaging Sensors: Towards a Smart Camera

Samiran Ganguly, Yunfei Gu+2 more|arXiv (Cornell University)|Mar 23, 2018

Neural Networks and Reservoir Computing15 references3 citations

TL;DR

This paper proposes a hardware-based spatio-temporal neural processing backend for imaging sensors, integrating Reservoir Computing for neural filtering, enhanced R-CNNs for spatial feature extraction, and Hierarchical Temporal Memory for motion prediction. The system enables real-time enhancement of image SNR beyond material limits and robust user-defined feature tracking, with scalable CMOS and emerging nanomaterials (e.g., spintronic devices) enabling ultra-low-power embedded neural processors for smart cameras.

ABSTRACT

In this work we show how we can build a technology platform for cognitive imaging sensors using recent advances in recurrent neural network architectures and training methods inspired from biology. We demonstrate learning and processing tasks specific to imaging sensors, including enhancement of sensitivity and signal-to-noise ratio (SNR) purely through neural filtering beyond the fundamental limits sensor materials, and inferencing and spatio-temporal pattern recognition capabilities of these networks with applications in object detection, motion tracking and prediction. We then show designs of unit hardware cells built using complementary metal-oxide semiconductor (CMOS) and emerging materials technologies for ultra-compact and energy-efficient embedded neural processors for smart cameras.

Motivation & Objective

• To develop a cognitive imaging sensor platform that integrates sensing, memory, and logic for real-time spatio-temporal processing.
• To overcome fundamental limits of sensor materials by using neural filtering to enhance signal-to-noise ratio (SNR) and sensitivity beyond physical constraints.
• To enable real-time user-defined feature tracking, including detection, spatial parameter extraction (position, size, rotation), and motion prediction.
• To design energy-efficient, compact neural processing hardware using CMOS and emerging nanomaterials (e.g., spintronic devices) for embedded deployment.

Proposed method

• Employed Echo-State Networks (ESNs), a form of Reservoir Computing, to perform neural filtering that inverts time-dependent sensor distortions and enhances SNR in noisy images.
• Used enhanced Region-based Convolutional Neural Networks (R-CNNs) to extract spatial features and their canonical motion variables: position (x,y), size (h,w), and orientation (θ, φx, φy).
• Applied Hierarchical Temporal Memory (HTM) to learn and predict the generative equations of motion for tracked features using spatio-temporal correlations.
• Designed a generalized stochastic neuron model using spintronic devices (e.g., magnetic tunnel junctions) with tunable energy barriers to emulate non-linear activation functions and stochasticity.
• Implemented synaptic weights via programmable conductors (I = GV) and used hybrid crossbar arrays of memristors to compactly realize synaptic connections.
• Modeled neuron dynamics using a charge-based capacitor (Q_in) and a non-linear transfer function f(α·dQ_in/dt), with added white Gaussian noise (I_rnd) to enable stochastic behavior essential for learning.

Experimental results

Research questions

• RQ1 Can neural filtering via Reservoir Computing recover signals from imaging sensors beyond the fundamental SNR limits imposed by sensor materials?
• RQ2 Can a hybrid neural architecture combining ESNs, R-CNNs, and HTM enable real-time detection, spatial parameter extraction, and motion prediction of user-defined features in video streams?
• RQ3 How can hardware-efficient, energy-compact neural processing units be designed using CMOS and emerging nanomaterials (e.g., spintronic devices) for embedded smart camera applications?
• RQ4 To what extent can the physics of magnetic tunnel junctions with tunable energy barriers support stochastic, low-power neural computation suitable for deep learning workloads?

Key findings

• Neural filtering using ESNs successfully enhanced image SNR and D* beyond the fundamental limits of sensor materials by learning and inverting time-varying distortions.
• Enhanced R-CNNs accurately extracted spatial features with canonical variables: position (x,y), size (h,w), and orientation (θ, φx, φy), enabling robust feature identification.
• HTM-based temporal inferencing learned and predicted motion trajectories of features in real time by modeling spatio-temporal correlations, enabling short-term motion forecasting.
• Spintronic-based stochastic neurons with tunable energy barriers (U = M_s·H_k·Ω/2) enabled hardware-compatible non-linear activation and stochasticity, with state retention times (τ) controllable via U/kT.
• The hybrid crossbar architecture using programmable memristors and metallic interconnects allowed compact, scalable implementation of synaptic weights and neuron arrays for large-scale neural networks.
• The proposed platform demonstrated feasibility of embedding cognitive processing directly near the image sensor using CMOS and emerging materials, enabling ultra-compact, low-power smart camera systems.