[论文解读] NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems

Jason Yik, Berghe, Korneel Van den+98 more|arXiv (Cornell University)|Apr 10, 2023

Advanced Memory and Neural Computing被引用 16

一句话总结

NeuroBench 引入一个双轨基准框架用于神经形态计算，一个算法轨用于硬件无关评估，一个系统轨用于硬件部署，以及若干任务的基线结果。

ABSTRACT

Neuromorphic computing shows promise for advancing computing efficiency and capabilities of AI applications using brain-inspired principles. However, the neuromorphic research field currently lacks standardized benchmarks, making it difficult to accurately measure technological advancements, compare performance with conventional methods, and identify promising future research directions. Prior neuromorphic computing benchmark efforts have not seen widespread adoption due to a lack of inclusive, actionable, and iterative benchmark design and guidelines. To address these shortcomings, we present NeuroBench: a benchmark framework for neuromorphic computing algorithms and systems. NeuroBench is a collaboratively-designed effort from an open community of researchers across industry and academia, aiming to provide a representative structure for standardizing the evaluation of neuromorphic approaches. The NeuroBench framework introduces a common set of tools and systematic methodology for inclusive benchmark measurement, delivering an objective reference framework for quantifying neuromorphic approaches in both hardware-independent (algorithm track) and hardware-dependent (system track) settings. In this article, we outline tasks and guidelines for benchmarks across multiple application domains, and present initial performance baselines across neuromorphic and conventional approaches for both benchmark tracks. NeuroBench is intended to continually expand its benchmarks and features to foster and track the progress made by the research community.

研究动机与目标

• 提供一个正式、包容的神经形态计算基准框架，以实现不同方法之间的公平比较。
• 引入两个连贯轨道（算法与系统），覆盖软件/设计与部署硬件视角。
• 提供一个社区驱动的、迭代的基准套件，配备开源工具和排行榜以跟踪进展。
• 在关键神经形态基准上给出基线结果，引导未来在算法效率和硬件部署方面的研究。

提出的方法

• 定义硬件无关（算法）和硬件相关（系统）基准轨道。
• 给出一个通用 harness 和度量标准，以在多样化的神经形态方法之间实现公平、可重复的评估。
• 提出四个算法轨道基准（少样本持续学习、基于事件的目标检测、运动皮质解码、混沌预测），并包含复杂度和正确性指标。
• 提供在 FSCIL 关键词任务上对 ANN 与 SNN 方法的基线算法结果比较。
• 概述系统轨道协议，以评估神经形态硬件在适用工作负载上的真实世界速度与效率。

Figure 1 : The two NeuroBench tracks: algorithms and systems. Grey boxes designate what is defined by the benchmark, and orange boxes indicate what is unique to each solution. Connecting arrows between the two tracks denote the co-innovation between the tracks and the cross-stack innovation enabled

实验结果

研究问题

• RQ1 如何标准化神经形态研究，以实现对多样化方法的公平、包容性基准？
• RQ2 哪些关键指标能够捕捉算法复杂度、正确性和硬件无关性能？
• RQ3 硬件无关的算法基准如何为后续系统轨道的硬件设计与部署策略提供信息？
• RQ4 在 NeuroBench 下，当前的 ANN 与 SNN 方法在具有代表性的神经形态任务上达到的基线性能如何？
• RQ5 NeuroBench 如何随时间演变以容纳新的模态和任务？

主要发现

Task	Dataset	Correctness metric	Task description
Keywords FSCIL	MSWC	Accuracy	Few-shot, continual learning of keyword classes.
Event Camera Object Detection	Prophesee 1MP Automotive	COCO mAP	Detecting automotive objects from event camera video.
NHP Motor Prediction	Primate Reaching	R^2	Predicting fingertip velocity from cortical recordings.
Chaotic Function Prediction	Mackey-Glass time series	sMAPE	Autoregressive modeling of chaotic functions.

• NeuroBench 定义了一个两轨框架（算法和系统），并配有一个通用 harness，使硬件无关的比较成为可能，并实现轨道之间的跨栈影响。
• 算法轨道基准同时捕捉正确性和复杂度指标，包括 footprint、模型执行速率、连接稀疏性、激活稀疏性和突触操作。
• 在 v1.0 的算法轨道中，确立了四个基准：FSCIL、事件相机目标检测、非人灵长类动物运动预测、以及混沌函数预测。
• 基线比较显示 FSCIL 任务的 M5 ANN 与 SNN 基线在 footprint、执行速率和突触操作轮廓方面存在差异，体现了准确性与效率之间的权衡。
• FSCIL 基线在基础类上的基线准确率分别为 97.09%（ANN）和 93.48%（SNN），在 footprint、执行速率、稀疏性和突触操作方面存在显著差异。
• 该框架是开源的，设计用于持续的社区驱动扩展，包括未来数据模态（如 IMU）和闭环任务的潜在增加。

Figure 2 : An overview of the NeuroBench algorithm track.