[论文解读] On the Algorithmic Power of Spiking Neural Networks

Hitron, Yael, Lynch, Nancy+2 more|arXiv (Cornell University)|Mar 28, 2018

Advanced Memory and Neural Computing参考文献 46被引用 12

一句话总结

该论文通过引入双重视角动力学模型，严格证明了脉冲神经网络（SNNs）能够以多项式时间求解二次优化和ℓ1最小化问题。论文证明了SNNs收敛到最优解的显式多项式时间界，表明在常规条件下其倾向于得到稀疏解（即ℓ1范数最小的解），并将SNNs框架化为一种新型的ℓ1最小化原始-对偶算法，具有理论保证。

ABSTRACT

Spiking Neural Networks (SNN) are mathematical models in neuroscience to describe the dynamics among a set of neurons that interact with each other by firing instantaneous signals, a.k.a., spikes. Interestingly, a recent advance in neuroscience [Barrett-Denève-Machens, NIPS 2013] showed that the neurons' firing rate, i.e., the average number of spikes fired per unit of time, can be characterized by the optimal solution of a quadratic program defined by the parameters of the dynamics. This indicated that SNN potentially has the computational power to solve non-trivial quadratic programs. However, the results were justified empirically without rigorous analysis. We put this into the context of natural algorithms and aim to investigate the algorithmic power of SNN. Especially, we emphasize on giving rigorous asymptotic analysis on the performance of SNN in solving optimization problems. To enforce a theoretical study, we first identify a simplified SNN model that is tractable for analysis. Next, we confirm the empirical observation in the work of Barrett et al. by giving an upper bound on the convergence rate of SNN in solving the quadratic program. Further, we observe that in the case where there are infinitely many optimal solutions, SNN tends to converge to the one with smaller l1 norm. We give an affirmative answer to our finding by showing that SNN can solve the l1 minimization problem under some regular conditions. Our main technical insight is a dual view of the SNN dynamics, under which SNN can be viewed as a new natural primal-dual algorithm for the l1 minimization problem. We believe that the dual view is of independent interest and may potentially find interesting interpretation in neuroscience.

研究动机与目标

• 严格分析脉冲神经网络（SNNs）在求解优化问题方面的算法能力。
• 验证并形式化Barrett等人（2013年）的经验证据，即SNN的发放率可求解二次规划问题。
• 研究SNN是否能高效求解ℓ1最小化问题，以及在何种条件下可以实现。
• 建立SNN动力学在对偶空间中的解释，将其视为ℓ1最小化的一种新型原始-对偶算法。
• 为SNNs求解凸优化问题提供显式的多项式时间收敛界。

提出的方法

• 提出一种简化的、可处理的SNN模型，具有静态连接和输入充电机制，以利于理论分析。
• 引入SNN动力学的双重视角，将系统解释为在对偶空间中运行的原始-对偶算法。
• 使用理想耦合和扰动技术，界定ℓ2残差误差，并通过强对偶性将其与ℓ1误差关联。
• 采用对偶SNN公式来控制对偶变量的动力学，确保收敛到最优解。
• 使用带有狄拉克δ函数和阈值发放规则的脉冲序列模型，定义神经元的放电行为。
• 通过微分方程和类似李雅普诺夫的分析推导收敛速率，建立多项式时间保证。

实验结果

研究问题

• RQ1 SNNs能否以可证明的多项式时间收敛率收敛到对应二次规划的最优解？
• RQ2 当存在多个最优解时，SNN动力学是否在常规条件下倾向于选择ℓ1范数最小的解？
• RQ3 SNN动力学能否被解释为ℓ1最小化的一种原始-对偶算法？
• RQ4 SNNs求解优化问题的显式收敛界是什么？
• RQ5 SNN动力学的双重视角如何实现对其算法能力的严格分析？

主要发现

• SNN的发放率以多项式时间收敛界收敛到对应二次规划的最优解。
• 当存在多个最优解时，在常规条件下SNN会收敛到ℓ1范数最小的解。
• SNN动力学可被解释为一种新型的ℓ1最小化原始-对偶算法，对偶视角提供了理论洞见。
• SNN解的ℓ1误差以受ℓ2残差误差控制的速率收敛到最优值，其系数依赖于最小特征值和问题维度。
• 对于非负最小二乘问题，SNN在时间多项式有界于1/ϵ和问题参数的范围内达到ϵ-近似解，其时间界为 t ≥ Ω(√λmax·n / (ϵ·λmin·∥b∥2))。
• 对偶SNN动力学保持在有界对偶多面体内部，确保稳定性，并可通过对偶空间实现误差分析。