[论文解读] New Reinforcement Learning Using a Chaotic Neural Network for Emergence of "Thinking" - "Exploration" Grows into "Thinking" through Learning -

Katsunari Shibata, Yuki Goto|arXiv (Cornell University)|May 16, 2017

Neural Networks and Applications参考文献 5被引用 6

一句话总结

本文提出了一种基于混沌神经网络（ChNN）的新型强化学习框架，其中内在的混沌动力学在无需外部随机性的情况下产生探索行为，通过学习到的理性状态转移实现类似‘思考’的行为。该方法利用因果性轨迹和时序差分误差更新权重，在2D机器人避障任务中成功实现学习，探索能力得到提升，并涌现出目标导向行为。

ABSTRACT

Expectation for the emergence of higher functions is getting larger in the framework of end-to-end reinforcement learning using a recurrent neural network. However, the emergence of "thinking" that is a typical higher function is difficult to realize because "thinking" needs non fixed-point, flow-type attractors with both convergence and transition dynamics. Furthermore, in order to introduce "inspiration" or "discovery" in "thinking", not completely random but unexpected transition should be also required. By analogy to "chaotic itinerancy", we have hypothesized that "exploration" grows into "thinking" through learning by forming flow-type attractors on chaotic random-like dynamics. It is expected that if rational dynamics are learned in a chaotic neural network (ChNN), coexistence of rational state transition, inspiration-like state transition and also random-like exploration for unknown situation can be realized. Based on the above idea, we have proposed new reinforcement learning using a ChNN as an actor. The positioning of exploration is completely different from the conventional one. The chaotic dynamics inside the ChNN produces exploration factors by itself. Since external random numbers for stochastic action selection are not used, exploration factors cannot be isolated from the output. Therefore, the learning method is also completely different from the conventional one. At each non-feedback connection, one variable named causality trace takes in and maintains the input through the connection according to the change in its output. Using the trace and TD error, the weight is updated. In this paper, as the result of a recent simple task to see whether the new learning works or not, it is shown that a robot with two wheels and two visual sensors reaches a target while avoiding an obstacle after learning though there are still many rooms for improvement.

研究动机与目标

• 为解决强化学习中‘思考’行为的涌现问题，即具有启发式动力学的理性、多阶段状态转移。
• 克服传统强化学习依赖外部随机噪声进行探索的局限性，该方法在难以同时实现收敛与状态转移动力学方面存在挑战。
• 通过在循环神经网络中利用混沌穿梭（chaotic itinerancy）实现规划与发现等更高阶认知功能的涌现。
• 开发一种统一的确定性框架，将探索、学习与理性决策机制整合于一体。

提出的方法

• 采用混沌神经网络（ChNN）作为演员-评论家架构，其内部混沌动力学自然产生探索行为，无需外部随机噪声。
• 在每条连接上引入因果性轨迹 $ C_{ji} $，用于追踪输入-输出关系，其更新基于输出变化：$ C^{[l]}_{ji,t} = (1 - | abla x^{[l]}_{j,t}|)C^{[l]}_{ji,t-1} + abla x^{[l]}_{j,t} x^{[l-1]}_{i,t} $。
• 权重更新基于时序差分（TD）误差与因果性轨迹：$ abla w^{[l]}_{ji,t} = heta \cdot \hat{r}_t \cdot C^{[l]}_{ji,t} $，其中 $ \theta $ 为学习率。
• 评论家网络用于估计价值函数，而演员网络则基于ChNN内部动力学生成动作。
• 即使在学习之后，系统仍保持混沌行为，由正的李雅普诺夫指数证实。
• 探索并未与输出解耦；相反，它从ChNN的动力学中自然涌现，支持在理性与探索状态之间自适应切换。

实验结果

研究问题

• RQ1 神经网络中的内在混沌动力学是否能够支持‘思考’行为的涌现，如具有理性的多阶段状态转移？
• RQ2 如何将探索整合进学习过程，使其演化为结构化、目标导向的行为，而非外部注入？
• RQ3 因果性轨迹能否有效捕捉非线性混沌系统中的输入-输出依赖关系，以实现强化学习中的信用分配？
• RQ4 系统在学习过程中在多大程度上保持混沌动力学？这种特性如何影响行为的适应性？
• RQ5 基于因果性轨迹与TD误差的确定性学习规则，在复杂导航任务中是否能优于传统的随机探索方法？

主要发现

• 机器人在20×20的环境中成功学会抵达目标并避开障碍物，表明该方法能够有效实现复杂环境中的导航。
• 到达目标的平均步数随训练轮次减少，表明学习有效且策略性能持续提升。
• 随着学习进行，探索行为自动减少，表明系统已内化理性行为，无需依赖衰减的探索率。
• 演员网络表现出非均匀的神经元激活，许多神经元饱和在±0.5，表明存在强烈且结构化的动力学。
• 李雅普诺夫指数保持为正值（超过0.0），证实系统在整个学习过程中仍保持混沌特性，支持持续的探索与适应能力。
• 当环境条件发生变化时，李雅普诺夫指数再次上升，表明混沌动力学被重新激活，可能有助于适应新情境。