[论文解读] Visualizing Critic Match Loss Landscapes for Interpretation of Online Reinforcement Learning Control Algorithms

Jingyi Liu, Jian Guo+1 more|arXiv (Cornell University)|Mar 15, 2026

Adaptive Dynamic Programming Control被引用 0

一句话总结

本文提出一种通过在低维子空间构建 critic 匹配损失景观来可视化并量化在线 actor–critic 强化学习中的 critic 学习过程的方法，从而对控制任务中的收敛性与稳定性进行定性与定量解释。

ABSTRACT

Reinforcement learning has proven its power on various occasions. However, its performance is not always guaranteed when system dynamics change. Instead, it largely relies on users' empirical experience. For reinforcement learning algorithms with an actor-critic structure, the critic neural network reflects the approximation and optimization process in the RL algorithm. Analyzing the performance of the critic neural network helps to understand the mechanism of the algorithm. To support systematic interpretation of such algorithms in dynamic control problems, this work proposes a critic match loss landscape visualization method for online reinforcement learning. The method constructs a loss landscape by projecting recorded critic parameter trajectories onto a low-dimensional linear subspace. The critic match loss is evaluated over the projected parameter grid using fixed reference state samples and temporal-difference targets. This yields a three-dimensional loss surface together with a two-dimensional optimization path that characterizes critic learning behavior. To extend analysis beyond visual inspection, quantitative landscape indices and a normalized system performance index are introduced, enabling structured comparison across different training outcomes. The approach is demonstrated using the Action-Dependent Heuristic Dynamic Programming algorithm on cart-pole and spacecraft attitude control tasks. Comparative analyses across projection methods and training stages reveal distinct landscape characteristics associated with stable convergence and unstable learning. The proposed framework enables both qualitative and quantitative interpretation of critic optimization behavior in online reinforcement learning.

研究动机与目标

• 为在线强化学习控制算法的解释提供动机，聚焦 critic 组件。
• 开发一个可视化框架（critic 匹配损失景观）以理解在线训练下 critic 的学习 dynamics。
• 引入定量景观指标，便于跨运行的客观比较。
• 将 critic 景观几何与动态控制问题中的实际系统性能联系起来。

提出的方法

• 通过固定参考数据/TD 目标并以一组 critic 权重的网格来评估 TD 误差来构建 critic 匹配损失。
• 在每次回合结束时记录 critic 权重，并将其投影到通过权重轨迹的 PCA 得到的两个正交方向上。
• 生成一个三维损失景观和一个二维优化路径，以在最终策略周围可视化 critic 学习。
• 引入定量景观指标（尖度、盆地面积、局部各向异性）并对损失进行归一化以便跨运行比较。
• 定义系统性能指标 J_H，以在固定时域内将景观几何与控制性能联系起来。

Figure 1 : Structure of ADHDP

实验结果

研究问题

• RQ1 如何通过固定参考损失景观解释在线 RL 中的 critic 学习？
• RQ2 不同的投影方法和训练阶段是否产生与收敛或不稳定相关的不同景观几何？
• RQ3 定量景观指标是否能预测或解释在线 RL 控制中的收敛/发散？
• RQ4 critic 景观如何与跨任务的实际动态系统性能相关？

主要发现

• 在收敛的倒立摆训练中，critic 损失景观平滑，PCA 方向解释方差分别为 69.9% 和 25.7%（总计 95.6%）。
• 在倒立摆情形中，critic 损失趋向于零，actor 梯度幅值也随之下降，表明收敛到一个次优但稳定的解。
• 在航天器姿态任务中，训练中 critic 损失和 actor 梯度出现尖峰，最终策略导致发散。
• critic 景观方法提供了定性几何视图，并通过引入的指标提供了定量基础，用于比较不同运行并解释训练行为。
• 该框架通过归一化的时域成本 J_H 将景观几何与系统性能联系起来，便于跨任务比较。

(a) State trajectory