[论文解读] WINFlowNets: Warm-up Integrated Networks Training of Generative Flow Networks for Robotics and Machine Fault Adaptation

Zahin Sufiyan, Shadan Golestan+3 more|arXiv (Cornell University)|Mar 18, 2026

Reinforcement Learning in Robotics被引用 0

一句话总结

WINFlowNets 通过暖身+双训练框架在共享回放缓冲区下联合训练流量与检索网络，使得在动态且易出错的机器人任务中能够持续适应，并在平均回报与稳定性方面优于 CFlowNets 与常规 RL 基线。

ABSTRACT

Generative Flow Networks for continuous scenarios (CFlowNets) have shown promise in solving sequential decision-making tasks by learning stochastic policies using a flow and a retrieval network. Despite their demonstrated efficiency compared to state-of-the-art Reinforcement Learning (RL) algorithms, their practical application in robotic control tasks is constrained by the reliance on pre-training the retrieval network. This dependency poses challenges in dynamic robotic environments, where pre-training data may not be readily available or representative of the current environment. This paper introduces WINFlowNets, a novel CFlowNets framework that enables the co-training of flow and retrieval networks. WINFlowNets begins with a warm-up phase for the retrieval network to bootstrap its policy, followed by a shared training architecture and a shared replay buffer for co-training both networks. Experiments in simulated robotic environments demonstrate that WINFlowNets surpasses CFlowNets and state-of-the-art RL algorithms in terms of average reward and training stability. Furthermore, WINFlowNets exhibits strong adaptive capability in fault environments, making it suitable for tasks that demand quick adaptation with limited sample data. These findings highlight WINFlowNets' potential for deployment in dynamic and malfunction-prone robotic systems, where traditional pre-training or sample inefficient data collection may be impractical.

研究动机与目标

• 为动态环境与故障条件下的连续机器人控制，激发鲁棒的序列决策能力。
• 通过共同训练流量与检索组件，消除对预训练检索网络的依赖。
• 提出一个两阶段训练方案（暖身+双训练）并使用共享回放缓冲区，以实现持续适应。
• 在模拟机器人故障中展示相比 CFlowNets 和传统 RL 算法的平均回报和稳定性提升。

提出的方法

• 引入 WINFlowNets：一个具备共享回放缓冲区的两网络 GFlowNet 框架。
• 暖身阶段利用观测到的转移来训练检索网络 Gϕ，使其预测前序状态。
• 双训练阶段联合更新流网络 Fθ 和 Gϕ，使用流入/流出估计以及共享缓冲区。
• 流量匹配通过 Fθ 与 Gϕ 的 log-sum-exp 形式近似的流入 f+(s) 与流出 f−(s) 进行。
• 公式 2 表示基于采样动作与奖励的连续流量匹配损失。
• 训练不进行 Gϕ 的预训练，从而使其能够适应分布外和故障场景。

(a) Directed acyclic graph representation of the decision-making process in GFlowNet.

实验结果

研究问题

• RQ1 在不进行预训练的情况下对流量与检索网络进行联合训练，是否能提升对动态和易出错环境的适应性？
• RQ2 暖身+双训练的 WINFlowNets 框架在正常与故障的机器人任务中是否优于标准 CFlowNets 与 RL 基线？
• RQ3 共享回放缓冲区与分离缓冲区相比，对学习稳定性和适应速度有何影响？

主要发现

Model	Final Performance	Sample Efficiency
SAC	-7.89 ± 0.16	0.67
PPO	-9.50 ± 0.37	3.39
DDPG	-9.55 ± 0.44	5.20
CFlowNets	-3.70 ± 0.05	0.10
WINFlowNets	-2.39 ± 0.17	0.72

• WINFlowNets 在普通 Reacher-v2 环境中平均回报方面超过 CFlowNets 和 RL 基线（PPO、SAC、DDPG）。
• 在故障情景中，WINFlowNets 相对于 CFlowNets 与 SAC 提高了最终性能，表现出更好的故障自适应性。
• 带有共享回放缓冲区的暖身+双训练结构相比缺少任一组件的变体，具有更稳定且更优的渐近性能。
• WINFlowNets 需要更多训练样本以达到其渐近性能，但由于持续适应性，最终策略质量更高。

(b) An overview of our proposed decision-making framework.