[论文解读] Automatic Curriculum Generation for Learning Adaptation in Networking

Zhengxu Xia, Yajie Zhou+2 more|arXiv (Cornell University)|Feb 12, 2022

Software-Defined Networks and 5G被引用 5

一句话总结

Genet 是一种新颖的课程学习框架，通过优先选择当前强化学习（RL）策略相较于基于规则的基线表现显著较差的网络环境，自动为深度强化学习（RL）在网络领域的训练生成课程。通过迭代识别并针对这些高差距环境进行再训练，Genet 在自适应视频流、拥塞控制和负载均衡任务中均提升了最终性能与泛化能力，优于标准RL训练和传统基线方法。

ABSTRACT

As deep reinforcement learning (RL) showcases its strengths in networking and systems, its pitfalls also come to the public's attention--when trained to handle a wide range of network workloads and previously unseen deployment environments, RL policies often manifest suboptimal performance and poor generalizability. To tackle these problems, we present Genet, a new training framework for learning better RL-based network adaptation algorithms. Genet is built on the concept of curriculum learning, which has proved effective against similar issues in other domains where RL is extensively employed. At a high level, curriculum learning gradually presents more difficult environments to the training, rather than choosing them randomly, so that the current RL model can make meaningful progress in training. However, applying curriculum learning in networking is challenging because it remains unknown how to measure the "difficulty" of a network environment. Instead of relying on handcrafted heuristics to determine the environment's difficulty level, our insight is to utilize traditional rule-based (non-RL) baselines: If the current RL model performs significantly worse in a network environment than the baselines, then the model's potential to improve when further trained in this environment is substantial. Therefore, Genet automatically searches for the environments where the current model falls significantly behind a traditional baseline scheme and iteratively promotes these environments as the training progresses. Through evaluating Genet on three use cases--adaptive video streaming, congestion control, and load balancing, we show that Genet produces RL policies which outperform both regularly trained RL policies and traditional baselines in each context, not only under synthetic workloads but also in real environments.

研究动机与目标

• 解决基于RL的网络自适应算法在多样化或狭窄网络分布上进行训练时存在的泛化能力差与性能不佳的问题。
• 克服在无法轻易量化难度的网络环境中识别‘有益’训练环境的挑战。
• 通过系统性地筛选高影响力环境构成课程，提升RL训练效率与最终性能。
• 通过有针对性的、数据驱动的课程生成，实现对未见网络条件的更好泛化能力。
• 提供一种可扩展、自动化的网络课程学习框架，利用现有的基于规则的基线作为性能基准。

提出的方法

• Genet 定义了‘与基线的差距’指标，即在特定网络环境中，当前RL策略与基于规则的基线之间的性能差异。
• 它自动识别出‘与基线差距’较大的环境，作为再训练的高潜力候选环境。
• 该框架迭代地将这些高差距环境纳入训练课程，逐步提升训练难度。
• 它将基于规则的基线用作最优或近似最优性能的代理，假设与基线差距较大的环境具有最大的改进空间。
• 课程根据RL策略性能的动态演变进行实时更新，确保持续聚焦于最具价值的训练样本。
• 当缺乏基于规则的基线时，Genet 支持回退策略，包括使用具有真实知识的最优解，或将预训练的RL模型作为基线。

实验结果

研究问题

• RQ1 自动课程学习能否提升基于RL的网络自适应算法的性能与泛化能力？
• RQ2 在难度难以直接观测的网络环境中，如何自动识别‘有益’的训练环境？
• RQ3 优先选择与基于规则的基线存在大性能差距的环境进行训练，是否比随机或启发式课程调度更有效？
• RQ4 Genet 能否在多样化的网络工作负载中超越标准RL训练和传统基于规则的基线？
• RQ5 当RL模型的决策粒度较粗时，‘与基线差距’作为课程信号存在哪些局限性？

主要发现

• Genet 显著提升了RL策略在三种网络应用场景下的最终性能：自适应视频流、拥塞控制和负载均衡。
• 在自适应视频流方面，Genet 训练的策略优于标准RL训练和传统ABR算法（如BBA和MPC）。
• 在拥塞控制方面，Genet 生成的课程使策略实现了更高的吞吐量和更低的丢包率，优于未使用课程学习的训练结果。
• 在负载均衡方面，Genet 同时提升了性能与泛化能力，尤其在未见网络条件下表现更优。
• ‘与基线差距’指标是训练潜力的强有效指标，因为差距较大的环境在再训练后始终能带来最大的性能提升。
• 即使RL模型的决策粒度过粗（例如基于监控间隔的策略），Genet 仍能提升性能，但在需要精细适应的场景中，性能提升有限。