[论文解读] How to Train your Quadrotor: A Framework for Consistently Smooth and Responsive Flight Control via Reinforcement Learning

Siddharth Mysore, Bassel Mabsout+2 more|arXiv (Cornell University)|Dec 11, 2020

Reinforcement Learning in Robotics参考文献 42被引用 14

一句话总结

该论文提出了RE+AL，一种强化学习框架，通过重新设计奖励结构和状态表示，提升了四旋翼飞行器控制策略的平滑性与现实世界可迁移性。通过采用乘法奖励组合和受RC遥控器启发的训练信号，RE+AL将电机控制振荡频率从330Hz降低至130Hz，实现了100%可飞行的智能体，并在真实硬件上实现了比调校后的PID控制器更优的跟踪精度与能效表现。

ABSTRACT

We focus on the problem of reliably training Reinforcement Learning (RL) models (agents) for stable low-level control in embedded systems and test our methods on a high-performance, custom-built quadrotor platform. A common but often under-studied problem in developing RL agents for continuous control is that the control policies developed are not always smooth. This lack of smoothness can be a major problem when learning controllers %intended for deployment on real hardware as it can result in control instability and hardware failure. Issues of noisy control are further accentuated when training RL agents in simulation due to simulators ultimately being imperfect representations of reality - what is known as the reality gap. To combat issues of instability in RL agents, we propose a systematic framework, `REinforcement-based transferable Agents through Learning' (RE+AL), for designing simulated training environments which preserve the quality of trained agents when transferred to real platforms. RE+AL is an evolution of the Neuroflight infrastructure detailed in technical reports prepared by members of our research group. Neuroflight is a state-of-the-art framework for training RL agents for low-level attitude control. RE+AL improves and completes Neuroflight by solving a number of important limitations that hindered the deployment of Neuroflight to real hardware. We benchmark RE+AL on the NF1 racing quadrotor developed as part of Neuroflight. We demonstrate that RE+AL significantly mitigates the previously observed issues of smoothness in RL agents. Additionally, RE+AL is shown to consistently train agents that are flight-capable and with minimal degradation in controller quality upon transfer. RE+AL agents also learn to perform better than a tuned PID controller, with better tracking errors, smoother control and reduced power consumption.

研究动机与目标

• 解决基于强化学习的四旋翼飞行器控制中持续存在的不稳定、非平滑策略问题，这些策略在从仿真环境迁移到真实硬件时难以可靠工作。
• 通过设计更贴近真实世界动力学与控制行为的仿真环境，缩小现实差距。
• 开发一种系统化、可重复的训练流程，持续生成可飞行、低振荡的控制器，无需人工调参。
• 证明使用RE+AL训练的强化学习智能体在真实世界飞行性能指标（如跟踪误差与功耗）上可超越经典PID控制器。

提出的方法

• 设计一种乘法奖励组合，将进展、平滑性与控制努力惩罚相结合，以降低训练方差并提升策略一致性。
• 重构状态空间，使其更准确反映真实的RC遥控输入，提升策略与飞行员式控制行为的一致性。
• 构建模拟真实世界RC指令的训练信号，增强仿真与真实控制动力学之间的保真度。
• 在训练过程中引入早停机制，防止模型过度拟合仿真环境，保持策略的可迁移性。
• 使用SAC与PPO算法，结合新的奖励与状态设计，在NF1四旋翼平台训练智能体。
• 实现端到端的完整流水线，集成仿真、训练与固件编译，实现直接部署至嵌入式硬件。

实验结果

研究问题

• RQ1 重新设计的奖励结构是否能显著提升基于强化学习的四旋翼控制器在仿真到真实硬件迁移过程中的平滑性与可迁移性？
• RQ2 乘法奖励组合是否能降低训练方差，并在多种强化学习算法上实现更一致的策略学习？
• RQ3 状态空间与动作表示设计在多大程度上能提升仿真控制行为与真实世界RC飞行员输入之间的对齐？
• RQ4 使用该框架训练的强化学习智能体是否能在真实飞行性能上超越经过调校的经典PID控制器？
• RQ5 在仿真中进行更长的训练是否会降低迁移性能？如果是，早停机制在多大程度上可缓解此问题？

主要发现

• RE+AL在真实NF1四旋翼飞行器上实现了100%可飞行的智能体，而先前的Neuroflight基线方法仅30个中有1个可飞行。
• 使用RE+AL训练的电机控制信号的峰值振荡频率降低至130Hz，显著低于基线方法的330Hz。
• RE+AL智能体的平均跟踪误差为4.2 deg/s，展示了在真实飞行中出色的控制精度。
• 由于新奖励设计带来的更快收敛，训练时间缩短了10倍，从近9小时降至50分钟以内。
• RE+AL智能体在跟踪误差与功耗方面均优于调校后的PID控制器，且控制信号更平滑。
• 乘法奖励组合一致地降低了训练方差，并有效避免了局部极小值，尤其在Acrobot等复杂环境中表现显著。