[论文解读] A Survey on Model-based Reinforcement Learning

Fan-Ming Luo, Xu Tian+4 more|arXiv (Cornell University)|Jun 19, 2022

Reinforcement Learning in Robotics被引用 26

一句话总结

本文综述基于模型的强化学习（MBRL），聚焦环境模型的学习与在深度强化学习中的应用，分析模型与策略之间的差异，并介绍相关RL范式的进展及在现实世界中的适用性。

ABSTRACT

Reinforcement learning (RL) solves sequential decision-making problems via a trial-and-error process interacting with the environment. While RL achieves outstanding success in playing complex video games that allow huge trial-and-error, making errors is always undesired in the real world. To improve the sample efficiency and thus reduce the errors, model-based reinforcement learning (MBRL) is believed to be a promising direction, which builds environment models in which the trial-and-errors can take place without real costs. In this survey, we take a review of MBRL with a focus on the recent progress in deep RL. For non-tabular environments, there is always a generalization error between the learned environment model and the real environment. As such, it is of great importance to analyze the discrepancy between policy training in the environment model and that in the real environment, which in turn guides the algorithm design for better model learning, model usage, and policy training. Besides, we also discuss the recent advances of model-based techniques in other forms of RL, including offline RL, goal-conditioned RL, multi-agent RL, and meta-RL. Moreover, we discuss the applicability and advantages of MBRL in real-world tasks. Finally, we end this survey by discussing the promising prospects for the future development of MBRL. We think that MBRL has great potential and advantages in real-world applications that were overlooked, and we hope this survey could attract more research on MBRL.

研究动机与目标

• Explain why MBRL can improve sample efficiency over model-free methods in DRL.
• Review classical and modern methods for learning environment models (tabular and function-approximation).
• Discuss how models are used (planning, rollout, and integration with various RL forms) and analyze policy/value discrepancies.
• Summarize recent advances in model-based techniques for offline, goal-conditioned, multi-agent, and meta-RL.
• Highlight real-world applicability and future directions for MBRL.

提出的方法

• Describe tabular and neural network-based model learning approaches for MDPs, including M, R learning and likelihood-based objectives.
• Discuss prediction loss (one-step) and probabilistic modeling to capture aleatoric uncertainty.
• Present simulation lemmas that bound value evaluation error under model error (Theorem 1 and Theorem 2).
• Introduce distribution matching (JS divergence, Wasserstein) for long-horizon effect mitigation (Simulation Lemma III).
• Explore robust learning via policy-distrbution considerations and CVaR for outlier policies.
• Survey model variants (multistep and backward models) and representation learning for complex environments.

实验结果

研究问题

• RQ1 How does model approximation affect policy/value performance when trained in a learned MDP versus the real environment?
• RQ2 What theoretical bounds exist for value evaluation error given model and reward errors?
• RQ3 How do distribution-matching and adversarial techniques influence the quality of learned transition models?
• RQ4 What are effective strategies to reduce compounding errors in long rollouts and in partially observable or high-dimensional tasks?
• RQ5 How can MBRL be integrated with offline, goal-conditioned, multi-agent, and meta-RL frameworks?

主要发现

• Model errors propagate to value errors with horizon-dependent (often quadratic) growth under certain conditions.
• Probabilistic/learned models can capture aleatoric uncertainty and improve robustness compared to deterministic one-step predictors.
• Simulation lemmas provide bounds on policy evaluation error linking model error to performance loss; shorter rollouts mitigate compounding error.
• Distribution matching (JS/Wasserstein) can improve long-horizon behavior and reduce sample complexity in some setups.
• Lipschitz-constrained models can bound multi-step prediction errors and control compounding effects.
• Dreamer and related latent-dynamics models show strong performance in vision-based tasks via world-models and latent planning.