[논문 리뷰] COMBO: Conservative Offline Model-Based Policy Optimization

Tianhe Yu, Aviral Kumar외 4명|arXiv (Cornell University)|2021. 02. 16.

Reinforcement Learning in Robotics참고 문헌 59인용 수 52

한 줄 요약

COMBO는 오프라인 데이터와 모델 생성 데이터를 사용하여 불확실성 추정 없이 보수적인 Q-함수를 최적화하고, 명확한 하한을 제공하며 오프라인 RL에서 강한 일반화를 보인다.

ABSTRACT

Model-based algorithms, which learn a dynamics model from logged experience and perform some sort of pessimistic planning under the learned model, have emerged as a promising paradigm for offline reinforcement learning (offline RL). However, practical variants of such model-based algorithms rely on explicit uncertainty quantification for incorporating pessimism. Uncertainty estimation with complex models, such as deep neural networks, can be difficult and unreliable. We overcome this limitation by developing a new model-based offline RL algorithm, COMBO, that regularizes the value function on out-of-support state-action tuples generated via rollouts under the learned model. This results in a conservative estimate of the value function for out-of-support state-action tuples, without requiring explicit uncertainty estimation. We theoretically show that our method optimizes a lower bound on the true policy value, that this bound is tighter than that of prior methods, and our approach satisfies a policy improvement guarantee in the offline setting. Through experiments, we find that COMBO consistently performs as well or better as compared to prior offline model-free and model-based methods on widely studied offline RL benchmarks, including image-based tasks.

연구 동기 및 목표

• Address offline RL distribution shift without relying on uncertain model error estimates.
• Leverage both offline data and synthetic model rollouts to train a conservative value function.
• Provide theoretical guarantees of policy improvement and a lower bound on true returns.
• Demonstrate strong performance on tasks requiring generalization and image-based offline RL benchmarks.

제안 방법

• Train a probabilistic dynamics model on the offline dataset.
• Use a conservative Q-function update (Eq. 2) that penalizes out-of-support model rollouts via a distribution-balanced Bellman backup.
• Interleave offline data and model-generated data in Bellman backups (Dyna-like augmentation).
• Define sampling distributions ρ(s,a) and d_f(s,a) to push down Q-values on out-of-support tuples and push up on dataset-supported tuples.
• Improve the policy using a conservative critic via Eq. 3, ensuring safe policy improvement over the behavior policy.
• Provide offline hyperparameter tuning by monitoring the regularization objective in Eq. 2 to avoid online rollouts.

실험 결과

연구 질문

• RQ1 Can COMBO achieve reliable policy improvement in offline RL without explicit uncertainty quantification?
• RQ2 How does COMBO compare to prior offline model-free and model-based methods on generalization tasks and standard benchmarks (including image-based tasks)?
• RQ3 Does incorporating model rollouts with a conservative critic improve out-of-distribution generalization without oracle uncertainty?
• RQ4 What theoretical guarantees does COMBO provide regarding lower bounds on returns and safe policy improvement?

주요 결과

• COMBO outperforms MOPO, MOReL, and CQL on tasks requiring out-of-distribution generalization (halfcheetah-jump and sawyer-door-close).
• On ant-angle, COMBO improves by ~8% over MOPO, ~4% over MOReL, and ~12% over CQL.
• COMBO matches or exceeds prior offline RL methods on standard benchmarks and achieves the highest score in 9 of 12 MuJoCo domains from D4RL.
• Uncertainty estimation with deep nets is unreliable in offline MB RL; COMBO avoids this by not requiring an uncertainty oracle.
• Theoretical results show COMBO learns a Q-function that lower-bounds the true Q-function and provides safe policy improvement guarantees