[Paper Review] Robust Model-based Reinforcement Learning for Autonomous Greenhouse Control

Wanpeng Zhang, Xiaoyan Cao+4 more|arXiv (Cornell University)|Aug 26, 2021

Greenhouse Technology and Climate Control30 references20 citations

TL;DR

This paper proposes a robust model-based reinforcement learning framework for autonomous greenhouse control that improves sample efficiency and safety. By using an ensemble of environment models for simulation-based policy optimization and a sample dropout module to prioritize worst-case scenarios, the method achieves higher crop yield and retention rates under extreme conditions compared to baseline approaches.

ABSTRACT

Due to the high efficiency and less weather dependency, autonomous greenhouses provide an ideal solution to meet the increasing demand for fresh food. However, managers are faced with some challenges in finding appropriate control strategies for crop growth, since the decision space of the greenhouse control problem is an astronomical number. Therefore, an intelligent closed-loop control framework is highly desired to generate an automatic control policy. As a powerful tool for optimal control, reinforcement learning (RL) algorithms can surpass human beings' decision-making and can also be seamlessly integrated into the closed-loop control framework. However, in complex real-world scenarios such as agricultural automation control, where the interaction with the environment is time-consuming and expensive, the application of RL algorithms encounters two main challenges, i.e., sample efficiency and safety. Although model-based RL methods can greatly mitigate the efficiency problem of greenhouse control, the safety problem has not got too much attention. In this paper, we present a model-based robust RL framework for autonomous greenhouse control to meet the sample efficiency and safety challenges. Specifically, our framework introduces an ensemble of environment models to work as a simulator and assist in policy optimization, thereby addressing the low sample efficiency problem. As for the safety concern, we propose a sample dropout module to focus more on worst-case samples, which can help improve the adaptability of the greenhouse planting policy in extreme cases. Experimental results demonstrate that our approach can learn a more effective greenhouse planting policy with better robustness than existing methods.

Motivation & Objective

• To address the high sample cost and safety risks in real-world reinforcement learning for greenhouse automation.
• To improve sample efficiency in training by leveraging a learned ensemble of environment models for simulated rollouts.
• To enhance policy robustness against environmental perturbations through a sample dropout mechanism that emphasizes worst-case scenarios.
• To develop a closed-loop, data-driven control system that balances crop yield and energy efficiency in real-world greenhouses.
• To validate the framework’s superiority in both standard and anomalous environmental conditions through simulation and ablation studies.

Proposed method

• The framework employs an ensemble of neural network models to simulate the greenhouse environment, enabling efficient policy training via Dyna-style planning.
• Real-world interaction data is used to train the environment model ensemble, which then generates synthetic rollouts to augment the training data.
• A sample dropout module selectively removes high-reward samples during policy optimization to focus learning on low-probability, high-risk scenarios.
• The dropout mechanism uses a hyperparameter $ p $ to control the proportion of samples discarded, with $ p=0.8 $ found optimal in experiments.
• The policy is optimized using a model-based RL algorithm that balances exploration in the simulated environment with real-world data collection.
• Hyperparameter sensitivity is evaluated across standard and disturbed environments to assess robustness and performance stability.

Experimental results

Research questions

• RQ1 How can model-based reinforcement learning improve sample efficiency in real-world greenhouse control with costly real-world interactions?
• RQ2 To what extent does a sample dropout mechanism enhance policy robustness under extreme environmental conditions?
• RQ3 What is the optimal hyperparameter setting for the sample dropout module in terms of balancing performance and robustness?
• RQ4 How does the proposed framework compare to baseline RL methods in terms of crop yield and retention rate under normal and anomalous conditions?
• RQ5 Can the ensemble model-based approach generalize across diverse environmental perturbations such as extreme temperatures and low solar radiation?

Key findings

• The proposed method achieved a fresh weight of 45.24g and a retention rate of 85.35% under high air temperature conditions (35–40°C), significantly outperforming the baseline.
• Under low-temperature conditions (−2–10°C), the method achieved 38.49g fresh weight and 72.62% retention rate, demonstrating improved resilience.
• With high humidity (90%), the method produced 39.30g fresh weight and 74.16% retention rate, showing consistent performance under stress.
• In the absence of solar radiation (Iglob=0), the method achieved 43.71g fresh weight and 82.47% retention rate, indicating strong adaptability to low-light extremes.
• The optimal dropout hyperparameter $ p=0.8 $ was identified, as performance and robustness dropped significantly when $ p<0.8 $.
• The framework demonstrated superior robustness across four disturbed environments (varying temperature and CO₂), with the highest average net profit at $ p=0.8 $.