[Paper Review] An LLM-Based Digital Twin for Optimizing Human-in-the Loop Systems

Hanqing Yang, Marie Siew+1 more|arXiv (Cornell University)|Mar 25, 2024

Digital Transformation in Industry4 citations

TL;DR

This paper proposes an LLM-based digital twin that simulates diverse population groups' thermal preferences and behaviors in a mall to train a reinforcement learning (RL) agent for adaptive HVAC control. Using LLM-powered agents to generate synthetic human feedback, the AitL-RL framework outperforms fixed set-point control by balancing energy savings and occupant comfort through personalized, distributed temperature policies in real-time simulations.

ABSTRACT

The increasing prevalence of Cyber-Physical Systems and the Internet of Things (CPS-IoT) applications and Foundation Models are enabling new applications that leverage real-time control of the environment. For example, real-time control of Heating, Ventilation and Air-Conditioning (HVAC) systems can reduce its usage when not needed for the comfort of human occupants, hence reducing energy consumption. Collecting real-time feedback on human preferences in such human-in-the-loop (HITL) systems, however, is difficult in practice. We propose the use of large language models (LLMs) to deal with the challenges of dynamic environments and difficult-to-obtain data in CPS optimization. In this paper, we present a case study that employs LLM agents to mimic the behaviors and thermal preferences of various population groups (e.g. young families, the elderly) in a shopping mall. The aggregated thermal preferences are integrated into an agent-in-the-loop based reinforcement learning algorithm AitL-RL, which employs the LLM as a dynamic simulation of the physical environment to learn how to balance between energy savings and occupant comfort. Our results show that LLMs are capable of simulating complex population movements within large open spaces. Besides, AitL-RL demonstrates superior performance compared to the popular existing policy of set point control, suggesting that adaptive and personalized decision-making is critical for efficient optimization in CPS-IoT applications. Through this case study, we demonstrate the potential of integrating advanced Foundation Models like LLMs into CPS-IoT to enhance system adaptability and efficiency. The project's code can be found on our GitHub repository.

Motivation & Objective

• To address the challenge of collecting real-time human feedback in human-in-the-loop (HITL) cyber-physical systems (CPS-IoT), especially in public buildings like malls.
• To develop a scalable, privacy-preserving method for simulating complex human behaviors and preferences using large language models (LLMs).
• To optimize energy efficiency and occupant comfort in HVAC systems through a digital twin-driven reinforcement learning framework.
• To evaluate whether LLM-generated data can effectively train RL policies that generalize to real-world, dynamic environments.
• To compare the performance of centralized versus distributed control policies in balancing energy cost and user comfort.

Proposed method

• LLM-powered autonomous agents simulate distinct population groups (e.g., young families, elderly couples) with unique mobility patterns and thermal preferences in a mall environment.
• GPT-3.5 and GPT-4 are used to generate synthetic user arrivals, movements, and comfort feedback across stores in each simulation iteration, with only inference (no fine-tuning) to minimize computational cost.
• The simulated data is used to pre-train an AitL-RL (Agent-in-the-Loop Reinforcement Learning) algorithm that learns to balance energy cost and user comfort via a weighted objective function.
• The RL policy is trained offline using the LLM-generated data and then evaluated in an online scenario with real-time control and feedback on new, unseen days.
• Two control strategies are compared: centralized (uniform temperature) and distributed (store-specific temperature control), with policy performance measured using a balanced score combining comfort and energy cost.
• The framework employs a multi-objective loss function with adjustable weights for comfort (wc) and energy cost (we), enabling trade-off analysis across different operational priorities.

Figure 1: The LLM-based Digital Twin Agent in the Loop Distributed Control (AitL-RL) Pipeline. The LLM-based digital twin simulates population behavior in the mall across the day, with multiple population groups such as "teen shoppers". Based on the simulation, user preferences are aggregated and in

Experimental results

Research questions

• RQ1 Can LLM-powered agents accurately simulate diverse human behaviors and thermal preferences in large, open public spaces like shopping malls?
• RQ2 Can synthetic data generated by LLMs effectively train a reinforcement learning policy for real-time HVAC control in human-in-the-loop systems?
• RQ3 How does distributed temperature control compare to centralized control in balancing energy savings and occupant comfort?
• RQ4 What is the impact of different objective function weights (comfort vs. energy cost) on the resulting control policy and system performance?
• RQ5 Can the AitL-RL framework generalize to unseen operational scenarios and outperform standard set-point control policies?

Key findings

• The AitL-RL framework significantly outperforms the fixed set-point control policy (25°C) in balancing energy cost and user comfort across all evaluated metrics.
• Distributed control policies, which tailor temperatures per store based on occupancy and group preferences, outperform centralized policies due to better personalization and responsiveness.
• The balanced weights approach (with both comfort and energy cost objectives) achieved the highest overall system score, demonstrating the importance of multi-objective optimization.
• The LLM-based digital twin successfully simulated complex human behaviors, including group-specific mobility and thermal preferences, across multiple stores in a large open space.
• In energy-focused settings (wc=0), the policy raised temperatures toward ambient (30°C), significantly reducing energy use, while in comfort-focused settings (we=0), it lowered temperatures to prioritize occupant satisfaction.
• The RL policy converged effectively across all training configurations, confirming the stability and scalability of the LLM-generated data for policy learning.

Figure 2: The layout of the shopping mall named Happy Mall, whose floor plan was generated by GPT 4 , after minor manual adjustments to better replicate a real mall setting.