[Paper Review] Optimal sequential decision making with probabilistic digital twins

Christian Agrell, Kristina Rognlien Dahl+1 more|arXiv (Cornell University)|Mar 12, 2021

Formal Methods in Verification4 citations

TL;DR

This paper introduces a probabilistic digital twin (PDT) framework for optimal sequential decision making under epistemic uncertainty, formalized using measure-theoretic conditioning and σ-algebras. It proposes a deep reinforcement learning (RL) approach with set-based neural networks to solve high-dimensional, partially observable decision problems, demonstrating superior performance in optimal information gathering for failure probability estimation—achieving 94% success rate versus 60% for baseline policies.

ABSTRACT

Digital twins are emerging in many industries, typically consisting of simulation models and data associated with a specific physical system. One of the main reasons for developing a digital twin, is to enable the simulation of possible consequences of a given action, without the need to interfere with the physical system itself. Physical systems of interest, and the environments they operate in, do not always behave deterministically. Moreover, information about the system and its environment is typically incomplete or imperfect. Probabilistic representations of systems and environments may therefore be called for, especially to support decisions in application areas where actions may have severe consequences. In this paper we introduce the probabilistic digital twin (PDT). We will start by discussing how epistemic uncertainty can be treated using measure theory, by modelling epistemic information via $σ$-algebras. Based on this, we give a formal definition of how epistemic uncertainty can be updated in a PDT. We then study the problem of optimal sequential decision making. That is, we consider the case where the outcome of each decision may inform the next. Within the PDT framework, we formulate this optimization problem. We discuss how this problem may be solved (at least in theory) via the maximum principle method or the dynamic programming principle. However, due to the curse of dimensionality, these methods are often not tractable in practice. To mend this, we propose a generic approximate solution using deep reinforcement learning together with neural networks defined on sets. We illustrate the method on a practical problem, considering optimal information gathering for the estimation of a failure probability.

Motivation & Objective

• To formalize epistemic uncertainty in digital twins using measure theory and σ-algebras for proper conditioning.
• To define a probabilistic digital twin (PDT) that supports uncertainty propagation and dynamic updating with new information.
• To formulate optimal sequential decision making as a partially observable Markov decision process (POMDP) within the PDT framework.
• To develop a scalable, approximate solution using deep reinforcement learning for high-dimensional, computationally intractable problems.
• To demonstrate the method on a practical problem: minimizing resource cost in estimating failure probability through optimal experiment selection.

Proposed method

• Formalize epistemic uncertainty using measure-theoretic conditioning via σ-algebras to represent information states.
• Define a probabilistic digital twin (PDT) as a Bayesian model that integrates simulation, data, and uncertainty propagation.
• Model sequential decision making as a POMDP, where actions affect belief states and outcomes are probabilistically observed.
• Propose a deep RL solution using a DQN-based agent with a set-structured neural network to represent value functions over information states.
• Implement experience replay and a target network to stabilize training, enabling stable policy learning in high-dimensional belief spaces.
• Use a reward function that penalizes experiment costs (e.g., -10 for measurement, -0.1 for finite element analysis) to minimize total cost under a 40-experiment limit.

Experimental results

Research questions

• RQ1 How can epistemic uncertainty in digital twins be formally represented and updated using measure-theoretic foundations?
• RQ2 How can a probabilistic digital twin be structured to support optimal sequential decision making under uncertainty?
• RQ3 What are the limitations of traditional dynamic programming and maximum principle methods in high-dimensional PDT problems?
• RQ4 Can deep reinforcement learning provide a scalable, practical solution for optimal information gathering in PDTs?
• RQ5 How does the proposed RL-based policy compare to random and heuristic policies in terms of success rate and cost efficiency?

Key findings

• The deep reinforcement learning agent achieved a 94% success rate in estimating failure probability within 40 experiments, significantly outperforming the random policy (60% success rate).
• The RL-based policy minimized total cost by intelligently selecting between low-cost (FE), medium-cost (lab), and high-cost (measurement) experiments based on uncertainty reduction.
• The use of experience replay and a target network was essential for stable training and convergence of the DQN agent in the high-dimensional belief space.
• The method successfully handles the curse of dimensionality by using a set-based neural network architecture to represent information states efficiently.
• The framework supports risk-aware decision making, with potential extension to time-inconsistent risk measures like CVaR via maximum principle-based methods.
• The proposed approach enables real-time, data-driven decision support in safety-critical systems by balancing information gain and resource cost.