[Paper Review] Transfer-Entropy-Regularized Markov Decision Processes

Takashi Tanaka, Henrik Sandberg+1 more|arXiv (Cornell University)|Aug 30, 2017

Advanced Thermodynamics and Statistical Mechanics52 references4 citations

TL;DR

This paper proposes a transfer-entropy-regularized Markov decision process (TERMDP) that minimizes a trade-off between classical state-dependent cost and information flow from states to actions. It develops an iterative forward-backward algorithm, inspired by the Arimoto-Blahut algorithm, which converges to stationary points of the nonconvex TERMDP, enabling optimal policy design under information constraints in control and thermodynamic systems.

ABSTRACT

We consider the framework of transfer-entropy-regularized Markov Decision Process (TERMDP) in which the weighted sum of the classical state-dependent cost and the transfer entropy from the state random process to the control random process is minimized. Although TERMDPs are generally formulated as nonconvex optimization problems, we derive an analytical necessary optimality condition expressed as a finite set of nonlinear equations, based on which an iterative forward-backward computational procedure similar to the Arimoto-Blahut algorithm is proposed. It is shown that every limit point of the sequence generated by the proposed algorithm is a stationary point of the TERMDP. Applications of TERMDPs are discussed in the context of networked control systems theory and non-equilibrium thermodynamics. The proposed algorithm is applied to an information-constrained maze navigation problem, whereby we study how the price of information qualitatively alters the optimal decision polices.

Motivation & Objective

• To formalize a novel optimal control framework that penalizes information flow from states to actions using transfer entropy.
• To derive a necessary optimality condition for TERMDP as a finite set of nonlinear equations.
• To develop a computationally efficient iterative algorithm for solving TERMDP, with convergence guarantees to stationary points.
• To demonstrate the framework's applicability in networked control systems and non-equilibrium thermodynamics.
• To show how information cost alters optimal decision policies in constrained environments like maze navigation.

Proposed method

• Formulates TERMDP as a nonconvex optimization problem minimizing the sum of state-dependent cost and transfer entropy from state to control process.
• Derives a necessary optimality condition expressed as a system of nonlinear equations involving conditional probabilities and Lagrange multipliers.
• Proposes a forward-backward iterative algorithm analogous to the Arimoto-Blahut algorithm, updating control policies and dual variables in alternating steps.
• Introduces a projection operator π that maps general policies to a restricted class with finite memory, preserving optimality under certain conditions.
• Establishes convergence by proving that every limit point of the algorithm sequence is a stationary point of the TERMDP.
• Validates the algorithm on an information-constrained maze navigation task to study policy shifts under varying information costs.

Experimental results

Research questions

• RQ1 How can information-theoretic constraints on state-to-action information flow be integrated into Markov decision processes to model real-world decision-making under limited sensing and communication?
• RQ2 What analytical conditions characterize optimal policies in TERMDPs, and how can they be solved numerically despite the nonconvexity of the problem?
• RQ3 How does the price of information influence the structure and performance of optimal control policies in constrained environments?
• RQ4 Can the proposed algorithm reliably compute stationary points of TERMDPs, and what convergence guarantees does it provide?
• RQ5 To what extent does TERMDP capture fundamental performance limits in networked control systems and non-equilibrium thermodynamics?

Key findings

• The proposed forward-backward algorithm converges to stationary points of the TERMDP, with every limit point satisfying the derived optimality conditions.
• The algorithm is a generalization of the Arimoto-Blahut algorithm for rate-distortion problems, adapted to transfer entropy minimization.
• The necessary optimality condition is derived as a finite set of nonlinear equations involving conditional probabilities and Lagrange multipliers.
• The algorithm's convergence is established via a projection operator π that preserves optimality within a restricted policy class.
• In a maze navigation task, increasing the information cost leads to simpler, less state-dependent policies, demonstrating a qualitative shift in decision-making behavior.
• The framework provides a physically and information-theoretically grounded alternative to prior models that use mutual information or KL divergence as cost functions.