[Paper Review] Markov Chains with Maximum Return Time Entropy for Robotic Surveillance

Xiaoming Duan, Mishel George+1 more|arXiv (Cornell University)|Mar 21, 2018

Vehicular Ad Hoc Networks (VANETs)24 references3 citations

TL;DR

This paper proposes maximizing return time entropy in Markov chains over directed graphs with travel times to enhance unpredictability in robotic surveillance. By modeling return time probabilities via a delayed linear system and using truncated entropy approximation with gradient projection, the method achieves superior performance against rational intruders, outperforming MinKemeny and MaxEntropyRate chains in both grid and real-world urban topologies when attack durations are small to moderate.

ABSTRACT

Motivated by robotic surveillance applications, this paper studies the novel problem of maximizing the return time entropy of a Markov chain, subject to a graph topology with travel times and stationary distribution. The return time entropy is the weighted average, over all graph nodes, of the entropy of the first return times of the Markov chain; this objective function is a function series that does not admit in general a closed form. The paper features theoretical and computational contributions. First, we obtain a discrete-time delayed linear system for the return time probability distribution and establish its convergence properties. We show that the objective function is continuous over a compact set and therefore admits a global maximum; a unique globally-optimal solution is known only for complete graphs with unitary travel times. We then establish upper and lower bounds between the return time entropy and the well-known entropy rate of the Markov chain. To compute the optimal Markov chain numerically, we establish the asymptotic equality between entropy, conditional entropy and truncated entropy, and propose an iteration to compute the gradient of the truncated entropy. Finally, we apply these results to the robotic surveillance problem. Our numerical results show that, for a model of rational intruder over prototypical graph topologies and test cases, the maximum return time entropy chain performs better than several existing Markov chains.

Motivation & Objective

• To design stochastic surveillance strategies that maximize unpredictability by optimizing return time entropy in Markov chains on directed graphs with travel times.
• To address the challenge of intruders who exploit predictable visit patterns by making return times maximally uncertain.
• To develop a numerically tractable optimization framework for return time entropy under stationary distribution and graph topology constraints.
• To evaluate the performance of the proposed strategy against rational intruders who time attacks based on observed inter-visit intervals.
• To establish theoretical bounds and convergence properties for the return time entropy objective function.

Proposed method

• Modeling return time probabilities using a discrete-time delayed linear system that incorporates integer-valued travel times on the graph.
• Proving continuity and compactness of the objective function, ensuring existence of a global maximum for the return time entropy optimization.
• Establishing asymptotic equivalence between truncated entropy, conditional entropy, and the original return time entropy for numerical computation.
• Deriving the gradient of the truncated return time entropy to enable gradient projection-based optimization.
• Applying the method to robotic surveillance by comparing the MaxReturnEntropy chain against MinKemeny and MaxEntropyRate chains in simulation.
• Using a rational intruder model that attacks after observing the surveillance agent's departure and not returning for a set number of steps.

Experimental results

Research questions

• RQ1 Can return time entropy be maximized in a Markov chain over a directed graph with non-unitary travel times, and is the optimization problem well-posed?
• RQ2 How does return time entropy relate to the well-known entropy rate of Markov chains, and what are the theoretical bounds between them?
• RQ3 What is the performance of the MaxReturnEntropy chain against a rational intruder who plans attacks based on observed inter-visit times?
• RQ4 Can the truncated return time entropy serve as a reliable approximation for the true objective in numerical optimization?
• RQ5 How does the MaxReturnEntropy strategy compare to existing Markov chains like MinKemeny and MaxEntropyRate in real-world and synthetic topologies?

Key findings

• The return time entropy is lower bounded by the entropy rate and upper bounded by n times the entropy rate, where n is the number of nodes.
• For complete graphs with unitary travel times, the optimal solution is analytically derived using the maximum entropy principle.
• The MaxReturnEntropy chain outperforms the MinKemeny and MaxEntropyRate chains in capturing rational intruders on a 4×4 grid when attack durations are small or moderate.
• On the San Francisco crime map, the MaxReturnEntropy chain also outperforms the MinKemeny chain under similar attack conditions.
• The truncated return time entropy is asymptotically equivalent to both the original objective and the conditional return time entropy, enabling reliable numerical optimization.
• The gradient of the truncated return time entropy is derived and used in a gradient projection method to compute the optimal Markov chain numerically.