[Paper Review] Directed Information, Causal Estimation, and Communication in Continuous Time

Tsachy Weissman, Young-Han Kim+1 more|arXiv (Cornell University)|Sep 2, 2011

Wireless Communication Security Techniques39 references4 citations

TL;DR

This paper introduces a rigorous notion of directed information for continuous-time stochastic processes by taking an infimum over time partitions, generalizing discrete-time directed information to continuous time. It establishes that Duncan’s classical relationship between causal estimation error and mutual information extends to feedback scenarios when replaced by directed information, and uses this to characterize feedback capacity for continuous-time Gaussian and Poisson channels.

ABSTRACT

A notion of directed information between two continuous-time processes is proposed. A key component in the definition is taking an infimum over all possible partitions of the time interval, which plays a role no less significant than the supremum over "space" partitions inherent in the definition of mutual information. Properties and operational interpretations in estimation and communication are then established for the proposed notion of directed information. For the continuous-time additive white Gaussian noise channel, it is shown that Duncan's classical relationship between causal estimation and information continues to hold in the presence of feedback upon replacing mutual information by directed information. A parallel result is established for the Poisson channel. The utility of this relationship is then demonstrated in computing the directed information rate between the input and output processes of a continuous-time Poisson channel with feedback, where the channel input process is constrained to be constant between events at the channel output. Finally, the capacity of a wide class of continuous-time channels with feedback is established via directed information, characterizing the fundamental limit on reliable communication.

Motivation & Objective

• To formalize a notion of directed information between continuous-time stochastic processes that captures causal information flow.
• To extend classical estimation-theoretic results—specifically Duncan’s theorem—into continuous time with feedback.
• To characterize the feedback capacity of a wide class of continuous-time channels using directed information.
• To demonstrate the utility of directed information in computing the directed information rate for a Poisson channel with input constraints.

Proposed method

• Define continuous-time directed information as the infimum over all finite time partitions of discrete-time directed information on subintervals.
• Use the limit of discrete-time directed information over increasingly fine partitions to define the continuous-time version.
• Leverage Birkhoff’s ergodic theorem and block-ergodicity to establish achievability of rates below the directed information rate.
• Construct a discrete-time channel model from continuous-time processes by sampling over small time intervals and using feedback delay to model causality.
• Apply the data processing inequality and properties of conditional mutual information to relate the discrete-time channel capacity to the continuous-time directed information.
• Use the fact that the noise process is stationary and block-ergodic to ensure ergodicity of the induced discrete-time channel.

Experimental results

Research questions

• RQ1 How can directed information be rigorously defined for continuous-time stochastic processes?
• RQ2 Does Duncan’s theorem relating causal estimation error and mutual information extend to continuous time when feedback is present?
• RQ3 Can the feedback capacity of continuous-time channels be characterized using directed information?
• RQ4 What is the directed information rate for a continuous-time Poisson channel where the input is constant between output events?
• RQ5 How does the presence of feedback alter the relationship between estimation error and information measures in continuous time?

Key findings

• The proposed definition of continuous-time directed information as the infimum over time partitions ensures consistency with discrete-time directed information and inherits its key properties.
• Duncan’s theorem generalizes to continuous time: the minimum causal estimation error in an additive white Gaussian noise channel with feedback is related to the directed information between input and output.
• A similar relationship holds for the Poisson channel, where directed information replaces mutual information in the estimation-error characterization.
• For a continuous-time Poisson channel with input constrained to be constant between output events, the directed information rate is computed explicitly using the generalized Duncan’s theorem.
• The feedback capacity of a wide class of continuous-time channels with stationary ergodic noise is characterized as the supremum of directed information over input distributions.
• The feedback capacity is shown to be equal to the infimum over time partitions of the directed information rate, establishing a continuous-time analog of the discrete-time feedback capacity formula.