[Paper Review] Quantifying multivariate redundancy with maximum entropy decompositions of mutual information

Daniel Chicharro|arXiv (Cornell University)|Aug 13, 2017

Computational Drug Discovery Methods65 references3 citations

TL;DR

This paper introduces a maximum entropy-based framework to quantify multivariate redundancy in information decomposition, using rooted tree-structured constraints to decompose mutual information into nonnegative, axiomatic components. It generalizes bivariate redundancy measures to multivariate settings, ensuring consistency with the redundancy lattice and enabling precise separation of unique, redundant, and synergistic information contributions.

ABSTRACT

Williams and Beer (2010) proposed a nonnegative mutual information decomposition, based on the construction of redundancy lattices, which allows separating the information that a set of variables contains about a target variable into nonnegative components interpretable as the unique information of some variables not provided by others as well as redundant and synergistic components. However, the definition of multivariate measures of redundancy that comply with nonnegativity and conform to certain axioms that capture conceptually desirable properties of redundancy has proven to be elusive. We here present a procedure to determine nonnegative multivariate redundancy measures, within the maximum entropy framework. In particular, we generalize existing bivariate maximum entropy measures of redundancy and unique information, defining measures of the redundant information that a group of variables has about a target, and of the unique redundant information that a group of variables has about a target that is not redundant with information from another group. The two key ingredients for this approach are: First, the identification of a type of constraints on entropy maximization that allows isolating components of redundancy and unique redundancy by mirroring them to synergy components. Second, the construction of rooted tree-based decompositions of the mutual information, which conform to the axioms of the redundancy lattice by the local implementation at each tree node of binary unfoldings of the information using hierarchically related maximum entropy constraints. Altogether, the proposed measures quantify the different multivariate redundancy contributions of a nonnegative mutual information decomposition consistent with the redundancy lattice.

Motivation & Objective

• To address the long-standing challenge of defining nonnegative, axiomatic multivariate redundancy measures in information decomposition.
• To extend bivariate maximum entropy redundancy measures to multivariate systems while preserving consistency with the redundancy lattice framework.
• To develop a method that isolates redundant and unique information components through entropy maximization under hierarchically structured constraints.
• To ensure the resulting decomposition satisfies key axioms of redundancy, such as the identity axiom, even in the presence of deterministic dependencies.
• To provide a general, scalable procedure for decomposing mutual information in multivariate systems using rooted tree-based unfoldings of information components.

Proposed method

• Uses maximum entropy distributions constrained by specific conditional independence and co-information constraints to isolate redundancy and unique redundancy components.
• Employs rooted tree-based decompositions where each node applies binary unfoldings of information using hierarchically related maximum entropy constraints.
• Generalizes bivariate redundancy measures by extending the constraints to multivariate settings, ensuring nonnegativity and axiomatic compliance.
• Applies induction to prove equivalence between actual and maximum entropy decompositions under the condition that at least one distribution in the family yields zero information for a subset.
• Handles deterministic target-source dependencies by separating stochastic and deterministic components, applying the same measures to the stochastic part.
• Relies on co-information constraints (e.g., $C(X;i;j|k) = 0$) to define redundancy, adjusting for cases where such constraints are infeasible due to deterministic overlaps.

Experimental results

Research questions

• RQ1 How can multivariate redundancy be quantified in a way that satisfies nonnegativity and the axioms of the redundancy lattice?
• RQ2 Can maximum entropy methods be generalized to decompose mutual information into nonnegative, interpretable components for multivariate systems?
• RQ3 What constraints on entropy maximization allow for the isolation of redundancy and unique redundancy components while mirroring them to synergy components?
• RQ4 How do deterministic dependencies between target and source variables affect the validity of co-information constraints in redundancy measures?
• RQ5 To what extent can the proposed framework be extended to multivariate systems using recursive, tree-based decompositions of information components?

Key findings

• The proposed maximum entropy framework successfully produces nonnegative, axiomatic measures of multivariate redundancy that conform to the redundancy lattice structure.
• The method generalizes bivariate redundancy measures to multivariate settings by applying hierarchical constraints through rooted tree decompositions.
• Equality between actual and maximum entropy decompositions is proven under the condition that at least one distribution in the family yields zero information for a subset of sources.
• The framework ensures that redundancy and unique redundancy terms are nonnegative and consistent with the identity axiom, even when sources are partially or fully included in the target.
• For systems with deterministic target-source dependencies, the method remains valid when applied to the stochastic components of redundancy, with deterministic parts treated separately.
• The approach provides a scalable, consistent, and information-theoretically grounded method for decomposing mutual information into unique, redundant, and synergistic contributions in multivariate systems.