[Paper Review] Information Theoretical Estimators Toolbox

Zoltán Szabó|arXiv (Cornell University)|May 8, 2014

Blind Source Separation Techniques16 references104 citations

TL;DR

This paper introduces ITE (Information Theoretical Estimators), a free, open-source, multi-platform Matlab/Octave toolbox that enables nonparametric estimation of diverse information-theoretic measures—such as entropy, mutual information, divergence, association, and distribution kernels—on continuous random variables. Its modular architecture allows seamless combination of estimators, rapid prototyping of new methods, and direct application in optimization problems, with a prototype implementation in independent subspace analysis (ISA) and its extensions.

ABSTRACT

We present ITE (information theoretical estimators) a free and open source, multi-platform, Matlab/Octave toolbox that is capable of estimating many different variants of entropy, mutual information, divergence, association measures, cross quantities, and kernels on distributions. Thanks to its highly modular design, ITE supports additionally (i) the combinations of the estimation techniques, (ii) the easy construction and embedding of novel information theoretical estimators, and (iii) their immediate application in information theoretical optimization problems. ITE also includes a prototype application in a central problem class of signal processing, independent subspace analysis and its extensions.

Motivation & Objective

• Address the lack of comprehensive, extensible, and open-source tools for estimating information-theoretic measures on continuous variables.
• Provide a unified, modular framework that supports the combination of estimation techniques and the integration of new estimators.
• Enable direct application of estimators in information-theoretic optimization problems, such as independent subspace analysis (ISA).
• Support a wide range of measures including Rényi, Tsallis, and Bregman divergences, kernel-based dependency measures, and distribution kernels.
• Facilitate scalable and robust estimation in high-dimensional settings, particularly for ISA with unknown source dimensions.

Proposed method

• Design a highly modular toolbox architecture where all estimators follow a unified template interface, enabling consistent calling syntax for base and meta estimators.
• Implement nonparametric estimation techniques, especially k-nearest neighbor (k-NN) methods, to avoid density estimation scaling issues in high dimensions.
• Support meta-estimator construction by combining existing base estimators (e.g., J-distance via symmetrized KL-divergence estimation).
• Integrate the ISA separation principle as a core application, enabling optimization of information-theoretic objectives like mutual information minimization or entropy maximization.
• Provide dedicated solvers for subproblems in ISA and extensions, including spectral clustering for dimension estimation and clustering of ICA components.
• Include built-in validation tools such as analytical vs. estimated value checks, Gram matrix positive semi-definiteness tests, and image registration benchmarks.

Experimental results

Research questions

• RQ1 How can a unified, extensible, and open-source toolbox be designed to estimate a broad range of information-theoretic measures on continuous variables?
• RQ2 To what extent can modular design enable the combination of existing estimators into new meta-estimators without rewriting core logic?
• RQ3 Can the toolbox efficiently support complex optimization problems in signal processing, such as independent subspace analysis with unknown source dimensions?
• RQ4 How does the toolbox ensure robustness and scalability in high-dimensional settings, particularly for large-scale ICA component clustering?
• RQ5 What is the impact of supporting multiplicative constant indicators in estimators for applications where scale-invariant measures are sufficient?

Key findings

• ITE supports the estimation of over 50 different information-theoretic quantities, including entropy, mutual information, divergence, association, cross-quantity, and distribution kernel measures.
• The toolbox enables the construction of new meta-estimators by combining existing base estimators, such as J-distance via symmetrized KL-divergence estimation using k-NN methods.
• The unified syntax and modular design allow any estimator to be directly used in optimization objectives, such as minimizing mutual information or maximizing entropy in ISA.
• The toolbox scales efficiently: spectral clustering-based solvers for ISA can handle up to a million ICA components on a standard desktop computer in minutes.
• The inclusion of validation tools—such as analytical vs. estimated value checks and Gram matrix tests—ensures estimator reliability and correctness.
• ITE is released under the GNU GPLv3 license and is multi-platform, supporting both Matlab and Octave environments with no external dependencies beyond standard toolboxes.