[Paper Review] Mutual Dependence: A Novel Method for Computing Dependencies Between Random Variables

Rahul Agarwal, Pierre Sacré+1 more|arXiv (Cornell University)|Jun 1, 2015

Advanced Clustering Algorithms Research4 citations

TL;DR

This paper introduces a novel, direct estimator for mutual dependence—a theoretically ideal dependency measure that satisfies Granger's axioms and is invariant under strictly monotonic transformations. Using a band-limited maximum likelihood (BLML) estimator, the method computes mutual dependence directly from data without density estimation or numerical integration, achieving faster convergence, lower bias, and superior computational efficiency compared to Pearson correlation and distance correlation across diverse nonlinearities and marginal distributions.

ABSTRACT

In data science, it is often required to estimate dependencies between different data sources. These dependencies are typically calculated using Pearson's correlation, distance correlation, and/or mutual information. However, none of these measures satisfy all the Granger's axioms for an "ideal measure". One such ideal measure, proposed by Granger himself, calculates the Bhattacharyya distance between the joint probability density function (pdf) and the product of marginal pdfs. We call this measure the mutual dependence. However, to date this measure has not been directly computable from data. In this paper, we use our recently introduced maximum likelihood non-parametric estimator for band-limited pdfs, to compute the mutual dependence directly from the data. We construct the estimator of mutual dependence and compare its performance to standard measures (Pearson's and distance correlation) for different known pdfs by computing convergence rates, computational complexity, and the ability to capture nonlinear dependencies. Our mutual dependence estimator requires fewer samples to converge to theoretical values, is faster to compute, and captures more complex dependencies than standard measures.

Motivation & Objective

• To develop a computationally efficient and accurate estimator for mutual dependence, an ideal dependency measure that satisfies Granger's axioms.
• To overcome the limitations of existing measures like Pearson correlation (fails on nonlinearities) and distance correlation (slow, non-invariant under monotonic transforms).
• To enable direct computation of mutual dependence from data by avoiding non-parametric density estimation and numerical integration.
• To demonstrate superior convergence rates and computational efficiency compared to standard dependency measures across various nonlinear and non-normal data distributions.

Proposed method

• Proposes a new dependency measure, mutual dependence $d$, defined as the Bhattacharyya distance between the joint pdf and the product of marginal pdfs.
• Employs the Band-Limited Maximum Likelihood (BLML) estimator to non-parametrically estimate pdfs under a known cut-off frequency $f_c$, ensuring smoothness and consistency.
• Derives an analytical expression for mutual dependence using BLML estimates, eliminating the need for numerical integration over joint and marginal pdfs.
• Computes the mutual dependence estimator $\hat{d}$ directly from data with $\mathcal{O}(B^2 + n)$ complexity, where $B \leq n$ is the number of non-empty bins.
• Uses Monte Carlo simulations to compare convergence, bias, variance, and computational time across $\hat{r}$, $\hat{R}$, and $\hat{d}$ under varying nonlinearities and generating pdfs.
• Sets the cut-off frequency $f_c = \frac{1}{1 - \rho^2}$ for the BLML estimator in simulations to ensure accurate representation of the underlying pdf structure.

Experimental results

Research questions

• RQ1 Can mutual dependence be computed directly from data without non-parametric density estimation or numerical integration?
• RQ2 How does the convergence rate of the proposed mutual dependence estimator compare to Pearson correlation and distance correlation across different types of nonlinear dependencies?
• RQ3 Does the mutual dependence estimator maintain low bias and variance across diverse marginal distributions and nonlinear functional forms?
• RQ4 What is the computational complexity of the proposed estimator, and how does it compare to the $\mathcal{O}(n^2)$ complexity of distance correlation?
• RQ5 How sensitive is the performance of the mutual dependence estimator to the choice of cut-off frequency in the BLML framework?

Key findings

• The mutual dependence estimator $\hat{d}$ converges faster than $\hat{r}$ and $\hat{R}$ across all tested nonlinearities, including linear, quadratic, cubic, and sinusoidal dependencies.
• For nonlinear data, $\hat{r}$ and $\hat{R}$ exhibit higher bias and variance, while $\hat{d}$ maintains uniformly low bias and variance across all nonlinearities and marginal distributions.
• The integrated mean squared error (IMSE) of $\hat{d}$ decreases steadily with increasing sample size for all data types, whereas $\hat{R}$'s IMSE does not decrease for cubic and band-limited data due to non-decreasing variance.
• The computational complexity of $\hat{d}$ is $\mathcal{O}(B^2 + n)$, which is significantly faster than $\mathcal{O}(n^2)$ for $\hat{R}$, especially in dense data regimes where $B \ll n$.
• The mutual dependence estimator achieves lower IMSE than both $\hat{r}$ and $\hat{R}$ at all sample sizes, indicating superior estimation accuracy and consistency.
• The estimator is invariant under strictly monotonic transformations, satisfying a key axiom for an ideal dependency measure, and outperforms $\hat{R}$ in this regard despite its theoretical shortcomings.