[Paper Review] From Distance Correlation to Multiscale Generalized Correlation

Cencheng Shen, Carey E. Priebe+1 more|arXiv (Cornell University)|Oct 26, 2017

Gene expression and cancer classification5 citations

TL;DR

This paper formalizes the population version of Multiscale Generalized Correlation (MGC) using characteristic functions and nearest-neighbor methods, establishing theoretical foundations that enhance the algorithmic Sample MGC. It proves asymptotic and finite-sample properties, demonstrating MGC's superior power in detecting general dependencies—especially nonlinear and multivariate ones—while retaining high power for monotone relationships.

ABSTRACT

Understanding and developing a correlation measure that can detect general dependencies is not only imperative to statistics and machine learning, but also crucial to general scientific discovery in the big data age. We proposed the Multiscale Generalized Correlation (MGC) in Shen et al. 2017 as a novel correlation measure, which worked well empirically and helped a number of real data discoveries. But there is a wide gap with respect to the theoretical side, e.g., the population statistic, the convergence from sample to population, how well does the algorithmic Sample MGC perform, etc. To better understand its underlying mechanism, in this paper we formalize the population version of local distance correlations, MGC, and the optimal local scale between the underlying random variables, by utilizing the characteristic functions and incorporating the nearest-neighbor machinery. The population version enables a seamless connection with, and significant improvement to, the algorithmic Sample MGC, both theoretically and in practice, which further allows a number of desirable asymptotic and finite-sample properties to be proved and explored for MGC. The advantages of MGC are further illustrated via a comprehensive set of simulations with linear, nonlinear, univariate, multivariate, and noisy dependencies, where it loses almost no power against monotone dependencies while achieving superior performance against general dependencies.

Motivation & Objective

• To close the theoretical gap in understanding Multiscale Generalized Correlation (MGC), particularly its population formulation and convergence from sample to population.
• To establish a rigorous theoretical framework for MGC by formalizing the population version of local distance correlations and optimal local scales.
• To improve the algorithmic Sample MGC through theoretical insights, enabling stronger asymptotic and finite-sample properties.
• To demonstrate MGC's superiority in detecting general dependencies—especially nonlinear, multivariate, and noisy relationships—while preserving power for monotone dependencies.

Proposed method

• Formalizing the population version of local distance correlation using characteristic functions to describe the underlying dependence structure.
• Incorporating nearest-neighbor machinery to estimate optimal local scales between random variables in the population setting.
• Deriving the population MGC statistic as a multiscale generalization of local correlation, capturing dependencies at multiple scales.
• Establishing a seamless theoretical connection between the population MGC and the algorithmic Sample MGC for improved convergence and performance.
• Using characteristic functions to characterize the joint distribution and dependence structure, enabling exact computation of population-level correlation measures.
• Proving asymptotic and finite-sample properties of MGC based on the formalized population framework, including consistency and power analysis.

Experimental results

Research questions

• RQ1 What is the population version of MGC, and how does it relate to the sample-based algorithmic implementation?
• RQ2 How does the optimal local scale between random variables emerge from the population formulation using characteristic functions?
• RQ3 To what extent does the theoretical framework improve the convergence and finite-sample performance of Sample MGC?
• RQ4 How does MGC compare in power to existing correlation measures across diverse dependency types, including linear, nonlinear, univariate, multivariate, and noisy dependencies?
• RQ5 Can the theoretical foundation of MGC explain its empirical success in real data discovery tasks?

Key findings

• The population version of MGC is formally derived using characteristic functions and nearest-neighbor methods, enabling a rigorous theoretical foundation.
• The theoretical framework establishes strong convergence properties, linking the population MGC to the algorithmic Sample MGC with improved asymptotic guarantees.
• MGC maintains almost no power loss against monotone dependencies while significantly outperforming existing methods in detecting general nonlinear and multivariate dependencies.
• The method achieves superior finite-sample performance due to the improved theoretical grounding, particularly in noisy and complex dependency structures.
• Comprehensive simulations confirm MGC's robustness and high power across diverse dependency types, including univariate, multivariate, and noisy settings.
• The formalization enables the proof of desirable asymptotic and finite-sample properties, such as consistency and sensitivity to general dependence.