[Paper Review] A survey of some recent developments in measures of association

Sourav Chatterjee|arXiv (Cornell University)|Nov 9, 2022

Advanced Harmonic Analysis Research4 citations

TL;DR

This paper introduces a new nonparametric correlation coefficient, ξₙ, that consistently estimates a measure of dependence between random variables, with the limit ξ(X,Y) equal to 0 if and only if X and Y are independent, and 1 if and only if Y is a measurable function of X. The method is based on ranks and order statistics, and the paper extends it to standard Borel spaces and conditional dependence via a generalized coefficient ξₙ(Z,Y|X).

ABSTRACT

This paper surveys some recent developments in measures of association related to a new coefficient of correlation introduced by the author. A straightforward extension of this coefficient to standard Borel spaces (which includes all Polish spaces), overlooked in the literature so far, is proposed at the end of the survey.

Motivation & Objective

• To propose a new nonparametric correlation coefficient ξₙ that consistently estimates a measure of dependence with clear interpretation.
• To address the limitation of classical correlation coefficients in detecting non-monotonic associations.
• To extend the coefficient to general standard Borel spaces, enabling broader application beyond real-valued variables.
• To define a conditional dependence coefficient ξₙ(Z,Y|X) that measures dependence of Y on Z given X.
• To provide a theoretical foundation and practical implementation via R packages for statistical testing and analysis.

Proposed method

• Define ξₙ(X,Y) using order statistics: sort data by X, compute rank-based differences in Y-ranks, and apply a normalized difference formula.
• Use the limiting form ξ(X,Y) = ∫Var(𝔼[1_{Y≥t}|X])dμ(t) / ∫Var(1_{Y≥t})dμ(t), where μ is the law of Y, to define the population-level dependence measure.
• Establish consistency: ξₙ(X,Y) → ξ(X,Y) almost surely as n→∞, with ξ(X,Y)=0 iff X⊥Y and ξ(X,Y)=1 iff Y=f(X) a.s.
• Construct a symmetrized version by taking max(ξₙ(X,Y), ξₙ(Y,X)) to improve interpretability.
• Generalize to conditional dependence via ξₙ(Z,Y|X) := (ξₙ(W,Y) - ξₙ(X,Y)) / (1 - ξₙ(X,Y)), where W=(X,Z).
• Use Borel isomorphisms to extend the coefficient to random variables taking values in standard Borel spaces.

Experimental results

Research questions

• RQ1 Can a nonparametric correlation coefficient be constructed that consistently estimates dependence and detects non-monotonic relationships?
• RQ2 Does the proposed coefficient ξₙ(X,Y) converge almost surely to a limit ξ(X,Y) that is 0 under independence and 1 when one variable is a measurable function of the other?
• RQ3 Can the coefficient be generalized to conditional dependence in multivariate and abstract probability spaces?
• RQ4 Is the generalized coefficient ξₙ(Z,Y|X) consistent and interpretable as a measure of conditional dependence?
• RQ5 Can the method be implemented efficiently and used for hypothesis testing of independence in real data?

Key findings

• The coefficient ξₙ(X,Y) converges almost surely to ξ(X,Y) ∈ [0,1], with ξ(X,Y)=0 iff X and Y are independent and ξ(X,Y)=1 iff Y=f(X) a.s.
• For n=100, the average P-value for testing independence between X and Y was 0.001, while for independent u and Y it was 0.491, indicating strong power to detect dependence.
• For n=1000, the average P-value for X and Y was 0.000, while for u and Y it remained 0.495, showing robust detection even at larger sample sizes.
• The generalized coefficient ξₙ(Z,Y|X) converges almost surely to a limit ξ(Z,Y|X) ∈ [0,1], which is 0 iff Y⊥Z|X and 1 iff Y is a measurable function of Z given X.
• The R package XICOR is available on CRAN for computing ξₙ, P-values for independence, and the generalized coefficient.
• The FOCI and KFOCI algorithms are implemented in R packages for conditional dependence estimation and variable selection.