[Paper Review] Limit theorems of Chatterjee's rank correlation

Zhexiao Lin, Han Fang|arXiv (Cornell University)|Apr 17, 2022

Advanced Statistical Methods and Models4 citations

TL;DR

This paper establishes the asymptotic normality of Chatterjee’s rank correlation and Azadkia-Chatterjee’s graph-based correlation under general conditions, proving that both statistics are asymptotically normal as long as Y is not a measurable function of X. It further shows the asymptotic variance is uniformly bounded by 36 and provides a consistent variance estimator, resolving a long-standing open problem in nonparametric dependence measurement.

ABSTRACT

Establishing the limiting distribution of Chatterjee's rank correlation for a general, possibly non-independent, pair of random variables has been eagerly awaited by many. This paper shows that (a) Chatterjee's rank correlation is asymptotically normal as long as one variable is not a measurable function of the other, (b) the corresponding asymptotic variance is uniformly bounded by 36, and (c) a consistent variance estimator exists. Similar results also hold for Azadkia-Chatterjee's graph-based correlation coefficient, a multivariate analogue of Chatterjee's original proposal. The proof is given by appealing to Hájek representation and Chatterjee's nearest-neighbor CLT.

Motivation & Objective

• To establish the limiting distribution of Chatterjee’s rank correlation for general, non-i.i.d. bivariate data where Y is not a measurable function of X.
• To resolve the open problem of deriving the asymptotic distribution of Chatterjee’s and Azadkia-Chatterjee’s correlation coefficients under weak dependence assumptions.
• To provide a consistent estimator for the asymptotic variance of these correlation measures, enabling valid inference in practice.
• To extend the theoretical results to the multivariate graph-based correlation coefficient proposed by Azadkia and Chatterjee.
• To establish uniform boundedness of the asymptotic variance by 36, ensuring stability across diverse dependence structures.

Proposed method

• Uses Hájek representation to express the correlation statistics as a sum of i.i.d. U-statistic-like terms, enabling central limit theorem (CLT) application.
• Applies Chatterjee’s nearest-neighbor CLT to handle the dependence induced by nearest-neighbor indices in high-dimensional settings.
• Derives a consistent variance estimator, denoted $ frac{1}{n^3} ext{sum of rank-based terms} $, which accounts for nearest-neighbor structure and ties.
• Employs symmetrization and Efron-Stein inequality to control the variance of the conditional expectation given the design points $ X_i $.
• Introduces a counterfactual potential outcome framework using $ ilde{X}_i, ilde{Y}_i $ to decouple the dependence between ranks and design points.
• Uses the conditional expectation $ ext{E}[F_Y(Y_1 igwedge ilde{Y}_1) ig| X_1] $ as a key building block to represent the asymptotic variance.

Experimental results

Research questions

• RQ1 Is Chatterjee’s rank correlation asymptotically normal when Y is not a measurable function of X?
• RQ2 What is the asymptotic variance of Chatterjee’s and Azadkia-Chatterjee’s correlation coefficients, and is it uniformly bounded?
• RQ3 Can a consistent variance estimator be constructed for these rank-based correlation measures?
• RQ4 Does the asymptotic normality hold under general dependence structures, including when X and Y are dependent?
• RQ5 How does the nearest-neighbor structure affect the asymptotic distribution of the correlation estimator?

Key findings

• Chatterjee’s rank correlation $ ar{ heta}_n $ is asymptotically normal under the condition that Y is not a measurable function of X, with convergence in distribution to a standard normal variate.
• The asymptotic variance of $ ar{ heta}_n $ is uniformly bounded by 36, regardless of the underlying distribution of $ (X,Y) $, provided Y is not a measurable function of X.
• A consistent estimator $ ilde{ au}^2 $ of the asymptotic variance is derived, which depends only on the ranks and nearest-neighbor indices of the data.
• The same asymptotic normality and variance bounds hold for Azadkia-Chatterjee’s graph-based correlation coefficient, extending the results to multivariate settings.
• The proof establishes that the difference between the conditional expectation of the rank-based statistic and its approximation converges in probability to zero, validating the Hájek representation.
• The variance estimator $ ilde{ au}^2 $ is explicitly constructed using rank-based U-statistic-like terms involving $ R_i igwedge R_{N(i)} $, nearest-neighbor indices, and indicator functions of rank orderings.