[Paper Review] Grounding Representation Similarity with Statistical Testing

Frances Ding, Jean-Stanislas Denain+1 more|arXiv (Cornell University)|Aug 3, 2021

Domain Adaptation and Few-Shot Learning41 references4 citations

TL;DR

This paper proposes a statistical testing framework to evaluate representation similarity metrics in neural networks by assessing their sensitivity to functional changes and specificity against irrelevant noise. It finds that widely used metrics like CKA and CCA fail in key benchmarks, while the classical Orthogonal Procrustes distance performs robustly across diverse functional behaviors, highlighting the need for systematic evaluation over intuition-driven metric selection.

ABSTRACT

To understand neural network behavior, recent works quantitatively compare different networks' learned representations using canonical correlation analysis (CCA), centered kernel alignment (CKA), and other dissimilarity measures. Unfortunately, these widely used measures often disagree on fundamental observations, such as whether deep networks differing only in random initialization learn similar representations. These disagreements raise the question: which, if any, of these dissimilarity measures should we believe? We provide a framework to ground this question through a concrete test: measures should have sensitivity to changes that affect functional behavior, and specificity against changes that do not. We quantify this through a variety of functional behaviors including probing accuracy and robustness to distribution shift, and examine changes such as varying random initialization and deleting principal components. We find that current metrics exhibit different weaknesses, note that a classical baseline performs surprisingly well, and highlight settings where all metrics appear to fail, thus providing a challenge set for further improvement.

Motivation & Objective

• To address the lack of consensus among representation similarity metrics like CKA and CCA, which often disagree on fundamental comparisons.
• To establish formal criteria—sensitivity to functional changes and specificity against noise—for evaluating these metrics.
• To construct a comprehensive benchmark using functional behaviors such as probing accuracy and out-of-distribution robustness to ground-truth metric performance.
• To identify failure cases where all metrics perform no better than chance, creating a challenge set for future metric development.
• To advocate for systematic evaluation over intuition-driven metric selection, countering recency bias that favors newer methods over classical baselines.

Proposed method

• Define representation similarity as a classification problem: whether two representations are functionally similar or different.
• Use statistical testing principles: metrics should be sensitive to changes that affect model behavior and specific against unimportant changes like random initialization.
• Construct a benchmark of 30,480 examples varying representations across random seed, layer depth, and low-rank approximations.
• Measure functional behavior via probing accuracy (syntactic information) and out-of-distribution (OOD) performance.
• Score metrics by their rank correlation with measured functional differences, using in-distribution representations to predict OOD behavior.
• Evaluate five metrics: CKA, CCA, PWCCA, Orthogonal Procrustes, and vector space alignment, with a focus on sensitivity to principal component deletion and random initialization.

Experimental results

Research questions

• RQ1 Do widely used representation similarity metrics like CKA and CCA reliably detect changes that affect functional behavior?
• RQ2 Are these metrics specific enough to ignore irrelevant changes such as random weight initialization?
• RQ3 Can representation similarity metrics predict out-of-distribution performance using only in-distribution representations?
• RQ4 Are there systematic failure cases where all metrics perform no better than chance?
• RQ5 Does a classical metric like Orthogonal Procrustes outperform modern alternatives in grounded benchmarks?

Key findings

• CCA fails specificity: it detects spurious similarities due to random initialization noise, treating even distant layers as similar.
• CKA fails sensitivity: it fails to detect meaningful changes in all but the top 10 principal components, missing moderate-sized differences.
• Orthogonal Procrustes consistently performs well across probing accuracy and OOD performance benchmarks, outperforming CKA and CCA.
• A challenge set was identified where no metric predicts OOD performance better than chance, highlighting a critical gap in current metric capabilities.
• The classical Orthogonal Procrustes distance is a surprisingly strong baseline, suggesting that newer metrics may not always offer advantages over established methods.
• Representation similarity metrics often fail to correlate with functional differences, especially in low signal-to-noise regimes like the Numerical stress test.