[Paper Review] Multi-set Canonical Correlation Analysis simply explained

Lucas C. Parra|ArXiv.org|Feb 11, 2018

Sensory Analysis and Statistical Methods12 references17 citations

TL;DR

This paper presents a simple, one-step solution for multiset canonical correlation analysis (MCCA) by computing eigenvectors of the matrix $\mathbf{D}^{-1}\mathbf{R}$, where $\mathbf{R}$ is the covariance matrix of concatenated data and $\mathbf{D}$ is its block-diagonal. The method maximizes inter-set correlation (ISC) without additional constraints and is equivalent to a two-step procedure involving PCA whitening followed by PCA on concatenated data.

ABSTRACT

There are a multitude of methods to perform multi-set correlated component analysis (MCCA), including some that require iterative solutions. The methods differ on the criterion they optimize and the constraints placed on the solutions. This note focuses perhaps on the simplest version, which can be solved in a single step as the eigenvectors of matrix ${\bf D}^{-1} {\bf R}$. Here ${\bf R}$ is the covariance matrix of the concatenated data, and ${\bf D}$ is its block-diagonal. This note shows that this solution maximizes inter-set correlation (ISC) without further constraints. It also relates the solution to a two step procedure, which first whitens each dataset using PCA, and then performs an additional PCA on the concatenated and whitened data. Both these solutions are known, although a clear derivation and simple implementation are hard to find. This short note aims to remedy this.

Motivation & Objective

• To provide a clear, concise derivation of a simple multiset CCA (MCCA) method that maximizes inter-set correlation (ISC).
• To show that the solution to MCCA can be obtained in a single step as the eigenvectors of $\mathbf{D}^{-1}\mathbf{R}$, avoiding iterative optimization.
• To clarify the equivalence between the one-step eigenvector solution and a two-step procedure involving PCA whitening of each data set followed by PCA on the concatenated whitened data.
• To offer a straightforward implementation of the method, making it accessible despite the complexity of prior formulations.

Proposed method

• The method formulates MCCA as maximizing the ratio of between-set covariance to within-set covariance, defining inter-set correlation (ISC) as $\rho = \frac{1}{N-1} \frac{r_B}{r_W}$.
• It derives the optimization problem by setting the derivative of $\rho$ with respect to each projection vector $\mathbf{v}^l$ to zero, leading to the generalized eigenvalue equation $\mathbf{R}\mathbf{v} = \mathbf{D}\mathbf{v}\lambda$.
• The solution is obtained as the eigenvectors of $\mathbf{D}^{-1}\mathbf{R}$, with the largest eigenvalue corresponding to the maximum ISC.
• The method is shown to be equivalent to a two-step procedure: first performing PCA on each data set to whiten it, then performing PCA on the concatenated whitened data.
• The transformation matrix $\mathbf{V}$ is computed as $\mathbf{V} = \mathbf{U}\mathbf{\Lambda}^{-1/2}\tilde{\mathbf{V}}$, where $\mathbf{U}$ and $\mathbf{\Lambda}$ are from the first PCA, and $\tilde{\mathbf{V}}$ from the second.
• A MATLAB implementation is provided that computes the solution using standard eigendecomposition, with the largest eigenvalue yielding the highest ISC.

Experimental results

Research questions

• RQ1 How can multiset canonical correlation analysis (MCCA) be solved in a single, non-iterative step with a clear mathematical derivation?
• RQ2 What is the relationship between the one-step eigenvector solution of $\mathbf{D}^{-1}\mathbf{R}$ and the two-step procedure of whitening each data set followed by PCA on concatenated data?
• RQ3 Does the one-step solution naturally enforce uncorrelated projections across components, and if so, how?
• RQ4 How does this method compare to other MCCA formulations in terms of simplicity, interpretability, and computational efficiency?
• RQ5 Can this method be implemented efficiently and robustly, especially in high-dimensional, low-sample regimes?

Key findings

• The one-step solution to MCCA is given by the eigenvectors of $\mathbf{D}^{-1}\mathbf{R}$, where $\mathbf{R}$ is the full covariance matrix and $\mathbf{D}$ is its block-diagonal, and this solution maximizes inter-set correlation (ISC).
• The solution automatically ensures that subsequent components are uncorrelated from earlier ones, as the eigenvectors diagonalize $\mathbf{D}$, satisfying the standard orthogonality constraint.
• The method is mathematically equivalent to a two-step procedure: first applying PCA to whiten each data set individually, then performing PCA on the concatenated, whitened data.
• The largest eigenvalue $\lambda$ of $\mathbf{D}^{-1}\mathbf{R}$ corresponds to the maximum ISC via $\rho = \frac{\lambda - 1}{N - 1}$, with $\rho \leq 1$.
• The method reduces to standard CCA when $N = 2$, and the resulting $\rho$ equals the Pearson correlation coefficient.
• A direct MATLAB implementation is provided that computes the solution using standard eigendecomposition, with the last eigenvector corresponding to the maximum ISC.