[Paper Review] Large-Scale Approximate Kernel Canonical Correlation Analysis

Weiran Wang, Karen Livescu|arXiv (Cornell University)|Jan 1, 2016

Neural Networks and Applications21 citations

TL;DR

This paper proposes a scalable stochastic optimization approach for large-scale approximate kernel canonical correlation analysis (KCCA), enabling efficient computation on massive datasets by combining random feature mapping with stochastic CCA. It successfully scales KCCA to 1.4 million speech samples with a 100,000-dimensional feature space on a standard workstation, overcoming the prohibitive computational cost of traditional KCCA.

ABSTRACT

Abstract: Kernel canonical correlation analysis (KCCA) is a nonlinear multi-view representation learning technique with broad applicability in statistics and machine learning. Although there is a closed-form solution for the KCCA objective, it involves solving an $N imes N$ eigenvalue system where $N$ is the training set size, making its computational requirements in both memory and time prohibitive for large-scale problems. Various approximation techniques have been developed for KCCA. A commonly used approach is to first transform the original inputs to an $M$-dimensional random feature space so that inner products in the feature space approximate kernel evaluations, and then apply linear CCA to the transformed inputs. In many applications, however, the dimensionality $M$ of the random feature space may need to be very large in order to obtain a sufficiently good approximation; it then becomes challenging to perform the linear CCA step on the resulting very high-dimensional data matrices. We show how to use a stochastic optimization algorithm, recently proposed for linear CCA and its neural-network extension, to further alleviate the computation requirements of approximate KCCA. This approach allows us to run approximate KCCA on a speech dataset with $1.4$ million training samples and a random feature space of dimensionality $M=100000$ on a typical workstation.

Motivation & Objective

• To address the high computational cost of kernel canonical correlation analysis (KCCA) on large-scale datasets.
• To reduce memory and time requirements in KCCA by leveraging random feature approximation.
• To improve scalability of approximate KCCA by applying stochastic optimization to high-dimensional random feature spaces.
• To enable practical deployment of KCCA on real-world large-scale datasets, such as speech data with over a million samples.
• To demonstrate that stochastic optimization can effectively handle high-dimensional random feature matrices without sacrificing approximation quality.

Proposed method

• Use random feature mapping to approximate kernel evaluations, transforming inputs into an M-dimensional space where inner products approximate kernel values.
• Apply linear CCA to the transformed data in the random feature space to learn nonlinear representations via kernel approximation.
• Introduce a stochastic optimization algorithm—previously developed for linear CCA and its neural network variant—to reduce computational cost during the CCA step.
• Optimize the CCA objective using mini-batches of data, significantly reducing memory and time complexity compared to batch eigen-decomposition.
• Combine random feature approximation with stochastic CCA to scale KCCA to datasets with N > 1M samples and M = 100,000 features.
• Use standard random Fourier features or other shift-invariant kernel approximations to maintain approximation accuracy while enabling efficient optimization.

Experimental results

Research questions

• RQ1 Can stochastic optimization be effectively applied to the linear CCA step in approximate KCCA to reduce computational cost?
• RQ2 How does the proposed method scale to large datasets with high-dimensional random feature spaces?
• RQ3 What is the trade-off between approximation accuracy and computational efficiency when using stochastic optimization in large-scale KCCA?
• RQ4 Can the method handle real-world datasets with over a million samples and high-dimensional feature maps?
• RQ5 Does the stochastic approach maintain the representational quality of traditional KCCA while enabling scalability?

Key findings

• The proposed method enables approximate KCCA to scale to datasets with 1.4 million training samples, which is infeasible with standard KCCA due to O(N²) memory and time complexity.
• Using a 100,000-dimensional random feature space, the method achieves effective nonlinear representation learning on large-scale speech data.
• Stochastic optimization reduces the computational burden of the CCA step, allowing training on a standard workstation without specialized hardware.
• The approach maintains the representational power of KCCA while significantly improving scalability compared to batch eigen-decomposition.
• The method demonstrates that combining random feature approximation with stochastic CCA is a viable and efficient solution for large-scale multi-view representation learning.
• The framework is general and can be extended to other kernel methods requiring high-dimensional feature mappings and efficient optimization.