[Paper Review] A General Scoring Rule for Randomized Kernel Approximation with Application to Canonical Correlation Analysis

Yinsong Wang, Shahin Shahrampour|arXiv (Cornell University)|Oct 11, 2019

Gaussian Processes and Bayesian Inference32 references4 citations

TL;DR

This paper proposes a general scoring rule for data-dependent random feature sampling that improves kernel approximation across diverse machine learning tasks, with a focus on Canonical Correlation Analysis (CCA). By deriving a principled distribution for feature sampling that maximizes canonical correlations, the method outperforms existing techniques in CCA experiments while recovering known methods like leverage scores and energy-based sampling.

ABSTRACT

Random features has been widely used for kernel approximation in large-scale machine learning. A number of recent studies have explored data-dependent sampling of features, modifying the stochastic oracle from which random features are sampled. While proposed techniques in this realm improve the approximation, their application is limited to a specific learning task. In this paper, we propose a general scoring rule for sampling random features, which can be employed for various applications with some adjustments. We first observe that our method can recover a number of data-dependent sampling methods (e.g., leverage scores and energy-based sampling). Then, we restrict our attention to a ubiquitous problem in statistics and machine learning, namely Canonical Correlation Analysis (CCA). We provide a principled guide for finding the distribution maximizing the canonical correlations, resulting in a novel data-dependent method for sampling features. Numerical experiments verify that our algorithm consistently outperforms other sampling techniques in the CCA task.

Motivation & Objective

• To develop a general scoring rule for random feature sampling applicable across multiple machine learning tasks.
• To provide a principled method for selecting sampling distributions that maximize canonical correlations in CCA.
• To unify and generalize existing data-dependent sampling techniques such as leverage scores and energy-based sampling.
• To improve kernel approximation accuracy in large-scale CCA by optimizing feature sampling distribution.
• To validate the method's superiority through numerical experiments on CCA tasks.

Proposed method

• The authors derive a general scoring rule that determines optimal sampling probabilities for random features based on data structure.
• The method formulates a distribution over features that maximizes expected canonical correlation in CCA.
• It generalizes existing sampling strategies by embedding them as special cases within the proposed scoring framework.
• The scoring rule is derived using a variational optimization approach to maximize the expected correlation between projected data subspaces.
• The method enables efficient sampling by assigning higher probabilities to features that contribute more to canonical correlation.
• The approach is adaptable to other kernel-based methods with minor adjustments to the scoring function.

Experimental results

Research questions

• RQ1 How can a general scoring rule be designed to improve random feature sampling across diverse machine learning tasks?
• RQ2 Can the proposed method recover known data-dependent sampling techniques such as leverage scores and energy-based sampling?
• RQ3 What is the optimal sampling distribution for maximizing canonical correlation in CCA?
• RQ4 How does the proposed method compare in performance to existing sampling strategies in CCA?
• RQ5 To what extent does the general scoring rule enhance kernel approximation accuracy in large-scale settings?

Key findings

• The proposed scoring rule recovers established data-dependent sampling methods like leverage scores and energy-based sampling as special cases.
• The method achieves higher canonical correlation values than baseline sampling techniques in CCA experiments.
• Numerical results show consistent performance gains across multiple datasets and settings.
• The method provides a principled, unified framework for feature sampling that extends beyond CCA to other kernel methods.
• The general scoring rule enables more accurate kernel approximation by focusing on features that most improve correlation in the target task.
• The algorithm demonstrates robustness and scalability in large-scale learning scenarios.