[Paper Review] Hierarchically Compositional Kernels for Scalable Nonparametric Learning

Jie Chen, Haim Avron+1 more|arXiv (Cornell University)|Aug 2, 2016

Sparse and Compressive Sensing Techniques40 references3 citations

TL;DR

This paper introduces hierarchically compositional kernels that enable scalable nonparametric learning by combining Nyström-based global low-rank approximation with local lossless approximations in a hierarchical partitioning of the data domain. The resulting kernel matrix exhibits a recursively off-diagonal low-rank structure, reducing training memory and arithmetic complexity from $O(n^2)$ and $O(n^3)$ to $O(nr)$ and $O(nr^2)$, respectively, while preserving strict positive-definiteness and achieving competitive performance with smaller rank $r$ than other approximate kernels.

ABSTRACT

We propose a novel class of kernels to alleviate the high computational cost of large-scale nonparametric learning with kernel methods. The proposed kernel is defined based on a hierarchical partitioning of the underlying data domain, where the Nyström method (a globally low-rank approximation) is married with a locally lossless approximation in a hierarchical fashion. The kernel maintains (strict) positive-definiteness. The corresponding kernel matrix admits a recursively off-diagonal low-rank structure, which allows for fast linear algebra computations. Suppressing the factor of data dimension, the memory and arithmetic complexities for training a regression or a classifier are reduced from $O(n^2)$ and $O(n^3)$ to $O(nr)$ and $O(nr^2)$, respectively, where $n$ is the number of training examples and $r$ is the rank on each level of the hierarchy. Although other randomized approximate kernels entail a similar complexity, empirical results show that the proposed kernel achieves a matching performance with a smaller $r$. We demonstrate comprehensive experiments to show the effective use of the proposed kernel on data sizes up to the order of millions.

Motivation & Objective

• To address the high computational cost of large-scale kernel methods, which scale poorly as $n$ increases due to dense, unstructured kernel matrices.
• To develop a kernel construction that maintains strict positive-definiteness while enabling efficient computation through structured low-rank approximations.
• To reduce the memory and arithmetic complexity of kernel ridge regression and classification from $O(n^2)$ and $O(n^3)$ to $O(nr)$ and $O(nr^2)$, respectively, by exploiting hierarchical data partitioning.
• To enable effective use of kernel methods on data sets of up to millions of points, where standard kernel methods become infeasible.
• To support principled parameter estimation via maximum likelihood, enabling optimization of kernel hyperparameters even in high-dimensional settings.

Proposed method

• The method constructs a kernel through hierarchical partitioning of the data domain into nested clusters, enabling recursive low-rank approximation of off-diagonal blocks.
• It combines the Nyström method for global low-rank approximation with local, lossless kernel evaluations within each partition to preserve accuracy and positive-definiteness.
• The kernel matrix is structured such that off-diagonal blocks are low-rank, enabling fast linear algebra operations via recursive matrix factorization.
• The construction ensures strict positive-definiteness by design, allowing the kernel to be used in Gaussian process and kernel ridge regression frameworks.
• Out-of-sample predictions are straightforward due to the explicit kernel function and hierarchical partitioning, which define cluster membership for new points via distance to centroids.
• The method supports efficient computation of log-determinant and matrix inverse terms via specialized algorithms for recursively low-rank matrices, enabling scalable maximum likelihood estimation.

Experimental results

Research questions

• RQ1 Can a kernel be constructed that maintains strict positive-definiteness while enabling sub-quadratic memory and sub-cubic arithmetic complexity for kernel methods on large-scale data?
• RQ2 How can hierarchical partitioning and low-rank approximation be combined to preserve kernel accuracy while reducing computational cost?
• RQ3 Does the proposed kernel achieve comparable or better performance than existing randomized approximate kernels with a smaller rank $r$?
• RQ4 Can the recursive low-rank structure of the kernel matrix be exploited to enable efficient maximum likelihood estimation for kernel hyperparameters?
• RQ5 Is the method scalable to data sets of up to millions of points, and how does it perform in practice compared to standard kernel methods?

Key findings

• The proposed kernel reduces training memory and arithmetic complexity from $O(n^2)$ and $O(n^3)$ to $O(nr)$ and $O(nr^2)$, respectively, with $r$ being the rank at each hierarchical level.
• Empirical results show that the proposed kernel achieves matching or better performance than other approximate kernels while requiring a smaller rank $r$, indicating higher efficiency.
• The kernel matrix structure enables fast computation of log-determinant and matrix inverse terms via recursive algorithms, supporting scalable maximum likelihood estimation.
• The method is effective on data sets of up to the order of millions of examples, demonstrating practical scalability beyond typical kernel method limits.
• The kernel maintains strict positive-definiteness, ensuring validity in all kernel-based learning and Gaussian process inference tasks.
• Out-of-sample extension is straightforward, as partition membership for new points is determined by minimal distance to cluster centers in the hierarchical partitioning.