[Paper Review] Overcoming the curse of dimensionality with Laplacian regularization in semi-supervised learning

Vivien Cabannes, Loucas Pillaud‐Vivien+2 more|arXiv (Cornell University)|Sep 9, 2020

Machine Learning and Data Classification52 references4 citations

TL;DR

This paper proposes a spectral filtering approach with Laplacian regularization in semi-supervised learning that overcomes the curse of dimensionality by leveraging reproducing kernel Hilbert space (RKHS) theory and low-rank approximation. It achieves fast convergence rates under low-density separation and reduces computational cost from $\mathcal{O}(n^3d^3)$ to $\mathcal{O}(n^{1+2\gamma}d\log^2 n)$ via a $n^{\gamma}\log n \times n^{\gamma}\log n$ matrix approximation, with $\gamma \in (0,1]$ depending on function regularity.

ABSTRACT

As annotations of data can be scarce in large-scale practical problems, leveraging unlabelled examples is one of the most important aspects of machine learning. This is the aim of semi-supervised learning. To benefit from the access to unlabelled data, it is natural to diffuse smoothly knowledge of labelled data to unlabelled one. This induces to the use of Laplacian regularization. Yet, current implementations of Laplacian regularization suffer from several drawbacks, notably the well-known curse of dimensionality. In this paper, we provide a statistical analysis to overcome those issues, and unveil a large body of spectral filtering methods that exhibit desirable behaviors. They are implemented through (reproducing) kernel methods, for which we provide realistic computational guidelines in order to make our method usable with large amounts of data.

Motivation & Objective

• Address the curse of dimensionality in Laplacian regularization for semi-supervised learning.
• Develop a statistically principled framework to ensure consistent estimators that avoid dimensionality-induced degradation.
• Provide scalable computational guidelines using kernel methods and low-rank approximation for large-scale data.
• Establish fast convergence rates under the low-density separation hypothesis.
• Enable practical deployment of Laplacian regularization in high-dimensional settings through computational efficiency.

Proposed method

• Proposes a spectral filtering framework based on functional space analysis and reproducing kernel Hilbert spaces (RKHS).
• Uses kernel methods to implement estimators that are consistent and avoid the curse of dimensionality.
• Applies low-rank approximation to the kernel matrix using random subsampling or leverage scores, reducing matrix size from $n(d+1) \times n(d+1)$ to $n^\gamma \log n \times n^\gamma \log n$.
• Employs a regularization scheme with a parameter $\lambda$ and a diffusion operator $\mathcal{L}$ to control smoothness and stability.
• Derives concentration bounds on the empirical projection operator $P$ using random matrix theory and sub-Gaussian tail estimates.
• Optimizes the approximation error via parameter tuning, particularly setting $ab = \log p$ for optimal bounds on operator norms.

Experimental results

Research questions

• RQ1 Can Laplacian regularization be made robust to high-dimensional input spaces in semi-supervised learning?
• RQ2 What theoretical conditions ensure that Laplacian-based estimators avoid the curse of dimensionality?
• RQ3 How can kernel methods and low-rank approximation be combined to maintain statistical consistency while reducing computational cost?
• RQ4 What are the optimal convergence rates of Laplacian-regularized estimators under low-density separation?
• RQ5 Can leverage score sampling improve the approximation quality and reduce the required sample size in the low-rank scheme?

Key findings

• The proposed estimator achieves fast convergence rates under the low-density separation hypothesis, avoiding the curse of dimensionality.
• The statistical error bound scales as $\mathcal{O}\left(\frac{\log p}{p}\right)$ and $\mathcal{O}\left(\frac{\log^a p}{p^a}\right)$ for the approximation error, with optimal tuning at $ab = \log p$.
• The computational cost is reduced from $\mathcal{O}(n^3d^3)$ to $\mathcal{O}(n^{1+2\gamma}d\log^2 n)$, where $\gamma \in (0,1]$ depends on the regularity of the underlying function.
• The method achieves consistency and fast convergence even in high-dimensional settings by leveraging the smoothness of the function and the structure of the RKHS.
• Leverage score sampling improves the approximation quality, allowing $p = (\mu\lambda)^\gamma \log n$ for a given capacity, enhancing stability.
• Theoretical bounds on the expected $L^2$ error are derived, showing dependence on $n$, $\lambda$, $\kappa^2$, and the function's regularity through $\|g_\rho\|_{L^2}$ and $\|\mathcal{L}g_\rho\|_{L^2}$.