[Paper Review] Spectral Analysis of Symmetric and Anti-Symmetric Pairwise Kernels

Tapio Pahikkala, Markus Viljanen+2 more|arXiv (Cornell University)|Jun 19, 2015

Face and Expression Recognition15 references3 citations

TL;DR

This paper provides a spectral analysis of symmetric and anti-symmetric pairwise kernels, showing that symmetrization and anti-symmetrization reduce the kernel's effective dimension and bound the regularization bias of the transformed kernels in terms of the original. The key contribution is a theoretical guarantee that enforcing symmetry or anti-symmetry does not compromise approximation power while potentially improving learning efficiency.

ABSTRACT

We consider the problem of learning regression functions from pairwise data when there exists prior knowledge that the relation to be learned is symmetric or anti-symmetric. Such prior knowledge is commonly enforced by symmetrizing or anti-symmetrizing pairwise kernel functions. Through spectral analysis, we show that these transformations reduce the kernel's effective dimension. Further, we provide an analysis of the approximation properties of the resulting kernels, and bound the regularization bias of the kernels in terms of the corresponding bias of the original kernel.

Motivation & Objective

• To analyze the spectral properties of symmetric and anti-symmetric pairwise kernels derived from original kernels.
• To investigate how symmetrization and anti-symmetrization affect the effective dimension of pairwise kernels.
• To bound the regularization bias of transformed kernels in relation to the original kernel's bias.
• To establish theoretical guarantees on the approximation properties of symmetric and anti-symmetric kernels.

Proposed method

• The paper employs spectral analysis of the integral operator associated with pairwise kernels, focusing on eigenvalues and eigenfunctions.
• It defines symmetrized and anti-symmetrized kernels via projection operators S and A acting on the original kernel K.
• The effective dimension of the kernel is analyzed using the trace of the operator (T_K + λI)^{-1}, which quantifies the kernel's capacity.
• Matrix inequalities such as Choi’s, Kadison’s, and Kantorovich-type inequalities are used to bound the regularization bias in terms of the original kernel’s spectral properties.
• The analysis assumes compact input space, continuous kernels, and a probability measure μ on X, with all functions in L²(𝒳, μ).
• Theoretical bounds are derived using the Löwner-Heinz theorem and properties of positive unital linear maps on bounded operators.

Experimental results

Research questions

• RQ1 How does symmetrization or anti-symmetrization of a pairwise kernel affect its effective dimension?
• RQ2 What is the relationship between the regularization bias of the original kernel and that of its symmetric or anti-symmetric counterpart?
• RQ3 Can symmetric and anti-symmetric kernels still approximate any continuous symmetric or anti-symmetric function arbitrarily well?
• RQ4 How do the spectral properties of the transformed kernels compare to those of the original kernel?

Key findings

• The effective dimension of both symmetrized and anti-symmetrized pairwise kernels is strictly smaller than that of the original kernel.
• The approximation properties of symmetric and anti-symmetric kernels are preserved, and they can approximate any continuous symmetric or anti-symmetric function arbitrarily well.
• The regularization bias of the symmetric and anti-symmetric kernels is bounded by a factor of (α + β)² / (4αβ) relative to the original kernel’s bias, where α and β are the smallest and largest eigenvalues of (T_K + λI).
• For a kernel with maximum value 1, the bound on the regularization bias factor is at most (1 + λ + λ)² / (4λ(1 + λ)) = (1 + 2λ)² / (4λ(1 + λ)), which approaches 1 as λ → 0.
• The bias of the regularized solution f^λ_K^S for symmetric f is identical to that of f^λ_K^PI, showing no additional bias from symmetrization when the true function is symmetric.
• The analysis confirms that enforcing symmetry or anti-symmetry does not reduce the expressive power of the kernel, but may improve learning efficiency through reduced effective dimension.