[Paper Review] On the Size of the Online Kernel Sparsification Dictionary

Yi Sun, Faustino Gomez+1 more|arXiv (Cornell University)|Jun 18, 2012

Advanced Adaptive Filtering Techniques26 references9 citations

TL;DR

This paper analyzes the size of the dictionary in online kernel sparsification using a novel formula linking the expected determinant of the kernel Gram matrix to the eigenvalues of the covariance operator. It proves that under technical conditions, the dictionary size grows sub-linearly with data points, ensuring consistency of the resulting kernel linear regressor.

ABSTRACT

We analyze the size of the dictionary constructed from online kernel sparsification, using a novel formula that expresses the expected determinant of the kernel Gram matrix in terms of the eigenvalues of the covariance operator. Using this formula, we are able to connect the cardinality of the dictionary with the eigen-decay of the covariance operator. In particular, we show that under certain technical conditions, the size of the dictionary will always grow sub-linearly in the number of data points, and, as a consequence, the kernel linear regressor constructed from the resulting dictionary is consistent.

Motivation & Objective

• To understand the theoretical behavior of dictionary size in online kernel sparsification.
• To establish a connection between dictionary cardinality and the eigen-decay properties of the covariance operator.
• To prove that dictionary size grows sub-linearly under certain conditions.
• To demonstrate the consistency of the kernel linear regressor constructed from the resulting dictionary.
• To provide a theoretical foundation for online kernel methods with controlled memory usage.

Proposed method

• Derives a novel analytical formula expressing the expected determinant of the kernel Gram matrix in terms of the eigenvalues of the covariance operator.
• Uses this formula to analyze the expected growth rate of the dictionary in online kernel sparsification.
• Applies spectral theory to relate the eigen-decay of the covariance operator to dictionary size dynamics.
• Employs probabilistic and asymptotic analysis to establish sub-linear growth under technical assumptions.
• Connects the theoretical growth bounds to the consistency of the resulting kernel linear regressor.
• Leverages tools from random matrix theory and kernel methods to derive the main results.

Experimental results

Research questions

• RQ1 How does the size of the online kernel sparsification dictionary scale with the number of data points?
• RQ2 What is the relationship between the eigen-decay of the covariance operator and dictionary size?
• RQ3 Under what conditions does the dictionary size grow sub-linearly?
• RQ4 Can the consistency of the kernel linear regressor be theoretically guaranteed from the dictionary construction?
• RQ5 How does the expected determinant of the kernel Gram matrix relate to the dictionary's cardinality?

Key findings

• The size of the online kernel sparsification dictionary grows sub-linearly with the number of data points under specified technical conditions.
• The sub-linear growth is directly linked to the eigen-decay rate of the covariance operator.
• The kernel linear regressor constructed from the dictionary is consistent, meaning it converges to the optimal solution as data increases.
• The theoretical analysis is based on a novel formula connecting the expected determinant of the Gram matrix to the eigenvalues of the covariance operator.
• The results provide a rigorous foundation for memory-efficient online kernel learning with theoretical guarantees.
• The findings validate the practical efficiency of online kernel sparsification in maintaining performance while limiting dictionary size.