[Paper Review] Finite Dictionary Variants of the Diffusion KLMS Algorithm

Rangeet Mitra, Vimal Bhatia|arXiv (Cornell University)|Sep 9, 2015

Advanced Adaptive Filtering Techniques3 citations

TL;DR

This paper proposes two finite-dictionary variants of the diffusion kernel least mean squares (KLMS) algorithm—quantised diffusion KLMS (QDKLMS) and fixed budget diffusion KLMS (FBDKLMS)—to address the infinite storage requirement of traditional diffusion KLMS. By employing online vector quantisation and dynamic dictionary pruning, both algorithms achieve faster convergence and reduced memory usage while maintaining high performance in distributed non-linear learning over networks with limited node storage.

ABSTRACT

The diffusion based distributed learning approaches have been found to be a viable solution for learning over linearly separable datasets over a network. However, approaches till date are suitable for linearly separable datasets and need to be extended to scenarios in which we need to learn a non-linearity. In such scenarios, the recently proposed diffusion kernel least mean squares (KLMS) has been found to be performing better than diffusion least mean squares (LMS). The drawback of diffusion KLMS is that it requires infinite storage for observations (also called dictionary). This paper formulates the diffusion KLMS in a fixed budget setting such that the storage requirement is curtailed while maintaining appreciable performance in terms of convergence. Simulations have been carried out to validate the two newly proposed algorithms named as quantised diffusion KLMS (QDKLMS) and fixed budget diffusion KLMS (FBDKLMS) against KLMS, which indicate that both the proposed algorithms deliver better performance as compared to the KLMS while reducing the dictionary size storage requirement.

Motivation & Objective

• Address the critical limitation of infinite dictionary storage in diffusion KLMS for distributed non-linear learning.
• Enable practical deployment of diffusion KLMS in resource-constrained networks such as wireless sensor networks.
• Develop finite-dictionary mechanisms that preserve convergence speed and accuracy while reducing per-node memory and computational load.
• Provide scalable, robust solutions for distributed kernel adaptive filtering in non-linear, non-separable data scenarios.
• Ensure compatibility with real-world systems by bounding dictionary size without sacrificing learning performance.

Proposed method

• Introduce QDKLMS, which uses online vector quantisation to dynamically update and limit the size of the local dictionary at each node.
• Implement FBDKLMS, which maintains a fixed dictionary budget by recursively pruning the least significant dictionary elements based on their contribution to the hypothesis.
• Apply the kernel trick to map data into a reproducing kernel Hilbert space (RKHS) for non-linear decision boundary estimation.
• Integrate the diffusion strategy to allow nodes to combine local estimates with weighted information from neighbors, enhancing convergence and robustness.
• Use significance-based pruning in FBDKLMS to remove dictionary elements with minimal impact on the overall hypothesis, ensuring adaptability and efficiency.
• Apply Silverman’s rule for kernel width selection and fix the step-size at 0.1 for consistent comparison across algorithms.

Experimental results

Research questions

• RQ1 Can the infinite dictionary requirement of diffusion KLMS be effectively mitigated in a distributed network setting without degrading learning performance?
• RQ2 How do QDKLMS and FBDKLMS compare in terms of convergence speed, steady-state error, and dictionary size under non-linear, non-separable data distributions?
• RQ3 What is the impact of network size scaling on the performance and memory efficiency of the proposed algorithms?
• RQ4 Can finite-dictionary variants maintain or improve convergence speed compared to standard diffusion KLMS while reducing storage overhead?
• RQ5 How does the significance-based pruning mechanism in FBDKLMS compare to quantisation-based updating in QDKLMS in terms of memory efficiency and error floor?

Key findings

• QDKLMS and FBDKLMS both achieve faster convergence than standard diffusion KLMS in non-stationary channel, crescent moon, and spiral datasets.
• FBDKLMS consistently converges to a lower dictionary size than QDKLMS, especially in smaller networks, while maintaining comparable or better performance.
• In the non-stationary equalisation channel, both proposed algorithms show faster transient response and lower steady-state mean square error (MSE) than KLMS.
• For the crescent moon and spiral datasets, FBDKLMS achieves a lower error floor and smaller dictionary size than QDKLMS as network size increases.
• With increasing network size, both QDKLMS and FBDKLMS exhibit a decreasing trend in converged MSE floor, indicating scalability and robustness.
• Simulations averaged over 200 Monte Carlo iterations confirm that FBDKLMS is more memory-efficient than QDKLMS, making it preferable for practical deployment.