[Paper Review] Reducing the Computational Complexity of Pseudoinverse for the Incremental Broad Learning System on Added Inputs

Hufei Zhu, Liu, Zhulin+2 more|arXiv (Cornell University)|Oct 17, 2019

Machine Learning and ELM7 references4 citations

TL;DR

This paper proposes an efficient implementation to reduce the computational complexity of pseudoinverse computation in the incremental Broad Learning System (BLS) for added inputs. By leveraging matrix inversion identities, it replaces costly $ q imes q $ matrix inversions with $ k imes k $ inversions when $ q > k $, achieving speedups of 1.35–1.57 on MNIST and 1.10–1.35 on NORB, with identical testing accuracy.

ABSTRACT

In this brief, we improve the Broad Learning System (BLS) [7] by reducing the computational complexity of the incremental learning for added inputs. We utilize the inverse of a sum of matrices in [8] to improve a step in the pseudoinverse of a row-partitioned matrix. Accordingly we propose two fast algorithms for the cases of q > k and q < k, respectively, where q and k denote the number of additional training samples and the total number of nodes, respectively. Specifically, when q > k, the proposed algorithm computes only a k * k matrix inverse, instead of a q * q matrix inverse in the existing algorithm. Accordingly it can reduce the complexity dramatically. Our simulations, which follow those for Table V in [7], show that the proposed algorithm and the existing algorithm achieve the same testing accuracy, while the speedups in BLS training time of the proposed algorithm over the existing algorithm are 1.24 - 1.30.

Motivation & Objective

• Address the high computational cost of pseudoinverse computation in incremental BLS when adding new input samples.
• Reduce the time complexity of matrix inversion and related multiplications in the BLS update process.
• Maintain the same testing accuracy while significantly accelerating training for large-scale and real-time applications.
• Develop a unified, efficient implementation applicable to both $ q > k $ and $ q imes k $ cases in incremental learning.
• Enable practical deployment of BLS in resource-constrained environments such as edge computing by improving training efficiency.

Proposed method

• Propose two distinct forms of efficient pseudoinverse computation: one for $ q > k $ and another for $ q imes k $, based on the structure of partitioned matrices.
• Utilize the matrix inversion lemma (Woodbury identity) to transform the original $ q imes q $ matrix inversion into a $ k imes k $ inversion when $ q > k $, drastically reducing computational cost.
• Derive new expressions (24b and 24c) that replace the original pseudoinverse computation step (15b), enabling faster matrix multiplications.
• For $ q > k $, the method reduces the number of floating-point operations by a factor of $ \frac{1}{2}(q/k)^3 $ for matrix inversion and improves two matrix multiplication operations by speed-up factors of $ ql/(2k^2) $ and $ q/k $, respectively.
• For $ q imes k $, the method accelerates one matrix multiplication by a factor of $ l/k $, reducing the computational load in this regime.
• Implement the new formulas within the incremental BLS framework, preserving the ridge regression approximation for output weight computation.

Experimental results

Research questions

• RQ1 Can the computational complexity of pseudoinverse computation in incremental BLS be reduced when adding new input samples?
• RQ2 What is the theoretical speedup gain achievable by replacing $ q imes q $ matrix inversion with $ k imes k $ inversion using matrix identities?
• RQ3 How does the proposed method affect the generalization performance and testing accuracy compared to the original BLS?
• RQ4 What are the practical speedup ratios in total training time across different datasets and input scaling regimes ($ q > k $ vs. $ q imes k $)?
• RQ5 Is the proposed method scalable and suitable for real-time or edge-computing applications where low latency is critical?

Key findings

• The proposed method achieves identical testing accuracy to the original BLS across all experiments, confirming that efficiency gains do not compromise performance.
• On the MNIST dataset, the proposed implementation achieves 1.35–1.57x speedup in total training time when $ q > k $, and 1.09–1.13x when $ q imes k $.
• On the NORB dataset, the speedup ranges from 1.10–1.35x when $ q > k $, and 1.07–1.09x when $ q imes k $, demonstrating consistent performance gains.
• The theoretical speedup for matrix inversion is $ \frac{1}{2}(q/k)^3 $, which is validated empirically in the numerical simulations.
• The method reduces the number of floating-point operations for matrix inversion and two matrix multiplications in the $ q > k $ case, significantly improving efficiency.
• The method is particularly effective in scenarios with large $ q $, where the computational savings are most pronounced, making it suitable for big data and real-time applications.