[Paper Review] Improving Efficiency of SVM k-fold Cross-validation by Alpha Seeding

Zeyi Wen, Bin Li+4 more|UWA Profiles and Research Repository (UWA)|Nov 23, 2016

Anomaly Detection Techniques and Applications17 references6 citations

TL;DR

This paper proposes three alpha seeding algorithms to accelerate k-fold cross-validation in SVMs by reusing support vector weights (alpha values) from the h-th fold to initialize the (h+1)-th fold. By exploiting the similarity between consecutive folds—where over 80% of training instances are shared—the method significantly reduces training time while maintaining identical accuracy to standard LibSVM cross-validation, achieving up to 32× speedup on large k (k=100).

ABSTRACT

The k-fold cross-validation is commonly used to evaluate the effectiveness of SVMs with the selected hyper-parameters. It is known that the SVM k-fold cross-validation is expensive, since it requires training k SVMs. However, little work has explored reusing the h-th SVM for training the (h+1)-th SVM for improving the efficiency of k-fold cross-validation. In this paper, we propose three algorithms that reuse the h-th SVM for improving the efficiency of training the (h+1)-th SVM. Our key idea is to efficiently identify the support vectors and to accurately estimate their associated weights (also called alpha values) of the next SVM by using the previous SVM. Our experimental results show that our algorithms are several times faster than the k-fold cross-validation which does not make use of the previously trained SVM. Moreover, our algorithms produce the same results (hence same accuracy) as the k-fold cross-validation which does not make use of the previously trained SVM.

Motivation & Objective

• Address the high computational cost of k-fold cross-validation in SVMs, which requires training k separate models.
• Overcome the inefficiency of retraining from scratch for each fold, especially when consecutive folds share substantial training data.
• Develop a method to reuse the alpha values (support vector weights) from the h-th fold to initialize the (h+1)-th fold, improving training convergence speed.
• Ensure the proposed method maintains the same model accuracy as standard k-fold cross-validation while drastically reducing training time.
• Design an efficient initialization strategy that avoids performance degradation due to poor alpha value initialization.

Proposed method

• Propose three progressive algorithms—ATO, MIR, and SIR—based on alpha seeding, where the alpha values from the h-th fold are used to initialize the (h+1)-th fold's optimization.
• Use the optimality conditions of SVMs to estimate optimal alpha values for the (h+1)-th fold based on the h-th fold's solution.
• Apply a shared-instance assumption: alpha values for overlapping training instances between folds are preserved or adjusted based on kernel similarity.
• Introduce a replacement mechanism where outdated support vectors in the h-th fold are replaced by new instances in the (h+1)-th fold, with dynamic alpha reassignment.
• Implement a fast initialization step that avoids full reinitialization, focusing only on newly added or removed instances to reduce overhead.
• Use Sequential Minimal Optimization (SMO) as the underlying solver, though the method is generalizable to other SVM solvers.

Experimental results

Research questions

• RQ1 Can alpha values from a previously trained SVM fold be effectively reused to accelerate training of the next fold in k-fold cross-validation?
• RQ2 How can alpha values be efficiently and accurately initialized for the (h+1)-th fold using the h-th fold's solution without degrading convergence or accuracy?
• RQ3 What is the impact of varying k on the performance gain from alpha seeding, especially when k is large (e.g., k=100)?
• RQ4 Does alpha seeding preserve the same model accuracy as standard k-fold cross-validation, or does it introduce approximation errors?
• RQ5 Can the proposed method achieve significant speedups over LibSVM while maintaining identical results across diverse datasets?

Key findings

• The proposed SIR algorithm achieves up to 32 times faster training than LibSVM in k-fold cross-validation when k=100, particularly on the Madelon dataset.
• For k=10, the SIR algorithm is several times faster than LibSVM, with consistent performance improvements across all tested datasets.
• The SIR algorithm maintains identical accuracy to LibSVM across all datasets, confirming that the speedup does not compromise model quality.
• The method is especially effective when k > 2, as consecutive folds share a large proportion of training instances (over 80% for k=10), enabling effective reuse.
• The alpha seeding initialization process is efficient enough to offset any overhead, resulting in net time savings even with large k.
• The experimental results confirm that the proposed method outperforms LibSVM in both speed and scalability, particularly for large k and high-dimensional datasets.