[Paper Review] Training Logistic Regression and SVM on 200GB Data Using b-Bit Minwise Hashing and Comparisons with Vowpal Wabbit (VW)

Ping Li, Anshumali Shrivastava+1 more|arXiv (Cornell University)|Aug 15, 2011

Face and Expression Recognition31 references3 citations

TL;DR

This paper proposes b-bit minwise hashing as a highly efficient method for training logistic regression and SVM on massive datasets, demonstrating that it achieves superior accuracy compared to Vowpal Wabbit (VW) hashing at significantly lower storage costs. Using a 200GB expanded rcv1 dataset, the authors show that b-bit hashing with only 30 hashed values per example matches VW's performance using 16,384 values, while preprocessing costs are manageable and GPU-accelerated.

ABSTRACT

We generated a dataset of 200 GB with 10^9 features, to test our recent b-bit minwise hashing algorithms for training very large-scale logistic regression and SVM. The results confirm our prior work that, compared with the VW hashing algorithm (which has the same variance as random projections), b-bit minwise hashing is substantially more accurate at the same storage. For example, with merely 30 hashed values per data point, b-bit minwise hashing can achieve similar accuracies as VW with 2^14 hashed values per data point. We demonstrate that the preprocessing cost of b-bit minwise hashing is roughly on the same order of magnitude as the data loading time. Furthermore, by using a GPU, the preprocessing cost can be reduced to a small fraction of the data loading time. Minwise hashing has been widely used in industry, at least in the context of search. One reason for its popularity is that one can efficiently simulate permutations by (e.g.,) universal hashing. In other words, there is no need to store the permutation matrix. In this paper, we empirically verify this practice, by demonstrating that even using the simplest 2-universal hashing does not degrade the learning performance.

Motivation & Objective

• To evaluate b-bit minwise hashing on a large-scale dataset (200GB expanded rcv1) to assess its performance in real-world machine learning scenarios.
• To compare b-bit minwise hashing directly with Vowpal Wabbit (VW) hashing in terms of test accuracy and storage efficiency.
• To validate the use of 2-universal hashing as a practical and effective alternative to storing full permutations in minwise hashing.
• To demonstrate that preprocessing costs for b-bit minwise hashing are negligible in practice, especially when amortized over multiple training runs.
• To provide a reproducible experimental setup using LIBLINEAR and hashed data to enable easy validation by other researchers.

Proposed method

• The authors expanded the rcv1 dataset by including original features, all pairwise combinations, and 1/30 of the 3-way feature combinations to create a 200GB dataset in LibSVM format.
• b-bit minwise hashing was applied using 2-universal hashing functions to simulate permutations, replacing the need to store full permutation mappings.
• The hashing process maps each non-zero feature index to a new location using linear congruential functions: $ h_j(t) = \{c_{1,j} + c_{2,j} \cdot t \ \text{mod}\ p\} \ \text{mod}\ D $, where $ p > D $ is prime.
• For each data point, the minimum hashed value across $ k $ permutations is recorded, forming a compact $ k $-dimensional representation.
• The resulting hashed data were used to train logistic regression and SVM models using LIBLINEAR, a widely used solver for large-scale linear models.
• Performance was evaluated using test accuracy, with comparisons made across varying numbers of hashed values ($ k $) between b-bit minwise hashing and VW hashing.

Experimental results

Research questions

• RQ1 Can b-bit minwise hashing achieve comparable or better test accuracy than Vowpal Wabbit hashing when using significantly fewer hashed values?
• RQ2 What is the impact of using 2-universal hashing instead of perfect permutations on the learning performance of b-bit minwise hashing?
• RQ3 How do preprocessing and data loading times compare in large-scale learning pipelines, and is preprocessing a bottleneck?
• RQ4 Can b-bit minwise hashing be effectively used with standard solvers like LIBLINEAR for training on datasets up to 200GB in size?
• RQ5 Is the one-time preprocessing cost of b-bit minwise hashing justified by its reusability across multiple training experiments and hyperparameter tuning tasks?

Key findings

• With only 30 hashed values per data point, b-bit minwise hashing achieved test accuracy comparable to Vowpal Wabbit (VW) using 16,384 hashed values.
• The preprocessing cost for b-bit minwise hashing was on the same order of magnitude as data loading time for the 200GB dataset, even without GPU acceleration.
• Using a GPU, preprocessing time was reduced to less than 1/7th of the data loading time, making it a minor bottleneck.
• The use of 2-universal hashing did not degrade model performance, as shown by overlapping test accuracy curves between permutations and 2-universal hashing on the webspam dataset.
• The same preprocessed data can be reused across multiple training runs, including cross-validation with different C values or data splits, significantly reducing overall computational cost.
• b-bit minwise hashing outperforms VW hashing in terms of variance reduction, confirming theoretical expectations that it has lower variance than random projections (which VW emulates).