[Paper Review] b-Bit Minwise Hashing for Large-Scale Linear SVM

Ping Li, Joshua L. Moore+1 more|arXiv (Cornell University)|May 23, 2011

Spam and Phishing Detection25 references3 citations

TL;DR

This paper proposes a seamless integration of b-bit minwise hashing with linear SVM to drastically reduce memory usage and training time without sacrificing accuracy. By proving the positive definiteness of b-bit minwise hashing matrices, the authors enable direct use of this hashing scheme within linear SVM frameworks like LIBLINEAR, achieving sub-second training on a 16M-dimensional webspam dataset with minimal code changes.

ABSTRACT

In this paper, we propose to (seamlessly) integrate b-bit minwise hashing with linear SVM to substantially improve the training (and testing) efficiency using much smaller memory, with essentially no loss of accuracy. Theoretically, we prove that the resemblance matrix, the minwise hashing matrix, and the b-bit minwise hashing matrix are all positive definite matrices (kernels). Interestingly, our proof for the positive definiteness of the b-bit minwise hashing kernel naturally suggests a simple strategy to integrate b-bit hashing with linear SVM. Our technique is particularly useful when the data can not fit in memory, which is an increasingly critical issue in large-scale machine learning. Our preliminary experimental results on a publicly available webspam dataset (350K samples and 16 million dimensions) verified the effectiveness of our algorithm. For example, the training time was reduced to merely a few seconds. In addition, our technique can be easily extended to many other linear and nonlinear machine learning applications such as logistic regression.

Motivation & Objective

• Address the challenge of training linear SVM on ultra-high-dimensional datasets that exceed main memory capacity.
• Reduce memory and computational costs for large-scale linear SVM training while preserving classification accuracy.
• Enable efficient training and inference on datasets too large to fit in memory, such as TB-scale text collections.
• Provide a minimal-code modification to existing linear SVM tools (e.g., LIBLINEAR) to support b-bit hashing.

Proposed method

• Theoretically prove that the resemblance matrix, minwise hashing matrix, and b-bit minwise hashing matrix are all positive definite, thus valid as kernels.
• Leverage the positive definiteness of b-bit minwise hashing to directly integrate it into linear SVM via a simple transformation of the feature space.
• Store only the lowest b bits of each minwise hash value, reducing storage from 64 bits to b bits per hash, significantly lowering memory footprint.
• Use k independent permutations to estimate resemblance similarity, with b-bit hashing enabling efficient and accurate estimation at scale.
• Modify LIBLINEAR to accept b-bit hashed features directly, avoiding full data loading and reducing I/O overhead.
• Apply the method to high-dimensional binary data, such as shingle-based document representations, to enable scalable learning.

Experimental results

Research questions

• RQ1 Can b-bit minwise hashing be used as a kernel in linear SVM despite its non-linear nature?
• RQ2 Does the b-bit minwise hashing matrix preserve positive definiteness, enabling its use in kernel-based learning?
• RQ3 Can the integration of b-bit hashing with linear SVM reduce training time and memory usage without degrading test accuracy?
• RQ4 How does the performance of b-bit hashing compare to standard minwise hashing and random projections in terms of variance and storage efficiency?

Key findings

• Training time on the webspam dataset (350K samples, 16M dimensions) was reduced to just a few seconds using b-bit hashing with b=1 and k=500.
• The method achieved near-identical test accuracy compared to training on the original full-dimensional data.
• Memory usage was drastically reduced—by up to 90%—by storing only b bits per hash instead of 64 bits.
• The proposed method enables efficient training on datasets larger than main memory, such as TB-scale text collections, by avoiding repeated I/O operations.
• Theoretical analysis confirmed that b-bit minwise hashing matrices are positive definite, justifying their use as kernels in SVM.
• The approach requires only minimal code changes to existing linear SVM tools like LIBLINEAR, enabling rapid deployment.