[Paper Review] PQk-means: Billion-scale Clustering for Product-quantized Codes

Yusuke Matsui, Keisuke Ogaki+2 more|arXiv (Cornell University)|Sep 12, 2017

Advanced Image and Video Retrieval Techniques25 references3 citations

TL;DR

PQk-means is a memory-efficient, billion-scale clustering method that operates directly on product-quantized (PQ) codes instead of original high-dimensional vectors. By using a sparse voting scheme for center updates and a PQTable for fast assignment, it achieves 14-hour clustering of one billion 128D SIFT features with K=10⁵ using only 32 GB RAM—demonstrating competitive accuracy even with 32-bit PQ codes.

ABSTRACT

Data clustering is a fundamental operation in data analysis. For handling large-scale data, the standard k-means clustering method is not only slow, but also memory-inefficient. We propose an efficient clustering method for billion-scale feature vectors, called PQk-means. By first compressing input vectors into short product-quantized (PQ) codes, PQk-means achieves fast and memory-efficient clustering, even for high-dimensional vectors. Similar to k-means, PQk-means repeats the assignment and update steps, both of which can be performed in the PQ-code domain. Experimental results show that even short-length (32 bit) PQ-codes can produce competitive results compared with k-means. This result is of practical importance for clustering in memory-restricted environments. Using the proposed PQk-means scheme, the clustering of one billion 128D SIFT features with K = 10^5 is achieved within 14 hours, using just 32 GB of memory consumption on a single computer.

Motivation & Objective

• Address the high memory and computational cost of standard k-means on billion-scale, high-dimensional data.
• Enable efficient clustering on a single machine with limited memory (e.g., <32 GB RAM) for large-scale datasets.
• Maintain clustering accuracy while drastically reducing memory footprint through product quantization.
• Develop a fast and exact center update mechanism for PQ codes, which lack explicit averaging semantics.
• Ensure that original vectors can be approximately reconstructed from PQ codes post-clustering, preserving utility for downstream tasks.

Proposed method

• Compress high-dimensional feature vectors into short product-quantized (PQ) codes using product quantization (PQ).
• Perform clustering directly in the PQ-code domain, avoiding storage and computation on original high-dimensional vectors.
• Use a PQTable to accelerate the nearest-center assignment step by enabling fast lookup of the closest PQ code to each cluster center.
• Propose a sparse voting scheme for center updates: maintain a frequency histogram of assigned PQ codes per cluster and compute the mode of non-zero entries to determine the new center.
• Ensure exact center computation by focusing only on non-zero histogram entries, avoiding brute-force evaluation of all possible PQ code combinations.
• Iteratively alternate between assignment (using PQTable) and update (using sparse voting) steps until convergence.

Experimental results

Research questions

• RQ1 Can clustering be performed efficiently on billion-scale, high-dimensional data using only 32 GB of RAM on a single machine?
• RQ2 How does the accuracy of PQk-means compare to standard k-means and other large-scale clustering methods when using short PQ codes (e.g., 32-bit)?
• RQ3 Can the original high-dimensional vectors be approximately reconstructed from the PQ codes after clustering, preserving utility for downstream applications?
• RQ4 Is the sparse voting scheme for PQ code center updates both computationally efficient and exact compared to brute-force methods?
• RQ5 How does PQk-means scale in terms of runtime and memory usage across different large-scale datasets like SIFT1B and Deep1B?

Key findings

• PQk-means successfully clustered one billion 128D SIFT features with K=10⁵ in 14 hours using only 32 GB of RAM, while standard k-means would require 512 GB.
• With 32-bit PQ codes, PQk-means achieved clustering accuracy competitive with standard k-means and outperformed Bk-means, even when Bk-means used 64-bit codes.
• The sparse voting scheme enabled exact center computation with significant speedup, avoiding the computational burden of evaluating all possible PQ code combinations.
• Runtime performance on SIFT1B and Deep1B datasets was predictable and consistent, with clustering of 10⁹ vectors and K=10⁵ completed in 14 hours (SIFT1B) and 12 hours (Deep1B).
• PQk-means maintained high accuracy even with short 32-bit PQ codes, demonstrating practical viability for memory-constrained environments.
• Unlike IQ-means, PQk-means achieved significantly better accuracy on the YFCC100M dataset while remaining memory-efficient, and unlike Ak-means, it required far less memory by operating on compressed codes.