[Paper Review] Norm-Explicit Quantization: Improving Vector Quantization for Maximum Inner Product Search

Xinyan Dai, Xiao Yan+3 more|arXiv (Cornell University)|Nov 12, 2019

Advanced Image and Video Retrieval Techniques27 references4 citations

TL;DR

This paper proposes Norm-Explicit Quantization (NEQ), a general framework that improves vector quantization for Maximum Inner Product Search (MIPS) by explicitly quantizing vector norms using dedicated codebooks, while reusing existing methods for direction quantization. NEQ significantly boosts recall performance across multiple VQ methods (PQ, OPQ, RQ, AQ) on diverse datasets, demonstrating that norm error has a greater impact on inner product accuracy than direction error, and that minimizing overall quantization error is not optimal for MIPS.

ABSTRACT

Vector quantization (VQ) techniques are widely used in similarity search for data compression, fast metric computation and etc. Originally designed for Euclidean distance, existing VQ techniques (e.g., PQ, AQ) explicitly or implicitly minimize the quantization error. In this paper, we present a new angle to analyze the quantization error, which decomposes the quantization error into norm error and direction error. We show that quantization errors in norm have much higher influence on inner products than quantization errors in direction, and small quantization error does not necessarily lead to good performance in maximum inner product search (MIPS). Based on this observation, we propose norm-explicit quantization (NEQ) --- a general paradigm that improves existing VQ techniques for MIPS. NEQ quantizes the norms of items in a dataset explicitly to reduce errors in norm, which is crucial for MIPS. For the direction vectors, NEQ can simply reuse an existing VQ technique to quantize them without modification. We conducted extensive experiments on a variety of datasets and parameter configurations. The experimental results show that NEQ improves the performance of various VQ techniques for MIPS, including PQ, OPQ, RQ and AQ.

Motivation & Objective

• To investigate whether minimizing total quantization error is optimal for Maximum Inner Product Search (MIPS), given that MIPS relies on inner products rather than Euclidean distances.
• To identify the relative impact of norm error versus direction error on inner product approximation accuracy in vector quantization.
• To develop a general-purpose framework that improves existing VQ techniques for MIPS without modifying their core codebook learning or inference procedures.
• To demonstrate that explicitly quantizing norms leads to better MIPS performance than minimizing overall quantization error, especially in high-dimensional, norm-diverse datasets.

Proposed method

• Decompose the quantization error into norm error (||x|| - ||x̃||) and angular error (1 - (xᵀx̃)/(||x|| ||x̃||)) to analyze their respective impacts on inner product accuracy.
• Propose Norm-Explicit Quantization (NEQ), which separates vector representation into norm and unit direction, and uses dedicated codebooks to quantize the norm explicitly.
• Reuse existing VQ techniques (e.g., PQ, RQ, AQ) unchanged for quantizing the unit-norm direction vector, preserving their learning and inference pipelines.
• Integrate NEQ with existing VQ methods by replacing the standard quantization of the full vector with separate norm and direction quantization, maintaining O(M) complexity for approximate inner product computation.
• Train codebooks using standard VQ optimization objectives for direction, while optimizing norm quantization independently to minimize norm error.
• Apply NEQ to multiple VQ baselines (PQ, OPQ, RQ, AQ) and evaluate on top-k MIPS using standard benchmarks (e.g., ImageNet, SIFT1M) with recall and query processing time as metrics.

Experimental results

Research questions

• RQ1 Does minimizing total quantization error lead to optimal performance in Maximum Inner Product Search (MIPS), given the unique properties of inner products?
• RQ2 Which component of quantization error—norm error or angular error—has a greater influence on inner product approximation accuracy in MIPS?
• RQ3 Can a general-purpose framework be designed to improve existing VQ techniques for MIPS without modifying their core learning or inference procedures?
• RQ4 Is it possible to achieve better MIPS recall by explicitly minimizing norm error, even if total quantization error increases slightly?

Key findings

• NEQ significantly improves recall performance across multiple VQ methods (PQ, OPQ, RQ, AQ) on diverse datasets, including ImageNet and SIFT1M, with consistent gains across different parameter settings.
• NE-RQ achieves higher recall than the state-of-the-art proximity graph method ip-NSW at the same query processing time on the ImageNet dataset, demonstrating NEQ's effectiveness in candidate generation.
• NE-PQ outperforms QUIP-cov(x), a specialized MIPS-aware VQ method, despite QUIP using a more complex codebook learning strategy that minimizes inner product error directly.
• NEQ reduces norm error substantially while slightly increasing total quantization error compared to baseline RQ, proving that minimizing total error is not optimal for MIPS.
• NEQ maintains low index size (e.g., 16-bit per item for NE-PQ) and achieves better recall than LSH-based methods (e.g., Norm-Range LSH, Simple-LSH) with the same or smaller index size.
• NEQ shows lower standard deviation in recall across multiple runs, indicating improved robustness of the codebook learning process.