[Paper Review] Mean Local Group Average Precision (mLGAP): A New Performance Metric for Hashing-based Retrieval

Pak Lun Kevin Ding, Yikang Li+1 more|arXiv (Cornell University)|Nov 24, 2018

Advanced Image and Video Retrieval Techniques24 references4 citations

TL;DR

This paper proposes Mean Local Group Average Precision (mLGAP), a new metric for evaluating hashing-based retrieval that balances retrieval accuracy and hash code utilization. Unlike mAP, which favors high accuracy at the cost of code collisions, mLGAP penalizes poor code dispersion and better reflects real-world retrieval performance, as shown in experiments on CIFAR-10 and CIFAR-100 with improved code uniformity and consistent ranking reliability.

ABSTRACT

The research on hashing techniques for visual data is gaining increased attention in recent years due to the need for compact representations supporting efficient search/retrieval in large-scale databases such as online images. Among many possibilities, Mean Average Precision(mAP) has emerged as the dominant performance metric for hashing-based retrieval. One glaring shortcoming of mAP is its inability in balancing retrieval accuracy and utilization of hash codes: pushing a system to attain higher mAP will inevitably lead to poorer utilization of the hash codes. Poor utilization of the hash codes hinders good retrieval because of increased collision of samples in the hash space. This means that a model giving a higher mAP values does not necessarily do a better job in retrieval. In this paper, we introduce a new metric named Mean Local Group Average Precision (mLGAP) for better evaluation of the performance of hashing-based retrieval. The new metric provides a retrieval performance measure that also reconciles the utilization of hash codes, leading to a more practically meaningful performance metric than conventional ones like mAP. To this end, we start by mathematical analysis of the deficiencies of mAP for hashing-based retrieval. We then propose mLGAP and show why it is more appropriate for hashing-based retrieval. Experiments on image retrieval are used to demonstrate the effectiveness of the proposed metric.

Motivation & Objective

• To address the critical flaw in mAP, which prioritizes retrieval accuracy over hash code utilization, leading to high collision rates and poor code dispersion.
• To develop a performance metric that simultaneously evaluates retrieval accuracy and the uniformity of hash code distribution in the binary space.
• To demonstrate that high mAP values do not necessarily reflect better retrieval performance due to increased code collisions.
• To show that mLGAP provides a more reliable and practical evaluation by being robust to retrieval order changes and sensitive to code utilization.
• To validate that improved code uniformity preserves semantic and visual similarity in the hash space.

Proposed method

• Propose mLGAP as a new metric derived from mAP but incorporating local group average precision to reflect both accuracy and code dispersion.
• Mathematically analyze mAP's deficiency in penalizing code collisions and show that maximizing mAP leads to poor utilization of the hash space.
• Introduce a buffer zone mechanism in the loss function to allow similar samples to have a range of Hamming distances rather than forcing them to be identical, reducing collision risk.
• Design a modified objective function using L1 norm for Hamming distance and incorporate margin-based constraints (e.g., r₁, r₂) to control intra-class code proximity.
• Apply the new loss function to a deep neural network (e.g., DSH) with 3 convolutional and 2 fully connected layers, training on CIFAR-10 and CIFAR-100 with 12-bit codes.
• Set heuristic margin values: r₁ = k/12, r₂ = k/6, r₃ = k/6, r₄ = k/3 for CIFAR-100, and r₁ = k/6, r₂ = k/3 for CIFAR-10, with m = 2k to enforce minimum separation.

Experimental results

Research questions

• RQ1 Does mAP fail to reflect true retrieval performance due to its disregard for hash code utilization and collision rates?
• RQ2 Can a new metric be designed that balances retrieval accuracy and code dispersion to better reflect practical retrieval performance?
• RQ3 How does the proposed mLGAP metric perform compared to mAP when the same model is evaluated under different code distribution patterns?
• RQ4 To what extent does the new loss function improve code uniformity while maintaining or improving retrieval accuracy?
• RQ5 Is mLGAP robust to changes in retrieval order, unlike mAP, which is sensitive to such variations?

Key findings

• The proposed mLGAP metric effectively captures both retrieval accuracy and hash code utilization, offering a more balanced evaluation than mAP.
• Models trained with the new objective function show significantly improved uniformity in 12-bit hash code usage, as visualized in Fig. 8, reducing code collisions.
• On CIFAR-10, models with higher mLGAP values achieved better code dispersion, even when mAP was not the highest, indicating mLGAP's ability to detect better-performing systems.
• The new loss function with buffer zones reduced overfitting to identical codes for same-class samples, preserving semantic similarity in the hash space.
• mLGAP is robust to retrieval order changes, unlike mAP, which exhibits significant variance under different orderings due to collision sensitivity.
• Experiments on both CIFAR-10 and CIFAR-100 demonstrate that improved code uniformity correlates with better preservation of visual and semantic similarity in the hash space.