[Paper Review] ImageNet MPEG-7 Visual Descriptors - Technical Report

Frédéric Rayar|arXiv (Cornell University)|Feb 1, 2017

Advanced Image and Video Retrieval Techniques9 references3 citations

TL;DR

This technical report presents the extraction and public release of two MPEG-7 visual descriptors—Colour Layout Descriptor (CLD) and Edge Histogram Descriptor (EHD)—for the entire ImageNet dataset, comprising 14.197 million images across 21,841 synsets. The CLD captures spatial color distribution via DCT-based quantization of 8×8 blocks, while EHD encodes edge patterns using 4×4 block histograms; both are made publicly available in structured files for research use in content-based image retrieval and visual recognition.

ABSTRACT

ImageNet is a large scale and publicly available image database. It currently offers more than 14 millions of images, organised according to the WordNet hierarchy. One of the main objective of the creators is to provide to the research community a relevant database for visual recognition applications such as object recognition, image classification or object localisation. However, only a few visual descriptors of the images are available to be used by the researchers. Only SIFT-based features have been extracted from a subset of the collection. This technical report presents the extraction of some MPEG-7 visual descriptors from the ImageNet database. These descriptors are made publicly available in an effort towards open research.

Motivation & Objective

• To extend the utility of the ImageNet database by providing standardized MPEG-7 visual descriptors beyond existing SIFT features.
• To address the lack of comprehensive low-level visual descriptors in ImageNet, which currently offers only SIFT or attributes for small subsets.
• To support large-scale content-based image retrieval and visual recognition research by enabling fast, standardized feature extraction.
• To ensure interoperability and reproducibility in visual recognition by using standardized MPEG-7 descriptors.
• To make CLD and EHD features publicly available for the entire ImageNet dataset, enhancing open science in computer vision.

Proposed method

• ImageNet was processed in its entirety, with 21,841 synsets and 14,197,060 images analyzed, excluding 62 corrupted files.
• For CLD, each image was partitioned into 64 blocks of 8×8 pixels; representative colors were computed per block, followed by 2D DCT and zigzag reordering of low-frequency coefficients.
• The CLD descriptor was formed by quantizing the first 192 coefficients (3 channels × 64 coefficients), resulting in a 192-dimensional vector.
• For EHD, each image was divided into 16 blocks of 4×4 pixels; edge histograms were computed for five edge types (horizontal, vertical, diagonal, etc.) in each block.
• The EHD descriptor was constructed by concatenating the 80 values (16 blocks × 5 edge types), forming an 80-dimensional vector per image.
• Descriptors were stored in text files with one line per image, using the format: image_id;value_1;...;value_d, with d=192 for CLD and d=80 for EHD.

Experimental results

Research questions

• RQ1 Can standardized MPEG-7 visual descriptors be efficiently and consistently extracted at scale across the entire ImageNet dataset?
• RQ2 How do CLD and EHD descriptors compare in capturing spatial color and edge patterns for large-scale image recognition tasks?
• RQ3 To what extent can CLD and EHD enhance content-based image retrieval when used with existing ImageNet features like SIFT?
• RQ4 What is the feasibility and reliability of processing over 14 million images with MPEG-7 descriptors in a reproducible and publicly accessible manner?
• RQ5 Are there systematic differences in descriptor availability or quality across the ImageNet synsets, particularly for edge or color features?

Key findings

• The CLD descriptor was successfully computed for all 14,197,060 processed images, resulting in a 192-dimensional feature vector per image.
• The EHD descriptor was computed for all 14,197,060 images, resulting in an 80-dimensional feature vector per image.
• A total of 21,841 synset-specific descriptor files were generated, one per synset, with filenames in the format synset_id.ext (ext = cld or ehd).
• The complete datasets were released in two archives: ImageNet_MPEG-7_CLD_whole.zip (~1GB) and ImageNet_MPEG-7_EHD_whole.zip (~2GB).
• The processing pipeline ensured resolution invariance for CLD and consistent block-based histogram computation for EHD, enabling efficient and standardized feature extraction.
• The author notes that while most images have both CLD and EHD descriptors, no validation was performed to confirm this for all images.