[Paper Review] E$^2$BoWs: An End-to-End Bag-of-Words Model via Deep Convolutional Neural Network

Xiaobin Liu, Shiliang Zhang+2 more|arXiv (Cornell University)|Sep 18, 2017

Advanced Image and Video Retrieval Techniques29 references3 citations

TL;DR

This paper proposes E²BoWs, an end-to-end Bag-of-Words model using deep convolutional neural networks to generate semantically discriminative visual words. By replacing the final fully connected layer with a convolutional layer to produce category-specific semantic feature maps, and applying a novel Bag-of-Words Layer with sparsity regularization, the model achieves state-of-the-art accuracy and efficiency in large-scale image retrieval, outperforming both traditional BoW methods and recent deep learning baselines on CIFAR-10, CIFAR-100, MIRFLICKR-25K, and NUS-WIDE.

ABSTRACT

Traditional Bag-of-visual Words (BoWs) model is commonly generated with many steps including local feature extraction, codebook generation, and feature quantization, etc. Those steps are relatively independent with each other and are hard to be jointly optimized. Moreover, the dependency on hand-crafted local feature makes BoWs model not effective in conveying high-level semantics. These issues largely hinder the performance of BoWs model in large-scale image applications. To conquer these issues, we propose an End-to-End BoWs (E$^2$BoWs) model based on Deep Convolutional Neural Network (DCNN). Our model takes an image as input, then identifies and separates the semantic objects in it, and finally outputs the visual words with high semantic discriminative power. Specifically, our model firstly generates Semantic Feature Maps (SFMs) corresponding to different object categories through convolutional layers, then introduces Bag-of-Words Layers (BoWL) to generate visual words for each individual feature map. We also introduce a novel learning algorithm to reinforce the sparsity of the generated E$^2$BoWs model, which further ensures the time and memory efficiency. We evaluate the proposed E$^2$BoWs model on several image search datasets including CIFAR-10, CIFAR-100, MIRFLICKR-25K and NUS-WIDE. Experimental results show that our method achieves promising accuracy and efficiency compared with recent deep learning based retrieval works.

Motivation & Objective

• To overcome the limitations of traditional Bag-of-Words (BoW) models that rely on hand-crafted local features and multi-stage processing.
• To close the semantic gap in image retrieval by integrating deep convolutional neural networks (DCNNs) into the BoW framework for better semantic representation.
• To enable end-to-end training of the BoW model with joint optimization of feature learning and visual word generation.
• To ensure high efficiency and sparsity in the resulting visual word representation for scalable inverted file indexing and fast retrieval.
• To improve generalization and discriminative power over existing deep learning-based retrieval methods, especially for semantic similarity search.

Proposed method

• The model replaces the final fully connected layer of GoogLeNet with a convolutional layer to generate n semantic feature maps (SFMs) corresponding to n object categories.
• Each semantic feature map is processed by a novel Bag-of-Words Layer (BoWL) that generates m sparse visual words per map, resulting in m×n total visual words.
• A thresholding mechanism is applied to enforce sparsity in the visual word representation, improving memory and computational efficiency.
• A three-component loss function is introduced: (1) classification loss for training, (2) contrastive loss to encourage similar images to share more visual words, and (3) sparsity regularization to maintain low visual word count per image.
• The model is trained end-to-end using backpropagation, enabling joint optimization of feature extraction, visual word generation, and sparsity.
• The framework supports inverted file indexing and TF-IDF weighting, enabling efficient large-scale image retrieval.

Experimental results

Research questions

• RQ1 Can a deep convolutional neural network be used to generate visual words end-to-end, replacing the traditional multi-stage BoW pipeline?
• RQ2 Does replacing hand-crafted local features with deep features improve semantic discriminative power in image retrieval?
• RQ3 Can a novel Bag-of-Words Layer with sparsity regularization maintain high retrieval efficiency while improving accuracy?
• RQ4 How does the proposed E²BoWs model generalize across diverse datasets compared to standard deep features like GoogLeNet?
• RQ5 What is the trade-off between retrieval accuracy and computational efficiency when using learned visual words versus binary codes?

Key findings

• On CIFAR-10, E²BoWs-B (binarized version) achieves a mean average precision (mAP) of 0.563, outperforming the best baseline BHC [23] with mAP 0.543.
• On NUS-WIDE, the E²BoWs model achieves mAP 0.599, significantly outperforming GoogLeNet features (best baseline mAP 0.594) and demonstrating superior generalization.
• The average number of visual words per image on MIRFLICKR-25K is only 43.6, indicating high sparsity and efficiency, compared to 480,000 operations required by binary codes in BHC.
• The model reduces retrieval complexity by leveraging inverted file indexing, requiring only 8.64 operations per image on CIFAR-10, compared to 8,294 operations for linear search with binary codes.
• The proposed three-component loss function enables faster convergence and improves both semantic discrimination and sparsity, as validated across all benchmark datasets.
• Visual word representations generated by E²BoWs show stronger semantic cues—e.g., correctly identifying 'people' in an eye image—demonstrating superior semantic understanding over BHC [23].