[Paper Review] Local Color Contrastive Descriptor for Image Classification

Sheng Guo, Weilin Huang+1 more|arXiv (Cornell University)|Aug 3, 2015

Advanced Image and Video Retrieval Techniques38 references3 citations

TL;DR

This paper proposes the Local Color Contrastive Descriptor (LCCD), a novel local image descriptor that enhances SIFT by integrating color contrast information using f-divergence between color distributions in spatially adjacent regions across multiple color channels. LCCD significantly improves SIFT performance, achieving state-of-the-art accuracy of 65.80% on PASCAL VOC 2007 and boosting DeCAF's accuracy from 40.94% to 49.68% on SUN397.

ABSTRACT

Image representation and classification are two fundamental tasks towards multimedia content retrieval and understanding. The idea that shape and texture information (e.g. edge or orientation) are the key features for visual representation is ingrained and dominated in current multimedia and computer vision communities. A number of low-level features have been proposed by computing local gradients (e.g. SIFT, LBP and HOG), and have achieved great successes on numerous multimedia applications. In this paper, we present a simple yet efficient local descriptor for image classification, referred as Local Color Contrastive Descriptor (LCCD), by leveraging the neural mechanisms of color contrast. The idea originates from the observation in neural science that color and shape information are linked inextricably in visual cortical processing. The color contrast yields key information for visual color perception and provides strong linkage between color and shape. We propose a novel contrastive mechanism to compute the color contrast in both spatial location and multiple channels. The color contrast is computed by measuring \emph{f}-divergence between the color distributions of two regions. Our descriptor enriches local image representation with both color and contrast information. We verified experimentally that it can compensate strongly for the shape based descriptor (e.g. SIFT), while keeping computationally simple. Extensive experimental results on image classification show that our descriptor improves the performance of SIFT substantially by combinations, and achieves the state-of-the-art performance on three challenging benchmark datasets. It improves recent Deep Learning model (DeCAF) [1] largely from the accuracy of 40.94% to 49.68% in the large scale SUN397 database. Codes for the LCCD will be available.

Motivation & Objective

• To address the limitation of shape-only descriptors like SIFT by incorporating robust color contrast information for improved image representation.
• To develop a method that captures local color contrast in a way that is invariant to photometric variations and sensitive to spatial structure.
• To demonstrate that color contrast, derived from neural mechanisms of visual perception, can effectively complement gradient-based features like SIFT.
• To achieve state-of-the-art performance on benchmark image classification datasets by combining LCCD with SIFT.
• To provide a computationally efficient yet powerful alternative to deep learning models for local image representation, especially in scenarios where high-level features may lose local detail.

Proposed method

• Leverages f-divergence to compute color contrast between two spatially adjacent image regions, modeling the statistical difference in their color distributions.
• Applies the contrastive mechanism across multiple color channels (e.g., RGB) to enhance discriminative power and robustness to illumination changes.
• Uses local spatial neighborhoods to compute region-based color contrasts, preserving spatial locality and reducing sensitivity to noise from single-pixel operations.
• Combines the LCCD descriptor with SIFT via feature concatenation and Fisher Vector (FV) encoding for improved classification performance.
• Employs a contrastive mechanism that explicitly models the relationship between neighboring regions rather than treating color features independently.
• Validates the method on standard benchmarks using standard evaluation protocols, including PASCAL VOC 2007 and SUN397, with FV encoding and cross-validation.

Experimental results

Research questions

• RQ1 Can color contrast, derived from biological visual processing, be effectively modeled as a local image descriptor to improve upon shape-based features?
• RQ2 How does f-divergence-based color contrast computation compare to traditional color histogram or SIFT-based color descriptors in terms of robustness and discriminative power?
• RQ3 To what extent can LCCD complement SIFT in improving image classification accuracy across diverse and challenging datasets?
• RQ4 Does integrating color contrast information help resolve ambiguities in scene categories with similar global structures but distinct local color patterns?
• RQ5 Can a simple, non-deep learning descriptor like LCCD outperform or significantly enhance recent deep learning models such as DeCAF in terms of accuracy and local feature discrimination?

Key findings

• The LCCD+SIFT combination achieves a state-of-the-art accuracy of 65.80% on the PASCAL VOC 2007 benchmark, outperforming individual SIFT by 4% and prior combinations like LCS+SIFT by a significant margin.
• On the SUN397 dataset, LCCD+SIFT improves DeCAF’s accuracy from 40.94% to 49.68%, demonstrating strong complementary power even against advanced deep learning models.
• LCCD achieves substantial performance gains on ambiguous categories—such as 'swimming pool outdoor' and 'thrift shop'—where SIFT alone struggles due to similar global structures.
• The method improves classification accuracy by up to 30% in challenging categories like 'swimming pool outdoor' and 20% in 'thrift shop', highlighting its ability to capture fine-grained local color differences.
• LCCD outperforms existing color-enhanced descriptors such as Color SIFT (42.00% mAP) and LCS+SIFT (63.90% mAP), confirming its superior modeling of spatial and chromatic contrast.
• The improvement is most pronounced in categories with high visual ambiguity, where local color contrast provides critical discriminative cues that SIFT alone misses.