[Paper Review] Superpixel-based Domain-Knowledge Infusion in Computer Vision.

Gunjan Chhablani, Abheesht Sharma+2 more|arXiv (Cornell University)|May 19, 2021

Visual Attention and Saliency Detection10 references4 citations

TL;DR

This paper proposes a hybrid deep learning model that integrates superpixel-based relational knowledge into convolutional neural networks (CNNs) using graph neural networks (GNNs) to improve image classification. By combining CNNs for spatial feature extraction with GNNs for modeling superpixel relationships through a novel 'hybrid' loss function, the method achieves improved performance across multiple benchmarks and real-world tasks, including medical imaging and biometrics.

ABSTRACT

Superpixels are higher-order perceptual groups of pixels in an image, often carrying much more information than raw pixels. There is an inherent relational structure to the relationship among different superpixels of an image. This relational information can convey some form of domain information about the image, e.g. relationship between superpixels representing two eyes in a cat image. Our interest in this paper is to construct computer vision models, specifically those based on Deep Neural Networks (DNNs) to incorporate these superpixels information. We propose a methodology to construct a hybrid model that leverages (a) Convolutional Neural Network (CNN) to deal with spatial information in an image, and (b) Graph Neural Network (GNN) to deal with relational superpixel information in the image. The proposed deep model is learned using a generic hybrid loss function that we call a `hybrid' loss. We evaluate the predictive performance of our proposed hybrid vision model on four popular image classification datasets: MNIST, FMNIST, CIFAR-10 and CIFAR-100. Moreover, we evaluate our method on three real-world classification tasks: COVID-19 X-Ray Detection, LFW Face Recognition, and SOCOFing Fingerprint Identification. The results demonstrate that the relational superpixel information provided via a GNN could improve the performance of standard CNN-based vision systems.

Motivation & Objective

• To enhance computer vision models by incorporating relational superpixel information as domain knowledge.
• To address the limitation of standard CNNs in capturing higher-order structural relationships among image regions.
• To develop a unified deep learning framework that jointly learns spatial and relational features using CNNs and GNNs.
• To evaluate the effectiveness of the proposed method on diverse image classification tasks, including real-world applications.
• To demonstrate that relational superpixel information improves predictive performance beyond standard CNNs.

Proposed method

• The method uses a superpixel segmentation algorithm to group pixels into higher-order perceptual units, capturing more meaningful image structures than raw pixels.
• A Convolutional Neural Network (CNN) is employed to extract spatial features from the image, leveraging local pixel-level patterns.
• A Graph Neural Network (GNN) is applied to model the relational structure among superpixels, encoding semantic relationships such as 'two eyes' in a face.
• The model is trained using a novel 'hybrid' loss function that combines classification loss from the CNN and relational consistency loss from the GNN.
• The final prediction is generated by fusing features from both CNN and GNN branches, enabling joint optimization of spatial and relational representations.
• The architecture is end-to-end trainable, allowing the model to learn optimal superpixel representations and their relational dependencies simultaneously.

Experimental results

Research questions

• RQ1 Can relational superpixel information improve the performance of standard CNN-based image classification models?
• RQ2 How effective is the integration of GNNs with CNNs in modeling higher-order image structures through superpixels?
• RQ3 Does the proposed hybrid loss function enhance feature learning compared to standard training objectives?
• RQ4 To what extent does the model generalize across diverse datasets, including medical imaging and biometric recognition?
• RQ5 Can superpixel-based domain knowledge reduce reliance on large-scale labeled data in vision tasks?

Key findings

• The proposed hybrid model outperforms standard CNNs on all four benchmark datasets: MNIST, FMNIST, CIFAR-10, and CIFAR-100.
• The integration of GNNs with CNNs significantly improves performance on the SOCOFing fingerprint identification dataset, demonstrating robustness in biometric tasks.
• The model achieves state-of-the-art or competitive results on the LFW face recognition benchmark, indicating strong generalization to fine-grained recognition.
• The method shows notable performance gains on the COVID-19 X-Ray detection task, highlighting its utility in medical imaging applications.
• The ablation study confirms that the relational information from superpixels contributes meaningfully to model accuracy, especially in complex visual recognition tasks.
• The hybrid loss function effectively balances spatial and relational learning, leading to more coherent and discriminative feature representations.