[Paper Review] Image Classification with Rejection using Contextual Information

Filipe Condessa, José M. Bioucas‐Dias+3 more|arXiv (Cornell University)|Sep 3, 2015

Advanced Image and Video Retrieval Techniques14 references3 citations

TL;DR

This paper proposes a novel image classification framework that integrates rejection with multiscale contextual information using a graph-based energy minimization model. By combining multinomial logistic regression with spatial and multilevel similarity constraints, the method improves classification accuracy while intelligently rejecting uncertain or misclassified regions, especially effective in noisy, small, or unbalanced histopathology datasets with unknown classes.

ABSTRACT

We introduce a new supervised algorithm for image classification with rejection using multiscale contextual information. Rejection is desired in image-classification applications that require a robust classifier but not the classification of the entire image. The proposed algorithm combines local and multiscale contextual information with rejection, improving the classification performance. As a probabilistic model for classification, we adopt a multinomial logistic regression. The concept of rejection with contextual information is implemented by modeling the classification problem as an energy minimization problem over a graph representing local and multiscale similarities of the image. The rejection is introduced through an energy data term associated with the classification risk and the contextual information through an energy smoothness term associated with the local and multiscale similarities within the image. We illustrate the proposed method on the classification of images of H&E-stained teratoma tissues.

Motivation & Objective

• To address the challenge of robust image classification in histopathology when training data is small, unbalanced, noisy, or incomplete.
• To improve classification accuracy by incorporating spatial and multiscale contextual information into a rejection-aware framework.
• To reduce misclassification of unknown or ambiguous regions by enabling rejection of low-confidence predictions.
• To develop a unified model that embeds rejection within the classification process rather than applying it as a post-hoc rule.
• To evaluate the method on H&E-stained teratoma tissue images, where pixel-level ground truth is unavailable and class separability is low.

Proposed method

• The method models image classification as an energy minimization problem over a graph representing local and multiscale similarities between image patches.
• It uses a multinomial logistic regression model as the probabilistic classifier for initial class predictions.
• Rejection is introduced via a data term in the energy function that penalizes high classification risk, while smoothness terms enforce consistency based on contextual similarities.
• The energy function combines classification risk (via posterior probabilities) and spatial coherence (via patch similarities) to guide both classification and rejection decisions.
• The rejection threshold is determined by optimizing classification quality, balancing accuracy and rejection rate.
• The framework is trained and evaluated using a subset of 0.5% randomly selected image patches, with λ = 5 for regularization.

Experimental results

Research questions

• RQ1 Can combining contextual information with rejection improve classification accuracy in image classification tasks with limited or noisy training data?
• RQ2 How does the integration of spatial and multiscale similarities affect the performance of rejection-aware classifiers?
• RQ3 Does the proposed method outperform standard classification with rejection or context-only approaches in terms of classification quality and non-rejected accuracy?
• RQ4 To what extent does the method reduce misclassification of unknown or ambiguous tissue regions in histopathology images?
• RQ5 How does the performance vary across different image complexities and class distributions?

Key findings

• For images 1 and 2, the proposed method achieved higher classification quality than context-only classification, indicating effective rejection of misclassified regions while preserving correct classifications.
• For image 3, where context-only accuracy was already high, the method rejected a larger proportion of correctly classified samples, resulting in lower classification quality, highlighting the trade-off in complex cases.
• The performance of the proposed method was comparable to that of classification with rejection only using optimal rejected fraction, demonstrating that contextual information can substitute for manual threshold tuning.
• In image 1, the method significantly improved both classification quality and non-rejected accuracy by combining rejection and context.
• In image 2, class boundaries were effectively rejected, leading to high overall and class-specific rejection quality.
• In image 3, noisy training data led to excessive rejection of class boundaries, reducing classification quality below that of the baseline context-only model.