[Paper Review] Residual Pattern Learning for Pixel-wise Out-of-Distribution Detection in Semantic Segmentation

Yuyuan Liu, Choubo Ding+5 more|arXiv (Cornell University)|Nov 26, 2022

Remote-Sensing Image Classification4 citations

TL;DR

This paper proposes Residual Pattern Learning (RPL), a novel module that enhances pixel-wise out-of-distribution (OoD) detection in semantic segmentation without retraining the base model, thereby preserving in-distribution accuracy. By learning residual patterns of anomalies from intermediate features and using a positive energy loss and context-robust contrastive learning (CoroCL), RPL improves OoD detection by up to 10% in FPR and 7% in AuPRC across Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly datasets.

ABSTRACT

Semantic segmentation models classify pixels into a set of known (``in-distribution'') visual classes. When deployed in an open world, the reliability of these models depends on their ability not only to classify in-distribution pixels but also to detect out-of-distribution (OoD) pixels. Historically, the poor OoD detection performance of these models has motivated the design of methods based on model re-training using synthetic training images that include OoD visual objects. Although successful, these re-trained methods have two issues: 1) their in-distribution segmentation accuracy may drop during re-training, and 2) their OoD detection accuracy does not generalise well to new contexts (e.g., country surroundings) outside the training set (e.g., city surroundings). In this paper, we mitigate these issues with: (i) a new residual pattern learning (RPL) module that assists the segmentation model to detect OoD pixels without affecting the inlier segmentation performance; and (ii) a novel context-robust contrastive learning (CoroCL) that enforces RPL to robustly detect OoD pixels among various contexts. Our approach improves by around 10\% FPR and 7\% AuPRC the previous state-of-the-art in Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly datasets. Our code is available at: https://github.com/yyliu01/RPL.

Motivation & Objective

• Address the trade-off between in-distribution segmentation accuracy and OoD detection performance in retrained models.
• Overcome the context-robustness limitation of existing OoD detectors that fail under distribution shift (e.g., city to country scenes).
• Develop a plug-in module that enhances OoD detection without fine-tuning the base segmentation network.
• Improve sensitivity to small and subtle anomalies while minimizing false positives in inlier regions under varying contexts.
• Enable generalization of OoD detection across diverse real-world environments through context-aware representation learning.

Proposed method

• Introduce a residual pattern learning (RPL) module that operates on intermediate features of a frozen segmentation network to detect anomalies via induced uncertainty.
• Design a positive energy (PE) loss that focuses only on outlier energy scores, improving robustness to class imbalance in pixel-wise OoD detection.
• Propose context-robust contrastive learning (CoroCL) to align embeddings across different contexts, enhancing generalization to unseen environments.
• Train RPL using a projector head that matches the dimensionality of the RPL block’s output to enable effective contrastive optimization.
• Integrate RPL as a lightweight, trainable head on top of a pre-trained segmentation model, avoiding retraining of the main network.
• Use a two-stage training strategy: first train RPL with PE loss, then refine with CoroCL to improve context generalization.

Experimental results

Research questions

• RQ1 Can a plug-in module improve OoD detection without degrading in-distribution segmentation accuracy?
• RQ2 How can OoD detection be made robust to distribution shifts across different real-world contexts (e.g., urban to rural scenes)?
• RQ3 Does a positive energy loss outperform hinge-based energy losses in handling imbalanced inlier-outlier distributions in pixel-wise detection?
• RQ4 Can contrastive learning across diverse contexts improve the generalization of OoD detectors?
• RQ5 To what extent can RPL enhance existing SOTA methods like Meta-OoD and PEBAL?

Key findings

• RPL improves FPR by 10% and AuPRC by 7% over previous SOTA on Fishyscapes, Segment-Me-If-You-Can, and RoadAnomaly datasets.
• The positive energy loss improves AuPRC by 11.51% and reduces FPR by 1.99% on the SMIYC-Obstacle set compared to entropy-based optimization.
• Adding the dis-similarity (DS) regularisation increases AuPRC by 6.87% on FS-L&F, demonstrating improved outlier separation.
• CoroCL achieves 18.47% lower FPR and 12.12% higher AuPRC on SMIYC-Anomaly, and 12.92% lower FPR and 8.59% higher AuPRC on RoadAnomaly.
• RPL boosts Meta-OoD by 15% AuPRC on Fishyscapes Static and L&F, and improves PEBAL by over 70% AuPRC on SMIYC-Obstacle.
• Optimal performance is achieved when the projector dimension matches the RPL block output dimension (304), outperforming 512-dimensional projections by ~5% AuPRC.