[Paper Review] Scalable Certified Segmentation via Randomized Smoothing

Marc Fischer, Maximilian Baader+1 more|arXiv (Cornell University)|Jul 1, 2021

Remote Sensing and LiDAR Applications4 citations

TL;DR

This paper introduces SegCertify, a scalable randomized smoothing method for certified semantic segmentation that handles large-scale image and point cloud segmentation tasks by abstaining from uncertain predictions and applying multiple-testing corrections. It achieves state-of-the-art certified accuracy—up to 88% on Cityscapes and 55% on Pascal Context—while maintaining high mIoU scores.

ABSTRACT

We present a new certification method for image and point cloud segmentation based on randomized smoothing. The method leverages a novel scalable algorithm for prediction and certification that correctly accounts for multiple testing, necessary for ensuring statistical guarantees. The key to our approach is reliance on established multiple-testing correction mechanisms as well as the ability to abstain from classifying single pixels or points while still robustly segmenting the overall input. Our experimental evaluation on synthetic data and challenging datasets, such as Pascal Context, Cityscapes, and ShapeNet, shows that our algorithm can achieve, for the first time, competitive accuracy and certification guarantees on real-world segmentation tasks. We provide an implementation at https://github.com/eth-sri/segmentation-smoothing.

Motivation & Objective

• Address the challenge of certifying robustness in semantic segmentation, where standard methods fail due to high dimensionality and multiple testing issues.
• Overcome the instability caused by individual misclassified pixels or points that undermine certification in large-scale segmentation tasks.
• Enable practical certification for real-world datasets like Cityscapes, Pascal Context, and ShapeNet, where prior methods were computationally infeasible.
• Introduce a framework that supports multiple threat models, including ℓ²-bounded perturbations and 3D rotations, through a plug-and-play design.
• Achieve high certification rates and competitive accuracy by combining abstention with statistical correction for multiple hypothesis testing.

Proposed method

• Apply randomized smoothing to segmentation models by adding isotropic Gaussian noise to input pixels or points and aggregating predictions across multiple forward passes.
• Use a novel abstention mechanism that skips classification for components (pixels/points) with high uncertainty, reducing the number of individual tests and improving statistical power.
• Incorporate Holm’s step-down procedure as a multiple-testing correction to maintain valid statistical guarantees across all components in the segmentation output.
• Leverage the fact that abstention reduces the number of tests, thereby improving the trade-off between robustness and certification rate without sacrificing correctness.
• Design SegCertify as a plug-and-play inference-time defense that works with any pre-trained base model, enabling certification for ℓ² and other perturbation types.
• Use Monte Carlo sampling (with n=1000 or n=10000) to estimate smoothed predictions and compute certified radii using confidence intervals derived from the binomial distribution.

Experimental results

Research questions

• RQ1 Can randomized smoothing be scaled to semantic segmentation tasks with high-dimensional outputs while maintaining statistical certification guarantees?
• RQ2 How can the adverse impact of unstable or ambiguous components (e.g., boundary pixels) be mitigated in certified segmentation?
• RQ3 What is the effect of multiple hypothesis testing on certification performance, and how can it be corrected without sacrificing robustness or efficiency?
• RQ4 Can the proposed method achieve competitive certified accuracy on real-world datasets such as Cityscapes and Pascal Context?
• RQ5 How does the inclusion of geometric features (e.g., surface normals in 3D point clouds) affect certification performance and robustness?

Key findings

• On the Cityscapes dataset, the method achieves a certified pixel accuracy of up to 88% at a radius of 0.250, with mIoU of 0.60, significantly outperforming baseline methods.
• On Pascal Context, the method reaches 55% certified pixel accuracy at radius 0.250 and mIoU of 0.20, demonstrating strong performance on complex, long-tail segmentation tasks.
• For 3D point cloud segmentation on ShapeNet, the method achieves 0.69 certified accuracy under 3D rotation attacks, showing generalization beyond ℓ² perturbations.
• The use of abstention reduces the number of tested components, enabling higher certification rates and improving the multiple-testing correction trade-off.
• Higher values of the abstention threshold τ lead to more abstentions but allow for larger certified radii and improved statistical reliability.
• The method maintains strong performance even with noisy or non-deterministic base models, as the smoothing process stabilizes predictions through averaging over multiple stochastic forward passes.