[Paper Review] Multi-scale Diffusion Denoised Smoothing

Jongheon Jeong, Jinwoo Shin|arXiv (Cornell University)|Oct 25, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes multi-scale diffusion denoised smoothing, a method that improves the accuracy-robustness trade-off in randomized smoothing by selectively applying denoising across multiple noise scales using a single diffusion model. By fine-tuning the diffusion model to produce consistent outputs when images are recoverable and diverse outputs otherwise, the approach achieves state-of-the-art certified robustness on ImageNet while maintaining high accuracy close to non-smoothed models.

ABSTRACT

Along with recent diffusion models, randomized smoothing has become one of a few tangible approaches that offers adversarial robustness to models at scale, e.g., those of large pre-trained models. Specifically, one can perform randomized smoothing on any classifier via a simple "denoise-and-classify" pipeline, so-called denoised smoothing, given that an accurate denoiser is available - such as diffusion model. In this paper, we present scalable methods to address the current trade-off between certified robustness and accuracy in denoised smoothing. Our key idea is to "selectively" apply smoothing among multiple noise scales, coined multi-scale smoothing, which can be efficiently implemented with a single diffusion model. This approach also suggests a new objective to compare the collective robustness of multi-scale smoothed classifiers, and questions which representation of diffusion model would maximize the objective. To address this, we propose to further fine-tune diffusion model (a) to perform consistent denoising whenever the original image is recoverable, but (b) to generate rather diverse outputs otherwise. Our experiments show that the proposed multi-scale smoothing scheme combined with diffusion fine-tuning enables strong certified robustness available with high noise level while maintaining its accuracy close to non-smoothed classifiers.

Motivation & Objective

• To address the long-standing accuracy-robustness trade-off in randomized smoothing, particularly in large-scale models.
• To explore whether multi-scale smoothing can improve certified robustness without sacrificing accuracy.
• To develop a fine-tuning strategy for diffusion models that enhances performance in the denoised smoothing pipeline.
• To provide a provably robust, scalable defense method compatible with pre-trained classifiers.

Proposed method

• Proposes multi-scale smoothing by aggregating multiple smoothed classifiers with different noise scales, leveraging the scale-free nature of diffusion models.
• Introduces a novel optimization objective to maximize collective certified robustness across scales.
• Employs a single diffusion model to efficiently generate denoised inputs at multiple noise levels.
• Fine-tunes the diffusion model to perform consistent denoising when the original image is recoverable, and diverse outputs otherwise.
• Uses a 1D grid-search optimization to compute optimal noise scale weights that maximize certified radius under sum-to-one constraint.
• Applies KKT conditions and analytical lemmas to derive conditions under which certain noise scales contribute zero weight to the aggregation.

Experimental results

Research questions

• RQ1 Can multi-scale smoothing improve certified robustness while preserving high accuracy in denoised smoothing?
• RQ2 What is the optimal way to fine-tune a diffusion model to enhance the performance of denoised smoothing?
• RQ3 Does selective application of smoothing across scales lead to better robustness-accuracy trade-offs than uniform smoothing?
• RQ4 Can a single diffusion model effectively support multiple noise scales in a scalable and efficient manner?
• RQ5 What representation or training objective in the diffusion model maximizes collective robustness across scales?

Key findings

• The proposed multi-scale smoothing scheme achieves state-of-the-art ℓ₂-certified robustness on ImageNet, outperforming prior methods.
• The method maintains test accuracy close to that of non-smoothed classifiers, significantly improving upon the accuracy-robustness trade-off.
• Fine-tuning the diffusion model to produce consistent outputs for recoverable images and diverse outputs otherwise leads to improved robustness.
• The optimization framework successfully computes noise scale weights that maximize the certified robustness radius.
• Empirical results show that the method provides valid robustness certificates and is scalable to large models.
• The approach demonstrates that input-dependent smoothing with proper calibration can be both robust and accurate, unlike previous brittle schemes.