[Paper Review] Image Synthesis with a Single (Robust) Classifier

Shibani Santurkar, Dimitris Tsipras+4 more|arXiv (Cornell University)|Jun 6, 2019

Adversarial Robustness in Machine Learning50 references47 citations

TL;DR

The paper shows that adversarially robust classifiers can be used as a universal tool for multiple image synthesis tasks by maximizing class scores via gradient ascent, without task-specific models. It demonstrates generation, inpainting, translation, super-resolution, and interactive manipulation using a single robust classifier.

ABSTRACT

We show that the basic classification framework alone can be used to tackle some of the most challenging tasks in image synthesis. In contrast to other state-of-the-art approaches, the toolkit we develop is rather minimal: it uses a single, off-the-shelf classifier for all these tasks. The crux of our approach is that we train this classifier to be adversarially robust. It turns out that adversarial robustness is precisely what we need to directly manipulate salient features of the input. Overall, our findings demonstrate the utility of robustness in the broader machine learning context. Code and models for our experiments can be found at https://git.io/robust-apps.

Motivation & Objective

• Demonstrate that a single robust classifier can tackle diverse image synthesis tasks beyond classification.
• Show that maximizing class scores on robust models yields realistic, semantically meaningful inputs.
• Highlight the role of adversarial robustness as a primitive for semantic image manipulation across tasks.
• Explore the limits and scalability of this minimal toolkit on large datasets.

Proposed method

• Train a ResNet-50 classifier with adversarial robustness on each dataset.
• Generate samples by gradient ascent on target class scores via projected gradient descent (PGD) starting from class-conditional seeds.
• For inpainting, optimize the image to maximize the true class score while penalizing changes outside the corrupted region.
• For image-to-image translation, train a classifier on source/target domains and translate by maximizing the target class score.
• For super-resolution, maximize the high-level class score while constraining changes to the upsampled low-resolution input.
• For interactive manipulation, optionally maximize activations in the representation layer to paint specific features via PGD.
• Discuss seed distributions, robustness settings, and minimal tuning needed.

Experimental results

Research questions

• RQ1 Can a single robust classifier serve as a general primitive for multiple image synthesis tasks?
• RQ2 How does gradient-based manipulation of robust models compare to task-specific generative methods in quality and diversity?
• RQ3 What are the capabilities and limitations of using class-score maximization for generation, inpainting, translation, and super-resolution?
• RQ4 How does robustness influence the semantic content and controllability of synthesized images?

Key findings

• A single robust classifier can generate realistic and diverse images for many classes using targeted PGD.
• On CIFAR-10, Inception Score for the proposed method is 8.4±0.1 (vs 11.2±0.2 for BigGAN and 9.22 for WGAN-GP).
• On ImageNet (full data), the method achieves an Inception Score of 259.0±4, outperforming BigGAN’s 331.9±4.9 and WGAN-GP’s 233.1±1 in this setup.
• The method yields perceptually plausible inpainting results and meaningful image-to-image translations (e.g., horse↔zebra, apple↔orange, summer↔winter).
• Super-resolution via robust-model gradients produces sharper, semantically meaningful reconstructions, outperforming plain upsampling in PSNR on the restricted ImageNet task (21.53 vs 21.30).
• Interactive sketch-to-image and feature-painting tools can be built by optimizing class scores or activations of robust models.