[Paper Review] Improved Autoregressive Modeling with Distribution Smoothing

Chenlin Meng, Jiaming Song+3 more|arXiv (Cornell University)|Mar 28, 2021

Generative Adversarial Networks and Image Synthesis40 references4 citations

TL;DR

This paper proposes a two-stage autoregressive modeling approach that improves sample quality by first smoothing the data distribution with noise, training an autoregressive model on the smoothed distribution, and then denoising samples via gradient-based or conditional modeling. The method significantly enhances sample quality on synthetic and real-world image datasets while maintaining competitive likelihoods, inspired by randomized smoothing in adversarial defense.

ABSTRACT

While autoregressive models excel at image compression, their sample quality is often lacking. Although not realistic, generated images often have high likelihood according to the model, resembling the case of adversarial examples. Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling. We first model a smoothed version of the data distribution, and then reverse the smoothing process to recover the original data distribution. This procedure drastically improves the sample quality of existing autoregressive models on several synthetic and real-world image datasets while obtaining competitive likelihoods on synthetic datasets.

Motivation & Objective

• To address the poor sample quality of autoregressive models on real-world image datasets despite high likelihoods.
• To mitigate challenges from high Lipschitz constants and manifold structure in data distributions.
• To reduce compounding errors in autoregressive generation by smoothing the data distribution.
• To adapt randomized smoothing—previously used in adversarial defense—for generative modeling.
• To enable high-fidelity sample generation without sacrificing likelihood performance.

Proposed method

• Apply symmetric, stationary noise (e.g., Gaussian) to the data distribution to create a smoothed version, reducing sharp density transitions.
• Train an autoregressive model on the smoothed distribution to learn a more stable and fully supported density.
• Reverse the smoothing process via denoising: either using gradient-based denoising or a conditional autoregressive model to recover original data from noisy inputs.
• Use the denoising step to generate high-quality samples by starting from a noise-robust, smoothed distribution.
• Allow flexible noise distributions beyond Gaussian, enabling broader applicability compared to prior denoising methods.
• Integrate the two-stage process into existing autoregressive models without modifying their architecture.

Experimental results

Research questions

• RQ1 Can randomized smoothing of the data distribution improve sample quality in autoregressive models without degrading likelihood?
• RQ2 How does smoothing the data distribution reduce the impact of high Lipschitz constants and manifold structure in real-world data?
• RQ3 Can a two-stage approach—learning on a smoothed distribution and denoising—outperform standard autoregressive modeling in sample fidelity?
• RQ4 Does this method generalize to other tasks such as image inpainting and denoising?
• RQ5 How does the choice of noise scale affect the trade-off between sample quality and likelihood?

Key findings

• The proposed method drastically improves sample quality on both synthetic and real-world image datasets, including CIFAR-10 and CelebA, while maintaining competitive likelihoods on synthetic data.
• On the 2D synthetic dataset, the method outperforms RealNVP in sample quality, with human observers preferring samples from the smoothed model.
• The method achieves a Fréchet Inception Distance (FID) of 10.2 on CIFAR-10, significantly improving over baseline autoregressive models.
• The approach enables high-quality image generation even when the original model has poor sample quality, demonstrating robustness to compounding errors.
• The method generalizes to image inpainting and denoising, showing that the denoising head can be reused for downstream tasks.
• The ablation study confirms that smoothing the data distribution—not the model—leads to the most significant gains in sample quality.