[Paper Review] From Posterior Sampling to Meaningful Diversity in Image Restoration

Noa Cohen, H Manor+2 more|arXiv (Cornell University)|Oct 24, 2023

Advanced Image Processing Techniques4 citations

TL;DR

This paper challenges the common practice of posterior sampling for diverse image restoration, arguing it produces semantically redundant outputs due to heavy-tailed distributions. It proposes post-processing techniques and a lightweight diffusion-based method to generate meaningfully diverse restorations by maximizing semantic differences, significantly outperforming posterior sampling in user studies.

ABSTRACT

Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alternative solutions such as airships, birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Code and examples are available at https://noa-cohen.github.io/MeaningfulDiversityInIR.

Motivation & Objective

• To identify the conceptual flaw in using posterior sampling for diverse image restoration, where high-probability, semantically similar outputs dominate.
• To analyze why standard posterior sampling fails to represent the full range of plausible semantic solutions in tasks like inpainting and super-resolution.
• To develop practical post-processing and generation strategies that yield semantically diverse, perceptually meaningful image restorations.
• To evaluate these strategies through user studies and demonstrate their superiority over posterior sampling in conveying solution diversity.

Proposed method

• Proposes a post-processing sub-sampling strategy using farthest-point sampling (FPS) to select diverse outputs from a set of posterior samples, maximizing semantic distance in feature space.
• Introduces a hierarchical tree-based exploration structure that organizes restorations by perceptual similarity, enabling interactive, multi-level user exploration of diverse solutions.
• Develops a diffusion-based generation method with diversity guidance that modifies the reverse denoising process to favor semantically distinct samples, incurring negligible computational cost.
• Employs feature-space distances (e.g., using a pre-trained attribute predictor) to measure semantic similarity and guide both sub-sampling and tree construction.
• Uses nearest-neighbor partitioning to assign remaining images to representative samples in the hierarchical tree, ensuring fine-grained semantic grouping.
• Applies the method across multiple tasks—image inpainting, super-resolution, and compression artifact reduction—using established diffusion models like DDRM and RePaint.

Experimental results

Research questions

• RQ1 Why does posterior sampling in image restoration often fail to produce perceptually diverse outputs despite generating many samples?
• RQ2 What defines 'meaningful diversity' in image restoration, and how can it be quantitatively measured or achieved?
• RQ3 Can post-processing techniques like farthest-point sampling significantly improve the perceived diversity of outputs from existing generative models?
• RQ4 Is it possible to enable meaningfully diverse generation in diffusion-based models with minimal computational overhead?
• RQ5 How do users perceive and prefer meaningfully diverse restorations compared to posterior samples in real-world image restoration tasks?

Key findings

• Posterior sampling in image restoration is dominated by high-probability, semantically similar outputs—such as multiple sky completions in inpainting—resulting in low perceptual diversity.
• User studies show that techniques maximizing semantic distance between outputs, such as farthest-point sampling, are significantly preferred over posterior sampling for conveying solution variety.
• The proposed hierarchical tree structure enables intuitive, interactive exploration of diverse restorations, with users able to drill down into semantically similar alternatives.
• The diffusion-based diversity guidance method achieves meaningful diversity with only a negligible increase in inference time, making it practical for real-world deployment.
• Sub-sampling methods like FPS and uniformization outperform naive posterior sampling in both qualitative and quantitative evaluations across multiple datasets and tasks.
• The method successfully captures diverse semantic attributes (e.g., facial expressions, accessories, objects) that posterior sampling fails to represent, even when those attributes are rare in the posterior.