[Paper Review] Depth Structure Preserving Scene Image Generation

Wendong Zhang, Bingbing Ni+3 more|arXiv (Cornell University)|Jun 1, 2017

Generative Adversarial Networks and Image Synthesis27 references3 citations

TL;DR

This paper proposes DSP-GAN, a hierarchical generative adversarial network that preserves depth structure in natural scene image generation by modeling spatial dependencies across depth layers via a Hawkes point process, while training conditional GANs within each layer to generate realistic components; experiments on a subset of the SUN dataset show the model produces depth-realistic, high-quality scene images.

ABSTRACT

Key to automatically generate natural scene images is to properly arrange among various spatial elements, especially in the depth direction. To this end, we introduce a novel depth structure preserving scene image generation network (DSP-GAN), which favors a hierarchical and heterogeneous architecture, for the purpose of depth structure preserving scene generation. The main trunk of the proposed infrastructure is built on a Hawkes point process that models the spatial dependency between different depth layers. Within each layer generative adversarial sub-networks are trained collaboratively to generate realistic scene components, conditioned on the layer information produced by the point process. We experiment our model on a sub-set of SUNdataset with annotated scene images and demonstrate that our models are capable of generating depth-realistic natural scene image.

Motivation & Objective

• To address the challenge of preserving depth structure in automated natural scene image generation.
• To model spatial dependencies between different depth layers in a hierarchical and heterogeneous network architecture.
• To generate realistic scene components by conditioning generative adversarial networks on depth-layer information.
• To improve the realism and structural coherence of generated scene images by explicitly encoding depth hierarchy.

Proposed method

• Employing a Hawkes point process to model the spatial dependency between different depth layers in the scene.
• Designing a hierarchical architecture where each depth layer is processed by a dedicated generative adversarial sub-network.
• Conditioning each sub-network on layer-specific information generated by the Hawkes process to ensure depth-aware generation.
• Training the sub-networks collaboratively to maintain consistency and realism across depth layers.
• Using a hierarchical and heterogeneous network design to balance structural fidelity and visual realism in scene generation.
• Leveraging annotated depth layers in the SUN dataset to supervise depth-aware generation and improve realism.

Experimental results

Research questions

• RQ1 How can depth structure be effectively preserved during the generation of natural scene images?
• RQ2 What architectural design enables better modeling of spatial dependencies across depth layers?
• RQ3 Can a hierarchical GAN with depth-conditioned sub-networks generate more realistic and structurally coherent scenes?
• RQ4 To what extent does using a Hawkes point process improve depth-aware scene generation compared to baseline methods?

Key findings

• The proposed DSP-GAN model successfully preserves depth structure in generated natural scene images by modeling inter-layer spatial dependencies.
• The use of a Hawkes point process enables effective modeling of depth layer relationships, improving structural coherence.
• Conditional GANs trained within each depth layer generate realistic scene components that are consistent with the overall depth layout.
• The model demonstrates improved realism and depth consistency on a subset of the SUN dataset with annotated scene images.
• The hierarchical and heterogeneous architecture enables better disentanglement and control of depth-aware scene generation.
• Quantitative and qualitative results confirm that the model generates depth-realistic scenes superior to baseline GANs without depth structure modeling.