[Paper Review] Generative Adversarial Image Synthesis with Decision Tree Latent Controller

Takuhiro Kaneko, Kaoru Hiramatsu+1 more|arXiv (Cornell University)|May 27, 2018

Generative Adversarial Networks and Image Synthesis3 citations

TL;DR

This paper proposes DTLC-GAN, a generative adversarial network that learns hierarchically interpretable representations via a decision tree latent controller (DTLC), enabling coarse-to-fine image generation without detailed supervision. By integrating hierarchical conditional mutual information regularization and a curriculum learning strategy, DTLC-GAN disentangles semantic features across layers, achieving state-of-the-art inception scores (8.80 on CIFAR-10) and effective image retrieval with improved attribute-specific SSIM scores.

ABSTRACT

This paper proposes the decision tree latent controller generative adversarial network (DTLC-GAN), an extension of a GAN that can learn hierarchically interpretable representations without relying on detailed supervision. To impose a hierarchical inclusion structure on latent variables, we incorporate a new architecture called the DTLC into the generator input. The DTLC has a multiple-layer tree structure in which the ON or OFF of the child node codes is controlled by the parent node codes. By using this architecture hierarchically, we can obtain the latent space in which the lower layer codes are selectively used depending on the higher layer ones. To make the latent codes capture salient semantic features of images in a hierarchically disentangled manner in the DTLC, we also propose a hierarchical conditional mutual information regularization and optimize it with a newly defined curriculum learning method that we propose as well. This makes it possible to discover hierarchically interpretable representations in a layer-by-layer manner on the basis of information gain by only using a single DTLC-GAN model. We evaluated the DTLC-GAN on various datasets, i.e., MNIST, CIFAR-10, Tiny ImageNet, 3D Faces, and CelebA, and confirmed that the DTLC-GAN can learn hierarchically interpretable representations with either unsupervised or weakly supervised settings. Furthermore, we applied the DTLC-GAN to image-retrieval tasks and showed its effectiveness in representation learning.

Motivation & Objective

• To develop a deep generative model that learns hierarchically interpretable representations without requiring detailed annotations or supervision.
• To enable coarse-to-fine image generation by structuring latent variables through a decision tree architecture that enforces hierarchical inclusion.
• To discover disentangled semantic features across multiple layers—between layers, within layers, and across control targets—using only class-level or weak supervision.
• To improve representation learning for downstream tasks such as image retrieval by leveraging the disentangled, hierarchical latent space.

Proposed method

• Introduces the Decision Tree Latent Controller (DTLC), a multi-layer tree structure where child node codes are activated only when their parent nodes are ON, enforcing hierarchical inclusion in the latent space.
• Proposes Hierarchical Conditional Mutual Information (HCMI) regularization to encourage disentanglement of semantic features across layers by maximizing information gain at each level.
• Designs a curriculum learning method that trains the model layer-by-layer, starting from the top layer and progressively adding lower layers to stabilize training and improve disentanglement.
• Uses WGAN-GP with gradient penalty for training stability and evaluates performance using inception score and SSIM-based retrieval metrics.
• Employs auxiliary networks $Q_l$ to predict latent codes $\hat{\bm{c}}_l$ for downstream image retrieval tasks.
• Applies the model to both discrete and continuous latent codes, demonstrating category-specific control in 3D Faces and CelebA datasets.

Experimental results

Research questions

• RQ1 Can a GAN learn hierarchically disentangled representations without detailed supervision, relying only on class-level labels?
• RQ2 Can a decision tree-structured latent controller enforce hierarchical inclusion and enable coarse-to-fine image generation?
• RQ3 Does hierarchical conditional mutual information regularization improve disentanglement of semantic features across multiple layers?
• RQ4 Can the proposed curriculum learning method stabilize training and enhance the discovery of interpretable representations layer-by-layer?
• RQ5 Does the learned hierarchical latent space improve performance in image retrieval tasks?

Key findings

• DTLC-GAN achieves an inception score of 8.80 on CIFAR-10 using only class labels, outperforming prior methods under weak supervision.
• On the CelebA dataset, the model improves attribute-specific SSIM scores from 0.150 at the first layer to 0.265 for glasses and 0.326 for smiling at the third layer, indicating better alignment with semantic attributes.
• The model successfully generates diverse, high-fidelity images with controlled variations: e.g., changing second-layer codes affects broad attributes (e.g., glasses type), while lower-layer codes refine details (e.g., rim thickness).
• In 3D Faces, DTLC-GAN with continuous codes learns pose-specific semantic features conditioned on higher-level categorical codes, unlike flat InfoGANs where codes are independent.
• The curriculum learning method enables stable training and effective discovery of hierarchical representations, with performance improving as deeper layers are added.
• Image retrieval using hierarchical codes shows that deeper layers yield higher concordance with query attributes, confirming the utility of the disentangled latent space.