[Paper Review] Generative Neurosymbolic Machines

Jindong Jiang, Sungjin Ahn|arXiv (Cornell University)|Oct 23, 2020

Generative Adversarial Networks and Image Synthesis41 references9 citations

TL;DR

This paper proposes Generative Neurosymbolic Machines (GNM), a hierarchical generative model that combines distributed latent variables for flexible density modeling with symbolic, structured representations for interpretability and compositionality. By introducing a two-layer latent hierarchy and the StructDRAW autoregressive prior, GNM achieves state-of-the-art performance in both structured representation accuracy and image generation quality, significantly outperforming prior models on ARROW and MNIST-10 benchmarks.

ABSTRACT

Reconciling symbolic and distributed representations is a crucial challenge that can potentially resolve the limitations of current deep learning. Remarkable advances in this direction have been achieved recently via generative object-centric representation models. While learning a recognition model that infers object-centric symbolic representations like bounding boxes from raw images in an unsupervised way, no such model can provide another important ability of a generative model, i.e., generating (sampling) according to the structure of learned world density. In this paper, we propose Generative Neurosymbolic Machines, a generative model that combines the benefits of distributed and symbolic representations to support both structured representations of symbolic components and density-based generation. These two crucial properties are achieved by a two-layer latent hierarchy with the global distributed latent for flexible density modeling and the structured symbolic latent map. To increase the model flexibility in this hierarchical structure, we also propose the StructDRAW prior. In experiments, we show that the proposed model significantly outperforms the previous structured representation models as well as the state-of-the-art non-structured generative models in terms of both structure accuracy and image generation quality. Our code, datasets, and trained models are available at https://github.com/JindongJiang/GNM

Motivation & Objective

• To address the limitation of current deep learning models in simultaneously supporting structured, interpretable representations and density-based image generation.
• To reconcile symbolic and distributed representations in a unified generative framework that enables both recognition and imagination of complex scenes.
• To develop a hierarchical latent structure that supports both global density modeling and local symbolic compositionality in an unsupervised, end-to-end trainable manner.
• To improve the expressiveness of structured representations through a novel autoregressive prior, StructDRAW, that enables structured feature-drawing.
• To demonstrate that the proposed model achieves superior performance in both structure accuracy and image generation quality compared to state-of-the-art baselines.

Proposed method

• The model employs a two-layer latent hierarchy: a global distributed latent variable $\mathbf{z}^g$ for flexible density modeling and a structured symbolic latent map $\mathbf{z}^s$ for entity-based representations.
• The global latent $\mathbf{z}^g$ is used to condition the generation of the structured symbolic representation $\mathbf{z}^s$ via a conditional prior, enabling density-based sampling.
• The StructDRAW prior is introduced as an autoregressive prior over the structured latent map, enabling sequential, structured drawing of features with improved expressiveness.
• An MLP interaction layer is incorporated within StructDRAW to model dependencies between symbolic components, enhancing representational capacity.
• The model is trained end-to-end using a variational inference objective with a KL divergence term, optimizing both reconstruction and prior likelihood.
• The framework supports both recognition (inference of $\mathbf{z}^s$ from $\mathbf{x}$) and generation (sampling $\mathbf{x}$ from $\mathbf{z}^g$ and $\mathbf{z}^s$), enabling novel scene synthesis.

Experimental results

Research questions

• RQ1 Can a generative model simultaneously learn interpretable, symbolic, object-centric representations and generate images according to the underlying data density?
• RQ2 How can a hierarchical latent structure be designed to jointly support global density modeling and local symbolic compositionality?
• RQ3 What role does an autoregressive prior like StructDRAW play in improving the expressiveness and accuracy of structured latent representations?
• RQ4 To what extent does the proposed model outperform existing structured and non-structured generative models in both structure accuracy and image quality?
• RQ5 How do architectural components such as the global prior and MLP interaction layer contribute to the model's performance?

Key findings

• On the ARROW dataset, GNM-Struct achieved a structure accuracy of 0.976, significantly outperforming ConvDRAW (0.176) and GNM-Gaussian (0.784).
• On the MNIST-10 dataset, GNM-Struct achieved a structure accuracy of 0.824, substantially exceeding ConvDRAW (0.000) and GNM-Gaussian (0.096).
• GNM-Struct achieved a log-likelihood of 33,809 on ARROW and 10,450 on MNIST-10, outperforming all baselines in both metrics.
• Ablation studies showed that both the StructDRAW prior and the MLP interaction layer are critical components, as removing them degraded performance significantly.
• The model demonstrated graceful degradation of log-likelihood with increasing $\beta$ in the KL term, while maintaining high structure accuracy, unlike baselines that suffered severe performance drops.
• Visualization of latent traversals confirmed that GNM can generate coherent, structured scenes by independently controlling symbolic components like object position and direction.