[Paper Review] Photo-Realistic Blocksworld Dataset

Masataro Asai|arXiv (Cornell University)|Dec 5, 2018

Multimodal Machine Learning Applications19 references4 citations

TL;DR

This paper introduces a photo-realistic Blocksworld dataset generator that produces visually complex, realistic images of block-stacking environments to serve as a benchmark for neural-symbolic AI systems. The system uses a variational autoencoder (VAE) to learn symbolic state representations from raw images, enabling a classical planner to solve tasks via complete, systematic search; Latplan successfully solved 14 out of 30 test instances, demonstrating feasibility despite reconstruction errors in complex visual conditions.

ABSTRACT

In this report, we introduce an artificial dataset generator for Photo-realistic Blocksworld domain. Blocksworld is one of the oldest high-level task planning domain that is well defined but contains sufficient complexity, e.g., the conflicting subgoals and the decomposability into subproblems. We aim to make this dataset a benchmark for Neural-Symbolic integrated systems and accelerate the research in this area. The key advantage of such systems is the ability to obtain a symbolic model from the real-world input and perform a fast, systematic, complete algorithm for symbolic reasoning, without any supervision and the reward signal from the environment.

Motivation & Objective

• To create a visually realistic, photo-realistic dataset of the Blocksworld planning domain to accelerate research in neural-symbolic integration.
• To enable end-to-end learning of symbolic state representations from raw visual inputs without supervision or reward signals.
• To provide a benchmark for evaluating how well neural networks can reconstruct symbolic planning states from complex, realistic images.
• To test the feasibility of combining deep learning with classical symbolic planning in a visually grounded, high-level task planning setting.
• To extend the Atari Learning Environment paradigm to classical planning domains with visual realism and combinatorial subgoal interactions.

Proposed method

• The dataset generator renders realistic 3D scenes of block configurations using a modified Blocksworld environment with photo-realistic textures and lighting.
• A variational autoencoder (VAE) is trained on 2,500 randomly sampled states to learn a compact, disentangled latent representation of the visual input.
• The learned VAE embeddings are used as input to a classical planner (Fast Downward) with blind heuristics to evaluate planning feasibility.
• The system uses the AMA 1 PDDL generator to verify that the VAE-learned state representation matches ground-truth state nodes in the search space.
• Planning instances are generated via 3-, 7-, or 14-step random walks from random initial states to create diverse goal configurations.
• A neural-symbolic planner (Latplan) is applied to the reconstructed representations to test solution quality and completeness.

Experimental results

Research questions

• RQ1 Can a deep autoencoder learn a symbolic state representation from photo-realistic images of Blocksworld environments with sufficient accuracy for classical planning?
• RQ2 To what extent does visual complexity and realism hinder the ability of a VAE to reconstruct states that are semantically consistent with the underlying PDDL logic?
• RQ3 Can a classical planner, using only the VAE-learned representation as input, find correct and complete solutions without reward signals or action labels?
• RQ4 How does the performance of the neural-symbolic system degrade with increasing visual noise and environmental variation?
• RQ5 Does the presence of subgoal conflicts (e.g., Sussman’s anomaly) in the visual domain affect the reliability of representation learning and planning?

Key findings

• The VAE-based representation learning method enabled the Latplan system to generate correct plans for 14 out of 30 test instances, demonstrating partial success in end-to-end visual planning.
• The system failed on 16 instances primarily due to errors in the VAE’s reconstruction of the state, indicating limitations in representation accuracy under visual complexity.
• The search space of the 4-block, 3-stack environment contains 5,760 states and 34,560 transitions, providing a non-trivial benchmark for representation learning.
• Using blind heuristics in Fast Downward avoided scalability issues from full heuristic computation, allowing focus on representation fidelity rather than planning efficiency.
• The results suggest that current autoencoders struggle with the combinatorial and spatial reasoning demands of visual Blocksworld, especially when subgoal interactions are present.
• The dataset generator is publicly available at https://github.com/ibm/photorealistic-blocksworld, enabling reproducibility and future benchmarking.