[Paper Review] Image2Lego: Customized LEGO Set Generation from Images

Kyle R. Lennon, Katharina Fransen+6 more|arXiv (Cornell University)|Aug 19, 2021

Advanced Vision and Imaging24 references4 citations

TL;DR

This paper presents Image2Lego, an end-to-end pipeline that generates customizable 3D LEGO® brick models from 2D images using a novel octree-structured autoencoder for latent representation and a dedicated network to predict this latent space from images. The system enables the first direct conversion of photographs into physically buildable LEGO® sets with step-by-step instructions and animations, supporting flexible resolution and diverse applications including face modeling and text-to-LEGO® generation.

ABSTRACT

Although LEGO sets have entertained generations of children and adults, the challenge of designing customized builds matching the complexity of real-world or imagined scenes remains too great for the average enthusiast. In order to make this feat possible, we implement a system that generates a LEGO brick model from 2D images. We design a novel solution to this problem that uses an octree-structured autoencoder trained on 3D voxelized models to obtain a feasible latent representation for model reconstruction, and a separate network trained to predict this latent representation from 2D images. LEGO models are obtained by algorithmic conversion of the 3D voxelized model to bricks. We demonstrate first-of-its-kind conversion of photographs to 3D LEGO models. An octree architecture enables the flexibility to produce multiple resolutions to best fit a user's creative vision or design needs. In order to demonstrate the broad applicability of our system, we generate step-by-step building instructions and animations for LEGO models of objects and human faces. Finally, we test these automatically generated LEGO sets by constructing physical builds using real LEGO bricks.

Motivation & Objective

• To enable amateur builders to create complex, personalized LEGO® sets from 2D images without requiring advanced 3D modeling skills.
• To bridge the gap between 2D image input and physically realizable 3D LEGO® brick models through a complete, end-to-end pipeline.
• To support flexible resolution outputs to suit varying user needs, from small decorative models to detailed standalone sets.
• To extend the pipeline to support novel applications such as 3D face reconstruction and text-to-LEGO® generation via integration with DALL-E.
• To validate the system through real-world physical builds and automated instruction generation using LeoCAD.

Proposed method

• A novel octree-structured autoencoder is trained on 3D voxelized LEGO® models to learn a compact, hierarchical latent representation suitable for reconstruction.
• A separate deep learning network is trained to predict the latent code from a single 2D image, enabling image-to-3D model translation.
• The predicted latent code is decoded into a 3D voxel grid, which is then algorithmically converted into a set of compatible LEGO® bricks using a rule-based mapping strategy.
• The system supports multiple output resolutions (e.g., 8³, 16³, 32³) to balance detail and build feasibility, with higher resolutions preserving structural fidelity.
• Background matting is applied as a preprocessing step to isolate the subject from the image before input to the model, improving reconstruction quality.
• The pipeline is extended by integrating pre-trained models such as the Volumetric Regression Network (VRN) for 3D face reconstruction and DALL-E for text-to-image generation, enabling caption-driven LEGO® model creation.

Experimental results

Research questions

• RQ1 Can a deep learning pipeline accurately reconstruct 3D voxelized models from single 2D images with sufficient geometric fidelity for LEGO® conversion?
• RQ2 How can a latent space representation be designed to support flexible-resolution 3D model generation suitable for LEGO® brick construction?
• RQ3 To what extent can the system generate physically stable and buildable LEGO® models from diverse inputs, including photographs and text-generated images?
• RQ4 Can the pipeline be extended to support novel use cases such as 3D face modeling and text-to-LEGO® generation without compromising reconstruction quality?
• RQ5 How do different resolution levels affect the visual and structural fidelity of the resulting LEGO® models?

Key findings

• The system successfully converts real-world photographs into 3D LEGO®-compatible models with physically realizable structures, validated by physical construction using actual LEGO® bricks.
• At 16³ and 32³ resolutions, the generated models clearly preserve the defining features of the input objects, such as the shape of an airplane or a human face.
• The pipeline generates a complete set of building instructions and a parts list for the 16³ resolution airplane model, consisting of 40 bricks, using LeoCAD software.
• The system demonstrates multi-scale capability, with models at 8³, 16³, and 32³ resolutions all showing recognizable object structures, while 4³ resolution is too low for identification.
• Integration with DALL-E enables the generation of LEGO® models from textual descriptions, such as 'an armchair in the style of a gourd', demonstrating the system’s extensibility beyond photographic inputs.
• The method achieves the first known direct conversion of a photograph into a 3D LEGO® model with full build instructions and physical realizability, marking a significant advancement in creative 3D modeling accessibility.