[Paper Review] Blocks Assemble! Learning to Assemble with Large-Scale Structured Reinforcement Learning

Seyed Kamyar Seyed Ghasemipour, Daniel Freeman+4 more|arXiv (Cornell University)|Mar 15, 2022

Modular Robots and Swarm Intelligence4 citations

TL;DR

This paper introduces a physics-based 3D magnetic block assembly environment for training agents to assemble diverse blueprints using large-scale structured reinforcement learning. A single graph-based policy agent achieves zero-shot generalization to unseen blueprints and operates effectively in reset-free settings, demonstrating that multi-task training with structured representations enables robust, generalizable assembly without explicit planning or hierarchical structures.

ABSTRACT

Assembly of multi-part physical structures is both a valuable end product for autonomous robotics, as well as a valuable diagnostic task for open-ended training of embodied intelligent agents. We introduce a naturalistic physics-based environment with a set of connectable magnet blocks inspired by children's toy kits. The objective is to assemble blocks into a succession of target blueprints. Despite the simplicity of this objective, the compositional nature of building diverse blueprints from a set of blocks leads to an explosion of complexity in structures that agents encounter. Furthermore, assembly stresses agents' multi-step planning, physical reasoning, and bimanual coordination. We find that the combination of large-scale reinforcement learning and graph-based policies -- surprisingly without any additional complexity -- is an effective recipe for training agents that not only generalize to complex unseen blueprints in a zero-shot manner, but even operate in a reset-free setting without being trained to do so. Through extensive experiments, we highlight the importance of large-scale training, structured representations, contributions of multi-task vs. single-task learning, as well as the effects of curriculums, and discuss qualitative behaviors of trained agents.

Motivation & Objective

• To develop a naturalistic, physics-based environment for studying generalization in robotic assembly tasks.
• To investigate whether a single agent can learn to assemble diverse, complex blueprints without task-specific fine-tuning.
• To evaluate the effectiveness of structured representations and multi-task training in enabling zero-shot generalization and reset-free operation.
• To understand the relative contributions of large-scale training, curriculum learning, and policy architecture in solving compositional, multi-step assembly tasks.

Proposed method

• Design a 3D simulation environment with 16 magnetically connectable blocks of 6 distinct shapes, enabling dynamic, modular structure formation.
• Implement a graph-based policy network that represents blocks as nodes and connections as edges, enabling structured, permutation-invariant reasoning over object configurations.
• Train a single agent using large-scale, multi-task reinforcement learning across nearly 200 diverse blueprints of varying complexity.
• Use a curriculum strategy that prioritizes harder blueprints during training to improve sample efficiency and generalization.
• Enable direct block manipulation via virtual grippers, abstracting away complex grasping while preserving bimanual coordination and physical reasoning challenges.
• Define sparse, dense, and curriculum-based reward shaping to guide learning toward successful assembly and structural correctness.

Experimental results

Research questions

• RQ1 Can a single agent trained via large-scale multi-task RL generalize zero-shot to unseen, complex blueprints in a 3D block assembly environment?
• RQ2 How effective is a graph-based policy architecture in enabling generalization across diverse, compositional structures compared to standard feedforward policies?
• RQ3 What is the contribution of multi-task training versus single-task fine-tuning in achieving robust, generalizable assembly behavior?
• RQ4 Does curriculum learning that emphasizes harder blueprints improve sample efficiency and generalization performance?
• RQ5 Can agents trained in episodic, reset-based environments operate effectively in a continuous, reset-free setting without explicit training for it?

Key findings

• The graph-based policy agent achieved zero-shot generalization to unseen blueprints across all 199 target structures, including those with up to 16 blocks.
• Multi-task training across all blueprints enabled the agent to learn increasingly complex structures, with performance on larger blueprints improving significantly compared to single-task baselines.
• The agent successfully operated in a reset-free setting despite being trained only in episodic episodes, indicating emergent robustness and long-horizon planning capabilities.
• Large-scale training was essential: agents trained on fewer blueprints failed to generalize to complex or unseen structures.
• The combination of structured representations and multi-task learning outperformed alternatives, with no need for hierarchical planning or curriculum in the policy.
• Qualitative analysis revealed that agents discovered complex, coordinated maneuvers such as sequential attachment and stable intermediate configurations, indicating physical reasoning and bimanual coordination.