[Paper Review] ShapeStacks: Learning Vision-Based Physical Intuition for Generalised Object Stacking

Oliver Groth, Fabian B. Fuchs+2 more|arXiv (Cornell University)|Apr 21, 2018

Robot Manipulation and Learning22 references4 citations

TL;DR

This paper introduces ShapeStacks, a simulation-based dataset of 20,000 diverse geometric stack configurations with detailed physical stability annotations. It proposes a vision-based deep learning model trained on this data to predict structural stability from visual inputs, which generalizes to real-world block towers and enables active stacking by learning intuitive physical intuition and object stackability, even stabilizing previously unstable structures via counterbalancing.

ABSTRACT

Physical intuition is pivotal for intelligent agents to perform complex tasks. In this paper we investigate the passive acquisition of an intuitive understanding of physical principles as well as the active utilisation of this intuition in the context of generalised object stacking. To this end, we provide: a simulation-based dataset featuring 20,000 stack configurations composed of a variety of elementary geometric primitives richly annotated regarding semantics and structural stability. We train visual classifiers for binary stability prediction on the ShapeStacks data and scrutinise their learned physical intuition. Due to the richness of the training data our approach also generalises favourably to real-world scenarios achieving state-of-the-art stability prediction on a publicly available benchmark of block towers. We then leverage the physical intuition learned by our model to actively construct stable stacks and observe the emergence of an intuitive notion of stackability - an inherent object affordance - induced by the active stacking task. Our approach performs well even in challenging conditions where it considerably exceeds the stack height observed during training or in cases where initially unstable structures must be stabilised via counterbalancing.

Motivation & Objective

• To investigate how agents can acquire physical intuition passively from visual observations and apply it actively in manipulation tasks.
• To develop a generalizable stability prediction model that works across diverse object geometries, not just specific shapes.
• To demonstrate that object affordances like stackability can emerge organically from task-driven learning rather than being pre-defined.
• To evaluate whether visual-based physical intuition enables robust, stable stacking in simulation, including handling unstable or complex configurations.
• To assess the model’s ability to stabilize inherently unstable structures using counterbalancing strategies.

Proposed method

• The authors create ShapeStacks, a large-scale simulation-based dataset of 20,000 stack configurations using elementary geometric primitives (cubes, cuboids, cylinders, spheres), with rich annotations of mechanical failure points derived from physics simulations.
• A deep convolutional neural network is trained to classify stack stability using only RGB images, with supervision from physics-engine-derived labels of structural failure.
• The model’s internal representations are probed to assess whether they learn to localize instability causes, such as overhangs or center-of-mass misalignment.
• Stackability scores are extracted from the model’s feature activations to rank object suitability for stacking, guiding active tower construction in simulation.
• The stacking policy uses simulated annealing to explore stable placements, with object selection prioritized by stackability score.
• Counterbalancing is tested by freezing unstable stacks and asking the model to place a counterweight to prevent collapse, evaluating success rates across object types.

Experimental results

Research questions

• RQ1 Can a vision-based model trained on diverse stack configurations learn to predict structural stability with high accuracy, generalizing beyond its training distribution?
• RQ2 Does the model’s internal representation capture meaningful physical intuitions, such as localization of instability causes or center-of-mass violations?
• RQ3 Can the model’s learned physical intuition be leveraged to actively construct stable stacks without explicit physics engines during inference?
• RQ4 Does the model implicitly learn object-specific stackability as an emergent affordance from the stability prediction task?
• RQ5 Can the model stabilize inherently unstable structures by intelligently placing counterweights, indicating an intuitive grasp of balance?

Key findings

• The model achieves state-of-the-art performance on real-world block tower stability prediction, matching or exceeding prior SOTA without requiring a physics engine at test time.
• The model correctly localizes the root causes of instability in stacks, such as overhangs or center-of-mass misalignment, demonstrating internal physical reasoning.
• The stackability score derived from the model’s features successfully ranks object geometries by their suitability for stacking, enabling effective prioritization in active stacking.
• The model constructs stable towers up to 12 blocks high in diverse scenarios, with the model trained on mixed shapes (including spheres) outperforming one trained only on cubes.
• In counterbalancing tasks, the model achieves a 98% success rate with spherical counterweights and 94% with cuboids, indicating strong intuitive grasp of balance and stability.
• The model generalizes well to configurations unseen during training, including stacking heights beyond the maximum seen in the training data.