Break and Make: Interactive Structural Understanding Using LEGO Bricks

by   Aaron Walsman, et al.

Visual understanding of geometric structures with complex spatial relationships is a fundamental component of human intelligence. As children, we learn how to reason about structure not only from observation, but also by interacting with the world around us – by taking things apart and putting them back together again. The ability to reason about structure and compositionality allows us to not only build things, but also understand and reverse-engineer complex systems. In order to advance research in interactive reasoning for part-based geometric understanding, we propose a challenging new assembly problem using LEGO bricks that we call Break and Make. In this problem an agent is given a LEGO model and attempts to understand its structure by interactively inspecting and disassembling it. After this inspection period, the agent must then prove its understanding by rebuilding the model from scratch using low-level action primitives. In order to facilitate research on this problem we have built LTRON, a fully interactive 3D simulator that allows learning agents to assemble, disassemble and manipulate LEGO models. We pair this simulator with a new dataset of fan-made LEGO creations that have been uploaded to the internet in order to provide complex scenes containing over a thousand unique brick shapes. We take a first step towards solving this problem using sequence-to-sequence models that provide guidance for how to make progress on this challenging problem. Our simulator and data are available at Additional training code and PyTorch examples are available at


page 2

page 7

page 20

page 23


PackIt: A Virtual Environment for Geometric Planning

The ability to jointly understand the geometry of objects and plan actio...

Interscript: A dataset for interactive learning of scripts through error feedback

How can an end-user provide feedback if a deployed structured prediction...

Can NLP Models Correctly Reason Over Contexts that Break the Common Assumptions?

Pre-training on large corpora of text enables the language models to acq...

LAGNet: Logic-Aware Graph Network for Human Interaction Understanding

Compared with the progress made on human activity classification, much l...

InDL: A New Datasets and Benchmark for In-Diagram Logic Interpreting based on Visual Illusion

This paper introduces a novel approach to evaluating deep learning model...

Corrigibility with Utility Preservation

Corrigibility is a safety property for artificially intelligent agents. ...

Please sign up or login with your details

Forgot password? Click here to reset