American researchers present LegoGPT: an AI model that builds bookshelves and other constructions with Lego blocks based on text prompts.
With generative AI, you can create many things today. From texts to images, video or music: there’s an AI model for almost everything nowadays. Researchers from Carnegie Mellon University in Pittsburgh, USA, are making a bid for the most original model to date: LegoGPT. As the name suggests, this model specializes in playing with Lego blocks.
From Text to Lego
In a paper, the researchers demonstrate how the model works. At its core, it’s not much different from how other AI models work with text prompts. You describe what you want, and the model converts it into a digital twin of a Lego construction. If you link the model to a robotic arm, you don’t even have to place the blocks yourself. From a bookshelf to a guitar or a table: LegoGPT can handle various constructions.
LegoGPT predicts step by step which piece should be placed when and where. It constantly checks if the bricks are correctly placed, physically possible, and if the structure remains stable. If a structure proves unstable, the system automatically rolls back to the last stable design.
47,000 Lego Structures
To train the model, the researchers built a new dataset: StableText2Lego. It contains more than 47,000 Lego structures with over 28,000 unique objects, each coupled with detailed descriptions. Moreover, each design was analyzed for physical stability. These texts were generated by rendering Lego objects from 24 different angles, followed by text generation via GPT-4o. The model is a derivative of Meta’s Llama.
In addition to the basic structures, the model can also add textures and colors based on style-focused descriptions. In demonstrations, a robot also succeeded in automatically assembling some of the generated models physically. The complete dataset, model, and code have been made publicly available.