Researchers at MIT and Toyota Research Institute have developed SceneSmith, a system that uses collaborative AI agents and vision-language models to generate detailed 3D environments for robotics simulation and training
Robots capable of performing household or industrial tasks require extensive training data to operate safely and effectively in real-world environments. However, collecting this data through physical trials is slow, costly, and often impractical at scale. To address this, researchers at MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) and Toyota Research Institute have introduced SceneSmith, a system that uses multiple AI agents to generate realistic 3D indoor environments for robot simulation and skill development.
SceneSmith's agentic design process
SceneSmith employs three specialized AI agents, each powered by a state-of-the-art vision-language model (VLM), specifically GPT-5.2. The system's workflow is structured: a "designer" agent proposes the initial scene layout, a "critic" agent evaluates the realism and practicality of the design, and an "orchestrator" agent manages the iterative process, determining when the scene meets quality standards. This agentic collaboration enables the creation of virtual spaces-such as kitchens, bedrooms, and offices-that are more detailed and visually coherent than those produced by previous automated methods.
The VLMs draw on large-scale internet data, allowing the agents to generate scenes that reflect common spatial arrangements and object placements found in real environments. SceneSmith's approach supports the inclusion of articulated objects, such as cabinets that can be opened and closed, which are essential for training robots in manipulation tasks. The system's output is directly compatible with physics simulation software, enabling robots to practice complex actions before being deployed in the physical world.
Evaluation and measurable outcomes
In a recent study, the research team generated over 1,300 unique 3D scenes using SceneSmith, with each environment containing up to six times more objects than those created by prior scene-generation systems. The researchers tested robot action plans-known as policies-across 100 distinct virtual spaces, using a VLM agent to assess the success or failure of each simulated task. The model's evaluations aligned with human judgments in more than 99% of cases, suggesting that the system can reliably identify flawed robot behaviors before physical deployment.
To further assess realism, the team introduced pretrained robot policies-controllers trained primarily on real-world data-into SceneSmith-generated environments. In one example, a robot successfully completed the instruction "take the apple from the bowl and place it onto the cutting board," indicating that the virtual scenes were sufficiently similar to real settings for learned behaviors to transfer. Additional teleoperation experiments, in which humans remotely controlled robots to interact with the environment, demonstrated that the scenes supported sustained physical interaction, not just visual plausibility.
Limitations and practical considerations
While SceneSmith advances the realism and diversity of simulated environments, the process remains computationally intensive. Generating a single detailed scene can require several hours, as each agent scrutinizes and refines object placement and physical properties. The system's efficiency could improve with greater computing resources, but current performance may limit its use in time-sensitive applications. The researchers also note that expanding to deformable objects, such as sponges or textiles, will depend on the availability of suitable 3D model libraries.
Compared to established baselines like HSM and Holodeck, SceneSmith produces environments with higher object density and greater adherence to user prompts. In a user study involving over 200 participants, more than 90% rated SceneSmith's visuals as more realistic than those of competing systems. However, the realism of simulated environments does not guarantee that robots trained in these worlds will perform flawlessly in reality. The sim-to-real gap-differences between virtual and physical environments-remains a significant challenge for robotics deployment.
Research context and future directions
The SceneSmith project was presented as a spotlight at the International Conference on Machine Learning and is documented in a preprint paper by Nicholas Pfaff, Russ Tedrake, Thomas Cohn, Sergey Zakharov, and Rick Cory. The work received support from Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation. While the system represents a notable advance in simulation-ready environment generation, its broader impact will depend on further validation, efficiency improvements, and integration with real-world robot learning pipelines.
SceneSmith's agentic framework demonstrates that collaborative AI agents, guided by large vision-language models, can automate the creation of complex, physically plausible training environments for robots. However, the transition from simulation to reliable real-world performance will require continued attention to the limitations of virtual data, the need for human oversight, and the risks of overreliance on synthetic training scenarios.
Sim-to-real transfer is a central challenge in robotics and AI. While simulation allows for rapid, low-risk experimentation, virtual environments often fail to capture the full complexity, unpredictability, and noise of the physical world. Differences in lighting, friction, sensor noise, and object variability can cause robots trained in simulation to fail when deployed outside the lab. Bridging this gap requires not only more realistic simulations but also robust evaluation, domain adaptation techniques, and careful human supervision during deployment.
