Ep#113: MolmoBOT: Large-Scale Simulation Enables Zero-Shot Manipulation

Using simulation to teach robots would unblock the massive scale of robotics data that seems to be missing right now. The issue is usually sim-to-real transfer; MolmoBOT aims to solve this with sufficient diversity of simulation data, so that no real data is necessary for robot deployment. It has a few parts: MolmoBot-Engine produces diverse pick-and-place demonstrations across many different simulated environments; MolmoBot-Data is a dataset of 1.7 million expert trajectories for item manipulation and pick-and-place tasks; and they train a number of VLA policies which can achieve strong performance on real-world pick-and-place tasks with zero real robot data. Rose Hendrix, Snehal Jauhri, Abhay Deshpande, and Maya Guru join us to give us the details.

To learn more, watch RoboPapers episode 113 with Chris Paxton and Ruijie He!

Abstract

A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation.

We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.7 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot (a Molmo2-based VLM with a flow-matching action head), MolmoBot-Pi0 (replicating the π₀ architecture for controlled comparison), and MolmoBot-SPOC (a lightweight policy for edge deployment). Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments, reaching 79.2% success rate on real-world tabletop pick-and-place, outperforming π₀.₅ at 39.2%.

Learn More

Project Page: https://allenai.github.io/MolmoBot/

ArXiV PDF: https://arxiv.org/pdf/2603.16861

Transcript

Produced by ChatGPT and Riverside.

Introducing MolmoBot

Chris Paxton: Hey everybody, and welcome to another episode of RoboPapers. Today we’re excited to have the team behind the MolmoBot paper here to talk about their work. Would you like to introduce yourselves?

Rose Hendrix: I’m Rose Hendrix. Until recently, I was a research scientist at AI2, and now I’m at a startup.

Abhay Deshpande: I’m Abhay Deshpande. I conducted this work while I was a predoctoral researcher at AI2, and now I’m a PhD student at Princeton University.

Maya Guru: I’m Maya Guru. I’m currently a predoctoral researcher at AI2, and I’m looking for PhD positions for this fall.

Chris Paxton: Awesome. Would you like to get started and tell us a little about what you’ve been working on? Rose, maybe you can share your screen?

Abhay Deshpande: I’ll pull it up.

Rose Hendrix: Thanks, Abhay. On a previous episode of RoboPapers, the MolmoSpaces team came on to talk about their work, which is closely related to this. MolmoSpaces is about scaling synthetic environments and evaluation. MolmoBot is about cashing out that potential: scaling synthetic data generation, then training policies on that data to achieve zero-shot sim-to-real transfer for manipulation.

MolmoBot is a data engine and a family of policies. We show that the policies transfer in large-scale simulation evaluations and in the real world, both for tabletop pick-and-place and for mobile manipulation on the RB-Y1 platform. That includes opening doors, drawers, and cabinets, as well as object manipulation and pick-and-place.

Generating synthetic robot data at scale

Abhay Deshpande: As Rose mentioned, we have this large framework, MolmoSpaces, for synthetic data and large-scale simulation. Within that, we built MolmoBot-Engine, a data engine for generating extremely diverse, large-scale datasets.

The basic idea is to roll out heuristic demonstrators in these environments, then apply very aggressive randomization: camera extrinsics and intrinsics, textures, lighting, colors, and all sorts of other things. You keep rolling that out and scaling with compute, and you get high-throughput, large-scale data.

For a concrete example, you sample an arbitrary scene from MolmoSpaces. There are more than 100,000 scenes available. Then you sample a workspace, choose an object for a pick-and-place operation, place a receptacle for the object, and put the robot in the workspace.

From there, we randomize lighting, the robot’s initial configuration, inertial and frictional parameters, and other aspects of the scene. Then we roll out a scripted demonstrator. Since we have access to privileged information in simulation, a fairly simple heuristic policy can perform well under a wide variety of conditions.

Of course, there are so many things we could randomize: textures, lighting, robot configurations, camera poses, task instructions, templates, and much more. How do we decide which ones matter? What we found is that a good philosophy is basically: yes. If you can randomize it, you generally should. That extreme diversity goes a long way toward making policies robust in the real world.

We generate several different modalities. By default, we have two external cameras and a wrist camera. Since we’re using the DROID setup, we replicate those cameras in simulation: ZED 2 cameras for the external views and a ZED Mini for the wrist camera. We also have a randomized GoPro view.

The GoPro has a very wide field of view and significant fisheye distortion. We apply post-processing to warp the rectified image so it looks like what the real GoPro would produce.

In addition to RGB and depth, we save the camera intrinsics, since we’re randomizing them. We also save task-relevant object keypoints, which lets us compute visibility heuristics or do image-space reasoning downstream.

For actions, we save all of the action representations rather than prescribing just one. That way, people using MolmoBot-Data can support a wide variety of downstream applications. We also save proprioception, joint positions and velocities, end-effector poses and twists, and other task-relevant privileged state: the demonstrator’s current phase, contact forces and points, grasps, door states, and so on.

Finally, we label everything with rewards, success indicators, scene metadata, task prompts, and language referring expressions.

Chris Paxton: One question: all these grasps and door states you’re talking about—where do those come from? Are those heuristics?

Abhay Deshpande: Generally, they’re derived from the scripted heuristics or read directly from simulation. A door state might just be a joint angle, which we can access directly. The same goes for contact points and forces. Others, like rewards and success, are based on predicates: conditions that need to evaluate to true for us to say the task succeeded.

Rose Hendrix: This was also motivated by the fact that knowing what the demonstrator is trying to do can be a useful training signal. That’s a strength of synthetic data. If you’re collecting human teleoperation demonstrations, it isn’t necessarily possible to extract that kind of nuanced information about what the operator intended at any given moment.

RJ He: Was all the data scripted, or were there other sources?

Abhay Deshpande: It was all scripted. We found that fairly simple heuristic policies go a long way. We have around 1.7 million demonstrations, and they’re all generated with scripted policies. By randomizing enough dimensions, you get surprisingly far.

You can see in the data distribution that the scripted motions sometimes look a bit robotic—which, of course, is fitting for a robot. But we didn’t encounter major downstream problems, and the policies performed well in the real world. It might become more of an issue when co-training with datasets that have different action distributions. There is interesting ongoing work on making synthetic data match human action distributions, but that wasn’t something we focused on here.

Rose Hendrix: And just to be clear: all of our policies are trained strictly on synthetic data. There’s no human data, teleoperation data, or real-world robot data involved in their training.

Abhay Deshpande: Altogether, we have 1.7 million demonstrations across more than 94,000 environments, 11,000 object assets, and 9,000 receptacles, totaling around 5,700 hours of data. That makes it one of the largest open robotic datasets available, and the nice thing is that you can scale compute to generate it.

In terms of hours, that puts us well ahead of datasets like DROID, Open X-Embodiment, and RoboCasa365. Where we really stand out is environment diversity. By building on MolmoSpaces and its indoor environments, we get more than 94,000 unique environments—far more than most other datasets provide.

The policy architectures and evaluations

Maya Guru: We trained three different model classes. All take visual information and proprioception as inputs: our flagship MolmoBot model, MolmoBot-SPOC, and MolmoBot-Pi0. All three achieve zero-shot transfer to the real world and generalize to new environments, though at different performance levels.

For MolmoBot and MolmoBot-SPOC, the inputs include external and wrist cameras for the Franka, or head and wrist cameras for the RB-Y1, along with a language task instruction, proprioception, and optional point conditioning. For example, if we want the robot to open a specific drawer, we can point to that drawer and instruct it to open it.

Our flagship MolmoBot architecture uses the Molmo2 vision-language model with an action expert. MolmoBot-SPOC is adapted from previous work and uses a lightweight transformer. One reason to offer that architecture is that it makes reinforcement-learning fine-tuning more accessible.

We also trained MolmoBot-Pi0, using the π₀ architecture, from scratch on our synthetic data, without real-robot data. That serves as a comparison to isolate the value of our data. We know the architecture works with real-world training data, so how far can it get with only our synthetic dataset?

For real-world evaluation, we prioritized different environments and a range of objects and tasks, including pick-and-place with a Franka robot. MolmoBot outperformed the π₀ baselines, including MolmoBot-Pi0. The results are a strong signal both that our data can transfer to the real world and that the architecture of our flagship VLA matters.

We also evaluated against several baselines in simulation using the MolmoSpaces benchmarks. These tasks have been verified using scripted policies, so we know they’re feasible in simulation. MolmoBot outperformed the other VLAs there as well.

Rose Hendrix: The modularity of MolmoBot-Engine let us extend the data generation to a range of tasks and validate performance in simulation against state-of-the-art baselines. We included picking, pick-and-place, placing next to another object, and color-conditioned placement. For the color task, for example, we used two otherwise identical receptacles to test whether a policy would follow the instruction for the correct color.

Overall, MolmoBot outperformed the baselines in both simulation and the real world.

We also ran ablations of different parts of data generation and policy training. We made special pick-only datasets with a maximum budget of 50,000 trajectories and evaluated the resulting policies in simulation and in the real world.

For dataset size, the result wasn’t surprising: more data was better. The number of object classes was more interesting. Adding classes helped, but performance was still surprisingly good when we trained on just 25 object classes and evaluated on a mixture of seen and unseen classes. It wasn’t simply the case that the model performed well on seen classes and poorly on unseen ones.

The environment ablation was perhaps even more interesting. I think the lesson was that we didn’t go far enough toward removing diversity. Even with only 50 houses, you can have hundreds of visually distinct placements, and the cameras were still randomized. We designed those ablations based on our previous work with mobile policies, and I suspect the result might not hold for mobile policies. That’s something to investigate further.

My guess is that fully randomized cameras, together with a powerful visual backbone, are doing a lot of the heavy lifting in these object and environment ablations.

We also ablated the action representation. Absolute joint positions and joint deltas performed similarly in simulation, but absolute joint positions transferred better to the real world. There are other ablations in the paper, too, but one reason we wanted to build the engine was to make tightly controlled experiments like these possible. Each point on these curves represents a separately generated, purpose-built dataset.

Robustness demonstrations and limitations

Rose Hendrix: Snehal, can you talk about the qualitative results?

Snehal Jauhri: Sure. Here are a few tabletop demonstrations with the Franka DROID setup. One we’re especially happy with shows Abhay moving a standing desk up and down while the robot performs a task. The policy immediately adapts to the table height because training included so much variation in table heights.

It never saw an episode where the table height changed dynamically, but because it saw so many different static heights during training, it adapts as the height changes.

Another demonstration shows a dramatic camera-view change. Abhay picks up one of the ZED cameras and changes the angle completely, and again the policy adapts.

What’s also interesting is that there are two bowls, one brown and one black. Initially, the policy can see only one of them. It begins moving toward the black bowl with the banana because the brown bowl is out of frame. When the other bowl comes into view, it changes course and moves toward the brown bowl specified in the instruction.

We didn’t actually plan that demonstration, if I’m remembering correctly. It was something we noticed during testing.

We also have a mobile manipulation demonstration on the RB-Y1, using two wrist cameras and a head camera. It hasn’t seen this exact door in simulation, but the learned behavior transfers: it reaches for the handle and coordinates the motion of its base with the arm as it opens the door.

Rose Hendrix: There are some important limitations. First, the tasks are quasi-static and prehensile. That rules out a huge range of things we’d ultimately want robots to do. But our goal was to establish a proof of concept for synthetic data at scale, and then expand toward more complex behaviors.

Second, we only handle rigid, non-deformable objects. Plenty of real-world objects we’d like robots to manipulate are deformable.

Finally, the policies we trained use only RGB and proprioception. We observed some failure modes that seem pretty clearly attributable to the lack of depth information. We saved depth in the dataset we released; we just didn’t train on it. We expect using it could reduce some of those failures, and that’s worth exploring.

Chris Paxton: Did you use stereo RGB, or just monocular views?

Rose Hendrix: Not calibrated stereo. We did experiment with multiview policies—two external cameras and a wrist camera, for example—but not a calibrated stereo setup.

The big next step is task diversity. We’d especially like to cover non-prehensile manipulation and multifingered dexterity. There’s exciting work at AI2 and elsewhere on training state-based RL policies and then rolling them out as expert demonstrators to generate synthetic data. That’s a promising direction and something Maya is working on now.

Simulation itself is also improving. Researchers are developing more efficient ways to simulate deformable objects, for example, and every improvement to simulation can feed back into synthetic data generation.

I think force- and tactile-aware policies are particularly interesting for synthetic data. That information can potentially be much more accessible in simulation, and I think it’s clearly part of where the field is going.

We’ve open-sourced everything: the data, policies, training code, engine—the whole stack. We’re excited to see what people build on top of it. This was a very large project with a lot of collaborators, so thanks to everyone who contributed.

How should simulation and real-world data fit together?

Chris Paxton: Thank you for the presentation. I have a few follow-up questions. Let’s start at a high level. Most work in this space, especially on VLAs, is trained predominantly on real-world data. You’re one of the few teams that’s publicly all-in on sim-to-real.

How do you feel about that approach in hindsight? What role do you think real-world data and simulation should each play? It’s a very open-ended question.

Rose Hendrix: There might be a diversity of opinions on the team. I’ll let Abhay, Maya, and Snehal go first.

Abhay Deshpande: I don’t think the two approaches are mutually exclusive. They’re different tools for solving similar problems.

Simulation can give you a huge diversity of situations that would be expensive, intractable, or unsafe to collect in the real world. If you want to train on counterfactuals, high-speed interactions, or dangerous situations, you might be able to instrument those in the real world, but simulation could be safer and cheaper.

At the same time, real-world data collected on your robot, in your domain, for your task, is probably always going to be the gold standard. You could imagine combining it with synthetic data to get the last mile of generalization, or to push reliability toward the level you really need.

Chris Paxton: One follow-up on that. You mentioned dangerous situations as a reason to use simulation, but whenever I see somebody doing something crazy and dangerous with a robot—say, using π₀.₅ to operate a backhoe—it always seems to be in the real world. Why is that?

Abhay Deshpande: That’s a great point. Maybe there’s some selection bias. Right now, you might only be able to do that in the real world, because simulating a backhoe could be a huge pain in the ass.

So maybe it’s an argument that dangerous tasks should be trained in simulation, not necessarily that they currently are. That’s partly a limitation of what we can simulate. And because so much of the field is pursuing real-world data collection anyway, those demonstrations tend to happen in the real world.

Chris Paxton: That makes sense. Maya?

Maya Guru: I agree with a lot of Abhay’s points, especially that real-world data can be useful for the last mile or for the tail end of tasks that are difficult to simulate—tasks where building a simulation might be more effort than it’s worth.

But simulation gives you broad coverage across a lot of environments, and it’s easy to scale. Rose and I have also talked about how zero-shot sim-to-real transfer is a particularly interesting research problem. How can we get something trained only on simulated data to work in the real world? That’s a different challenge from training on real data in essentially the same distribution as the environment where you intend to deploy it.

Snehal Jauhri: There’s this data-pyramid argument. At the bottom of the pyramid is internet data: general knowledge of the world, how things look, and so on. At the top is a relatively small amount of real-world robot data. In the middle is simulation.

Simulation isn’t going to be internet-scale, but it can be much larger than real-world robot datasets while still providing useful, in-domain experience. It helps shape the policy toward the behaviors you ultimately want, and then you can use real-world fine-tuning.

There will be tasks where you can collect enough real data that you may not need the middle layer. And there will be tasks where direct sim-to-real transfer already works well, like quadruped locomotion, so you might not need much of the top layer. But for a lot of tasks, a mixture of the two will probably work well.

Chris Paxton: I’m curious about the middle layer of that pyramid. Which part of the robotics problem do you think is best solved in simulation?

Snehal Jauhri: I think policy shaping is a good fit. Consider trying to learn mobile manipulation from scratch entirely in the real world. It’s doable, but quite expensive. Even when collecting real data, we spend a lot of effort teaching relatively simple behaviors. Much of that could happen in simulation.

Our results suggest that the basics really should be learned in simulation, as long as you can figure out the domain randomization needed to make them transfer. That’s broadly what we’re doing in this work: pushing domain randomization far enough to get direct real-world transfer.

Rose Hendrix: I’ve thought a lot about this, and I have a metaphor. Bear with me for a moment.

Think of robot data collection as a food system. You have teleoperation and simulation. Teleoperation is like foraging, and simulation and synthetic data are like agriculture.

When you’re foraging, you can find extremely high-quality things—fancy mushrooms and all sorts of things that we don’t know how to grow at scale. They’re highly valued, and it’s well worth your time to collect them. Foraging isn’t going away.

But agriculture is the technology that helped unlock human civilization. We learned irrigation and crop rotation, and we kept getting better at it. Agriculture has a technological learning curve. Foraging doesn’t have that same kind of learning curve.

I think we’re still very early in synthetic data, but what’s compelling is that it’s something we can improve year after year. Teleoperation, in comparison, can look a little like a vending machine: you put money in, and data comes out.

Obviously, people are working very hard on improving teleoperation, and I don’t mean to denigrate that work. But synthetic data is a technology that can compound as we improve the process of producing it. That’s why I think it’s so worth working on.

RJ He: Building on that analogy, which dimensions of data still belong in the foraging category? What isn’t easy for simulation to recreate, where you still need teleoperation? And which dimensions have more or less been cracked, so you can exploit the agricultural advantages of simulation at scale?

Rose Hendrix: There’s still a lot that simulation isn’t good at. Deformable objects are probably the biggest class of things where the technology isn’t there yet.

Also, if you want extremely high reliability—five nines, say—there’s still a role for on-hardware teleoperation and final fine-tuning data. That’s the garnish. But I think most of the food pyramid could potentially be generated rather than collected.

Abhay Deshpande: There are so many different axes of diversity. The obvious one is scale: how many episodes or frames you have. But then there’s the diversity of environments, camera positions, camera poses, and so on.

I think simulation is particularly good at camera diversity, as we show here. MolmoBot has very strong camera generalization. When you post-train policies on DROID, they tend to fit very closely to those left- and right-shoulder camera views, and performance drops as you move to different camera positions. You don’t see that as much with MolmoBot, because camera randomization is such an easy knob to turn in simulation.

A weaker axis is task diversity. Every new task requires implementing a new demonstrator, testing it, tweaking it, and making sure the throughput is good. That demands much more human effort than randomizing camera poses or textures.

That’s a real limitation of MolmoBot as released, and it’s something people are working on. In the real world, a teleoperator can often change tasks just by deciding to do something different, on the same robot with the same cameras and environment. That flexibility is a valuable advantage of teleoperation, and it’s something synthetic data needs to catch up on.

Chris Paxton: That makes sense. One last question along these lines. Your group previously worked on MolmoAct and MolmoAct 2, which involved a lot of real-world data. Have you thought about combining that work with MolmoBot? Do you think the real-world data or those ideas would add something here?

Rose Hendrix: Definitely. One of the main things MolmoAct and MolmoAct 2 explored was reasoning, and almost a kind of test-time scaling.

One of the great things about synthetic data is how much additional training signal it can provide for that kind of reasoning. In MolmoAct, there was an entire process for tracking points on objects over time. In simulation, you can get that basically for free: render the segmentation mask and save it.

You could also expose the demonstrator phase or the current end-effector goal—what the robot is trying to reach at that moment. There are all sorts of additional signals that could help. I think combining those ideas with synthetic data is a very promising direction.

Why does the Molmo2 backbone matter?

RJ He: Let’s talk a little more about the architecture. Comparing MolmoBot with your implementation of π₀ on the same data, what do you think made MolmoBot perform so much better?

Maya Guru: I think one factor is Molmo2’s ability to reason about space. It’s a strong backbone for the kinds of generalization we’re looking for, such as localizing objects.

The MolmoBot-Pi0 comparison also shows the value of our data. We’re taking an architecture that we already know can perform well in the real world, training it with only synthetic data, and showing that it can still transfer. That’s a compelling result about the data itself.

But I do think the strength of Molmo2 contributes to the difference in performance between the architectures.

Rose Hendrix: MolmoAct 2 and MolmoBot use very similar base architectures, with some minor modifications. And from a close look at MolmoBot-Pi0 versus MolmoBot with the Molmo2 backbone, I think the backbone is probably a significant part of the difference. It’s roughly another billion parameters, and we’re testing generalization, which I suspect is coming in large part from the backbone.

Abhay Deshpande: I’d love to understand how much of that generalization comes from the backbone. We’ve seen impressive demonstrations on objects the policy hadn’t seen before. The semantic understanding and referring expressions presumably draw on the VLM’s pretraining.

It would be interesting to tease apart how we can best leverage that pretraining. For example, there are training techniques used in models like π₀.₅ where auxiliary objectives help preserve VLM capabilities, and flow-matching losses aren’t backpropagated through the VLM weights in the same way.

We didn’t do those things. If you tried to use our fine-tuned backbone for autoregressive text decoding, I’m not sure you’d get anything useful. Yet we still preserve some of that VLM understanding: the robot recognizes objects and referring expressions it never encountered in the synthetic robot data.

That might explain a lot of the gap between MolmoBot-Pi0 and MolmoBot. Molmo2 is a newer, larger VLM with stronger representations.

Rose Hendrix: To be clear, we don’t have an ablation that conclusively shows the backbone is responsible. That’s our interpretation, not something we’ve isolated experimentally. Future work!

Chris Paxton: Could you add enough objects to the synthetic dataset that you no longer needed a pretrained VLM backbone for generalization? You’d think one big advantage of simulation is that you can, at least in principle, throw an enormous number of objects into the training data.

Maya Guru: The SPOC results speak to that a little. MolmoBot-SPOC is a lightweight transformer with a text encoder and a vision encoder, but not a VLM backbone. We don’t see the same level of generalization, in either simulation or the real world.

It still does reasonable things. On the pick-and-place benchmark, I think its success rate was somewhere in the 50-percent range. So I wouldn’t say these capabilities depend entirely on the VLM backbone. But some of the generalization appears to be closely tied to it.

Chris Paxton: But that’s also a smaller architecture, presumably trained with fewer FLOPs. It could be that enough data or diversity in simulation would solve the problem. Do you think there’s a fundamental limitation of the data, or could you train your way out of it?

Abhay Deshpande: The controlled experiment would be to randomly initialize the same VLM architecture and then train it on a huge amount of synthetic robot data. That’s probably the only way to settle it. We can’t say conclusively from these comparisons.

In principle, if the evaluation distribution is sufficiently similar to the training distribution, you might expect enough robot data to be sufficient. We see something like that with large behavior models that don’t necessarily use a pretrained VLM backbone but still show impressive performance and generalization.

Maybe VLMs are a way to bootstrap from the robot data you have, rather than a fundamentally necessary ingredient.

Rose Hendrix: I have a caveat. I think we rely on the backbone’s visual understanding to bridge the sim-to-real gap. I’m not sure our transfer would be as good without a powerful visual backbone.

In previous projects, we’ve been able to get nice sim-to-real transfer without a language model, but not necessarily without a strong vision model such as SigLIP. Without one, policies can be much more sensitive to rendering differences.

So that’s the nuance: enough synthetic diversity may keep a pretrained backbone from collapsing onto a narrow set of objects, while the backbone itself helps with sim-to-real transfer. I suspect we need that visual representation. Again, that’s an opinion; we don’t have the ablation to prove it.

Abhay Deshpande: One important training detail is that we froze our visual backbones. As Rose said, pretrained vision representations are really good now.

We’ve also seen in work that evaluates real-world-trained policies in simulation that they can behave reasonably well despite the differences in rendering. Those pretrained visual representations are probably doing a lot of the heavy lifting in both directions.

Even if you show the model a strangely rendered potato, it still knows it’s a potato. That means we may not need to invest so much effort in perfect photorealism.

Scripted demonstrations and action diversity

RJ He: It would be interesting to understand exactly which ingredients help bridge the sim-to-real gap, and why these approaches work as well as they do now.

On the data side, you focused almost entirely on scripted demonstrations. Do you think you lose anything by relying on scripted data instead of collecting, say, DAgger data or other forms of expert correction?

Abhay Deshpande: One limitation of scripted data is that you’re prescribing a particular way to accomplish a task. You have a state machine, and you roll out that state machine. You can randomize things inside it—add action noise, perturb waypoints, randomly choose grasps—but you’re still following basically the same high-level strategy.

A human, especially across multiple operators, might complete the same task in completely different ways: different strategies, speeds, or approaches.

I don’t think that’s an unsolvable problem. There’s a lot we can do to increase the diversity of our demonstrators, and there’s interesting work on making synthetic demonstrations look more like human demonstrations. Improving the action distributions of scripted demonstrators is a promising research direction.

Research directions and closing thoughts

Chris Paxton: Any last questions, RJ? Otherwise we can wrap up with our usual final question: is there any recent work you’d like to give a shout-out to? A paper or demo that you thought was particularly cool?

Abhay Deshpande: One thing that’s been on everyone’s mind is the impressive ability of frontier AI models to do robotics-related tasks. What does that mean for the field, and how can we use those models in our research? In particular, how can we use them for simulation? That’s something I’ve been thinking about a lot lately.

Maya Guru: Similarly, I think we’re all still processing how impressive GPT-6 is. One thing I’m especially interested in is agentic robotics. How can we use these models to help robots acquire new capabilities? How can robots deploy those capabilities in the real world, almost like calling tools? I’m very interested in how that relates to task diversity: learning new tasks, and then being able to invoke them.

Snehal Jauhri: I’m interested in scaling this approach further. We used a large number of household environments, but they’re still one particular flavor of household environment.

There’s been good work on generating indoor 3D environments using agents and agentic pipelines. Work like SceneSmith, for example, has shown promising results in taking procedural generation much further.

What could we do with ten or a hundred times more environments, and more diversity? One of the nice things about simulation is that we can scale the data and investigate scaling laws. I’m very interested in indoor scene generation and where that goes next.

Rose Hendrix: I’ll cheat and pick two things.

First, in synthetic data and simulation, I’m excited by work coming out of Jeannette Bohg’s lab, including SimToolReal and Play2Perfect. I think dexterous manipulation is where we ultimately need to go, and it’s extraordinarily difficult to teleoperate dexterous hands.

You can map motions from humans, but that isn’t exactly the same thing, particularly when you have different kinds of dexterous manipulators: three-fingered hands, tools such as chopsticks, and so on. Teleoperation becomes very challenging.

State-based RL gives us a way to explore not just how to imitate human behavior, but how to get superhuman performance from expert demonstrators. If the demonstrators are trained with RL, there’s no reason they couldn’t become better than humans at certain tasks. I think that’s a very exciting direction.

Second, I’m interested in what Snehal mentioned: generating scenes much more freely. A major limitation of our current data is how hard it is to generate environments for long-horizon, compositional tasks.

Imagine training a robot to respond to, “I just got home; put all my groceries away.” You need a lot of specific assets, arranged in appropriate ways: a refrigerator with multiple compartments, different groceries, storage locations, and so on. It’s difficult to procedurally generate a huge range of coherent scenes that support tasks like that.

Agentic scene generation might unlock this. We could build much more varied environments specifically suited for complex, long-horizon tasks. That’s another line of work I’m really excited about.

Chris Paxton: We actually already had the SimToolReal team on the podcast, for what it’s worth. But yes, I hope to see a lot more interesting work in agentic robotics.

RJ He: One more question. We’ve talked a lot today about deformable objects as a challenge simulation hasn’t cracked. Do you think agentic methods could help solve that, and if so, how?

Rose Hendrix: The thing about deformables is that we can simulate them. We’ve all seen animated movies. You can simulate almost anything—it just takes a long time.

The problem is efficiency. The simulation can be too slow to use for large-scale data generation, and certainly too slow to learn from efficiently with RL.

I think agentic approaches will help in the way they’re helping many other areas: making the development process faster and better. If people focus on this problem and use modern coding tools to improve the simulators, the technology will get better. It’s a rising-tide-lifts-all-boats effect. The problem isn’t that deformable simulation is impossible. It’s that it’s slow, and slow is something we can work on.

RJ He: That makes sense. And it fits the larger theme of the discussion: betting on simulation means you can keep reinvesting in better tools, improving the technology and everything you can build on top of it. It’s exciting to see that approach paying off.

Chris Paxton: Absolutely. There’s a ton of potential in simulation, and I’m excited to see where this goes. Thank you all for coming on.

Rose Hendrix: Thank you so much for having us.

Abhay Deshpande: Thanks for having us.

Snehal Jauhri: Thank you.

Maya Guru: Thanks. Bye!