Ep#110: ENPIRE: Agentic Robot PolicySelf-Improvement in the Real World

In many fields, automated research has become increasingly plausible with the improving capabilities of frontier language models, and the field robotics may be no exception. In ENPIRE, Wenli Xiao, Jia Xie, and Tonghe Zhang build a harness framework that captures physical feedback and allows for automated robotics research, minimizing human effort while training policies that achieve a 99% success rate on challenging, dexterous tasks like organizing a pin box or fastening a zip tie.

Learn more in Episode 110 of RoboPapers, with Michael Cho, Chris Paxton, and Ruijie He!

Abstract

Achieving dexterous robotic manipulation in the real world heavily relies on human supervision and algorithm engineering, which becomes a central bottleneck in the pursuit of general physical intelligence. Although emerging coding agents can generate code to automate algorithm search, their successes remain largely confined in digital environments. We conjecture that the missing abstraction to automate robotics research is a repeatable feedback loop for real-world policy improvement: reset the scene, execute a policy, verify the outcome, and refine the next iteration. To bridge this gap, we introduce ENPIRE, a harness framework for coding agents that instantiates this physical feedback routine with four core modules: an Environment module (EN) for automatic reset and verification, a Policy Improvement module (PI) that launches policy refinement, a Rollout module (R) to evaluate policies with one or multiple physical robots operating in parallel, and an Evolution module (E) in which coding agents analyze logs, consult literature, improve training infrastructure and algorithm code to address failure modes. This closed-loop system transforms real-world manipulation learning into a controllable optimization procedure, minimizing human effort while allowing fair ablations across training recipe and agent variants. Powered by ENPIRE, frontier coding agents can autonomously train a policy to achieve a 99% success rate on challenging, dexterous manipulation tasks, such as organizing a pin box, fastening a zip tie, and tool use, a process that further accelerates when we dispatch an agent team on a robot fleet. Our results suggest a practical and scalable path toward deploying coding agents to autonomously advancing robotics in the physical world.

Learn More

Project Page:

ArXiV: https://arxiv.org/abs/2606.19980

Transcript

Michael Cho (00:06.088)Hello everyone, welcome to another episode of RoboPapers. Today we’re super happy to have Wenli, Jia Xie and Tonghe. They are three of the co-authors of this paper called ENPIRE: Agentic Robot Policy Self-Improvement in the Real World. Thank you so much for coming to the pod. Maybe as a start three of you can share a little bit about yourself first and then you guys can go through the paper.

Wenli Xiao (00:30.821)Okay, I’ll get started. Sorry. So my name is Wenli. I’m a final-year PhD at CMU, advised by Professor Guanya Shi. Previously I did two-year internship at NVIDIA GEAR Lab with Jim Fan and Yuke Zhu. And later I joined Physical Intelligence as intern.

Yeah, I did a bunch of VLA training, post-training, real-world RL and agentic stuff and now I’m mainly doing pretraining, studying the science of training and data. Yeah.

Michael Cho (01:14.488)Yeah, I think just yeah.

Jia Xie (01:16.463)Okay, sure. Yeah. Hi everyone. My name is Jia and I’m a master’s student at CMU, advised by Guanya. Of course we’re labmates. And I’ve interned at xDof as a research intern over the summer. And I’ve been working on some agent stuff and the extent stuff lately. So yeah, thanks for having us.

Tonghe Zhang (01:43.739)Okay. Hi, my name is Tonghe. I’m also a master’s student at Professor Guanya Shi’s lab. I was doing internship at Amazon this summer and previously I’ve been working on reinforcement learning. And thank you guys for inviting us.

Michael Cho (02:01.15)Alright, perfect. Great to have you guys. Yeah, Wenli, I know you have a slide, maybe you can go through with us.

Wenli Xiao (02:08.228)Okay, yeah. So yeah, today the main topic is to introduce our paper, but as we all know, recently there’s a major breakthrough in AGI and that affects robotics, that is GPT-6 Astra. There are a lot of demos from Twitter that shows people apply this model to do different tasks.

That related to robotics, like real-to-sim, sim-to-real, in-context learning, and code-as-policy. And we are very excited and we are kind of a little bit frustrating and also wondering how well robotics been solved. But we will leave the discussion to later section. And because even before the GPT-6 Astra

Already witnessed a huge improvement of coding agent this year because we felt like coding agent is a biggest thing in AI this year because starting from February or last year’s December we see like a lot of a lot very good coding agent like Opus and Codex. They can do a lot of stuff like starting from front-end to

Back-end and build a lot of great applications. And actually, people have been trying to use these models in robotics every time there’s a breakthrough in AI. And for this year there’s a CaP-X project, which we apply the and benchmark the state-of-the-art LLM model and VLM model to

Try to solve and hill-climb on the robotic simulation benchmarks and it did pretty well in simulation. And we also tried to use coding agent to just solve the very challenging manipulation benchmark. This is a very classical task which would you need to manipulate a dot to push a T-shaped object until you perfectly matches the target.

Wenli Xiao (04:31.492)And this was really challenging and people find a solution or people didn’t find a solution until diffusion policy. But now we just prompt the coding agent out of the box. This is actually Codex. We prompted to write a heuristic policy with no neural network training to achieve a hundred percent success rate. It just try and try and eventually it can like

Write very good heuristic b policy. And this applies to different coding agents like Codex Claude Code and Kimi from Moonshot. This was very exciting to me. And actually during the past two years there are many impressive works about applying coding agent for robotics.

So here I just like briefly divide them into three categories, like VLA plus harness, so you can achieve great performance that can be better than either VLA or code-as-policy itself. And there’s of course Code-as-Policy, which people apply coding agent to try and practice in simulation so they can learn some skills and later apply to the real-world.

Real world robotics. And there’s also physical autoresearch, which is the ENPIRE work that we are gonna discuss today. So the major difference between the first two categories and the physical autoresearch is even all of them applied the method in the real-world. For the first two part of first two categories, they need tons of human effort.

So if we summarize the procedure of the researchers do these projects, it’s like they apply coding agent to write code and do some practice in simulation and also with some guidance of human, it develops some impressive skills. And then the researchers, PhD students, deploy the policy in the real-world and watch the results.

Jia Xie (06:54.974)Okay.

Wenli Xiao (06:55.682)And maybe for the first trial it doesn’t work, or even the robot just do nonsense. And then the student just describes what happened and type in the terminal, and then the coding agent fix it and do another trial error and apply and suddenly it works, and then can submit paper. So in this process we observe that humans are a major bottleneck.

Because we become the agent’s actuators and sensors. Yeah, because the observation was from like six months ago, now with GPT 6 extra, maybe we don’t need to be the sensors of the robot. GPT can like if you set up a webcam in a third-person view, maybe GPT can observe that the policy doesn’t work.

But six months ago we didn’t have such good GPT and we have GPT 5.5 and we still wanna see how much it can do. So here is an example, we assemble the robot arm used some DaMiao motors because we are too lazy. We didn’t want to implement the firmware and we didn’t wanna implement the low-level control. So

We just plug in the USB-CAN to the computer that has Claude Code installed and we prompt it to implement the impedance controller for the arm and try it by itself. And with great expectation and the Claude just like just think and write, think and write, and after five minutes it finished. And here’s the result.

Michael Cho (08:50.424)Gosh. Ouch.

Wenli Xiao (08:51.676)Yeah. And we were w glad we muted this video. Yeah, we were screaming about it. Yeah, and then we took another night to redo the 3D printing and assemble the arm again. Yeah.

Chris Paxton (08:54.443)Did that break the robot? Is it? Yeah.

Michael Cho (09:03.74)My

Michael Cho (09:13.212)Well, you guys were very brave, man.

RJ He (09:13.527)I think you’re particularly making Chris sad, given his background.

Wenli Xiao (09:16.732)Yeah. So well it’s a little bit frustrating, but because autoresearch works so well in simulation that it works and your capacity to make a Twitter post. And there’s a lost also a lot of great applications of applying autoresearch in sim and digital world. So we started to think

Michael Cho (09:20.366)Ouch, ouch.

Wenli Xiao (09:43.857)What’s different between autoresearch in sim versus in the physical world? And we think that’s because in simulation or in sandbox you can do reset for infinitely many times. You have a verifiable reward, you can heal climate. And of course there’s a sandbox, you won’t break anything. But that may not be true because maybe with

A S I it can jailbreak the sandbox. But for robotics, we don’t even have this harness or sandbox in the real-world. So that’s the motivation of the entire project. So for this project we wanna take a wild step and see like what’s the minimum setup we can do to let AGI or coding agent to actually do research in the physical world.

Wenli Xiao (10:44.112)So like for this project, way b yeah.

Chris Paxton (10:47.937)Sorry, can I ask maybe this is maybe you’ll get to this later, but why can’t I just ask it to reset itself? You were saying like things like resetting and safety and all this are part of the problem that keeps you from doing it in the real-world, but like I don’t know, I’m prompting it. Why can’t I prompt it to do all those things?

Wenli Xiao (10:55.974)Uhhuh.

Wenli Xiao (11:01.52)Yeah, yeah. I and actually that’s a idea. B because so I if you if you’re gonna hear climb on something, you really wish the signal is correct and the environment is safe. And so I believe at some at certain time if you just prompt the GPT to write a safety module and write the auto reset layer.

Chris Paxton (11:09.388)Okay, we’ll get there.

Wenli Xiao (11:29.294)It can do that, and we actually observed that GPT-6 can do much better than GPT-5.5. But this is just a like system design in principle. Yeah. So in principle we need auto-research environment for the agent to hill-climb on, and we need verifiable reward to

Evaluate the policy or whatever the agent generates and then we can start the autoresearch loop. So wait actually let me switch to the website.

Wenli Xiao (12:22.298)Okay, so here is a diagram. You can see it’s basically like two autoresearch loop interleave with each other. So for the first one, it’s like we’re human user, we prompt the agent to write an auto-reset layer. Because for GPT-5.5 it lacks spatial reasoning capability and also it lacks context of the robot. For example, it doesn’t know

What does it mean to be safe for the robot? So we need a lot of human prompt and human guidance so to babysit the agent to write a reasonable auto reset r auto reset environment and to build the reward function. And after we have such environments, we just like do the run, let agent to write whatever it wants to hill-climb on the score. And

Yeah, so this sounds very like now in like to make things crystal clear, we’ll let Jia to describe what did we do for all these four different tasks. So Jia, can you hand over?

Jia Xie (13:38.569)Yeah, sure. Yeah, so

Wenli Xiao (13:40.592)Yeah, yeah, can you share screen yourself or you need me to scroll down?

Jia Xie (13:48.225)May maybe I’ll let you know when to go scroll down. So that was it?

Wenli Xiao (13:54.82)Okay, okay. Okay.

Jia Xie (13:59.926)Cool. So yeah, thank thanks for the overall like control to the whole paper. Yeah, as Wenli mentioned, all these modules are as we observe critical for this whole system to work. And to break it down a little bit and get into some details, the one of the most important modules here is reset. So for the task here, this is Push-T, which is the you can say the digital twin of the earlier task that Wenli showed, which in our case the agent like two, three months ago can solve it very, very easily. However, if we once we switch it over to real-world, a lot of the problems occur.

So re-reset is one of the most difficult part here as and sim if you think about it, you just call the reset function. But however, in the real-world, you can of course program a lot of the like controllers and just roll it out.

In a systematic way, but there are numerous edge cases that you need to very carefully engineer it so that it makes sure the whole cycle doesn’t break. Of course we can believe that Astra can have higher abilities to address these issues but still this reset

By itself is a big differentiator between real and sim. So yeah, I think we can maybe show a couple of examples. Yeah, yeah.

Wenli Xiao (16:00.593)Yeah, so I want to add some information. So for here the this Push-T task is like a real-world version of the simulation Push-T. And here it’s a l little bit funny because so the classic Push-T is down on Franka arm and where the you change the robot gripper with a long stick. So you can use a stick to pull the T, which is very like

Much easier because there’s no occlusion, but here we use YAM arm setup. And you can see if we if we use the arm to push a T, it will block the top camera. So eventually the policy here climbed on the agent. It’s like it will push a T a little bit and then go home and use the top camera to analyze and then plan again. And so here for the reset we

It stops.

Wenli Xiao (17:03.536)So we basically want the robot to pick up the T and place in the same initial position, no matter where it’s located on the desk. And wait. Yeah.

Wenli Xiao (17:29.052)Yeah, you can see

Wenli Xiao (17:37.852)Let me let me change your website. That one this one could be faster.

Wenli Xiao (17:50.855)Yeah, yeah, this is faster. So you can see no matter where place the T on the on the table, it can be reset. And we actually check the code generated by the agent. It’s it’s not as bad as I thought. It’s basically detect called CV2 to segment the red element from the top camera picture and to

In order to locate the T and then do some motion planning to pick up it and place it. So it’s basically a very classic scripted policy. And for the second task is pin insertion. And this is a little bit trickier because even for the state-of-the-art

Vision detection model, it’s hard to find the small object and even not even like provide a reasonable affordance, like grasp poses. So here, so we as human we guide the coding agent to do the detection in 2D space so that it only needs to worry about the xy direction. But that’s very hacky and

We believe that with GPT-6 we don’t need that and

But even with even with this harness or hacky way, we can achieve very impressive and robustness reset for this task. And then there’s also like we can we can have reset for the zip tie example and GPU insertion. But shamefully, that’s not very generalizable because

Wenli Xiao (19:52.155)Because with motion planning, there’s a very limited pose that is feasible for both arms to operate on. So we restricted the initial range to be a little bit small, but we didn’t fix it on the table. Yeah. That’s for the reset part. And

Do you guys have any questions? I’m not sure whether I describe it clearly.

Wenli Xiao (20:25.51)Okay.

Wenli Xiao (20:30.576)Yeah, that th that’s a that’s that’s a reset part. And then to provide clear signal to hill-climb. For the for the Push-T and Push-T pin insertion and zip tie part for and GPU insertion, it’s very easy. There’s heuristic based ways like detect the height of the object, detect the force, but for the zip tie insertion it’s a little bit tricky.

Like we asked Tonghe to write a reward function. He tried about three days and couldn’t find a reasonable one. So he actually did autoresearch based on the image or human demonstration to discover the reward function. Tonghe, do you wanna talk about it?

Michael Cho (21:25.774)I think you’re muted. Tonghe, I think you’re muted.

RJ He (21:26.878)Yeah.

Tonghe Zhang (21:28.346)Yeah, can you hear me now? Okay. So the zip-tie reward is more challenging because it relates to the form of objects that’s difficult to directly tell the size and the shape as a rigged body. So we did try a lot of ways to use simple rules to tell if it’s successful or not, but

Simce it’s deformable, the same position can have different outcomes. So w what the agent essentially does is that I give it SAM detection tools. I’ll also tell you that you have multiple wrist camera views or top camera to use to consolidate your evidence to make sure that it’s really patent through the head instead of flying over the head so that it’s actually inside the knot instead of just

Standing on the other side of the head. So the agent also gets some success and failure examples, which we can collect very easily. You can just plug it in, and then you will get a lot of examples by just doing some shaking of your hand and then you have lots of different camera views of what is a success. And you can also have what is a failure by just demonstrating a failure, which is way easier than teleoperating itself.

So the agent has success and failure examples. It has the tools to segment the zip-tie strap and the zip-tie head. And I also tell you can use multiple camera views to make sure it’s really passimg through. So the agent starts in all the research to synthesize a function, actually series of functions, to tell if the strap is passimg through the head and how long does it pass through.

And is this really intersecting with the head or it’s it’s actually not? So eventually the agent is able to hit eighty to ninety percent F1 score on the offline success or failure binary dataset that I provided. And it also knows how to accelerate this process to make the reward function very, very fast. I it reduced the time to tell from one second down to a hundred milliseconds.

Tonghe Zhang (23:46.38)So that this reward system becomes real-time and also mostly reliable. Yeah, that’s how we develop this reward function.

Wenli Xiao (23:53.764)Yeah, the code is just like go ahead, go ahead.

Michael Cho (23:54.73)Tonghe, I’m I’m yeah, I’m curious, yeah. Yeah, actually Tonghe, I’m curious, like you know, it took you three days, you know, original human effort, you gave up, you do this autoresearch thing, and then, you know, I guess the agent figures something out. Very good F1 score. So but eventually, like when you look at the actual reward function done by this agent, like it’s is it humanly like d do you think that you could have gotten it yourself? Or it’s it’s just

Is there any insight there that you th that you think it’s you couldn’t quite figure out yourself, even if you were to spend a lot more time on it?

Tonghe Zhang (24:30.83)Well yes, I think the agent does make some well, surprisimgly good, modifications to what I proposed earlier. First of all, the agent knows how to use depth camera to measure the correct 3D coordinates of the zip-tie strap and also understands how to use PCA to get the major axis of the strap. So that because you know strap can’t be can’t be tall, can be can be short.

But how do you tell the orientation of it? So it is PCA and it’s very, very efficient. It only costs you like tons of milliseconds to do the computation. And it also is surprisimgly good at accelerating this system. Like previously we were usimg PyTorch models and Codex or Cloud tell me that hey, actually there’s a way to get a vastly better way to build a TensorRT compilation graph from the SAM 3 model, which is

Something I actually didn’t know b before he told me. And you know when I search on Hugging Face, there’s actually no such implementation for SAM 3 models yet. And I was like, okay, so can you show me how to do that? And well what it does is first it tells me that you don’t actually need 1080 pixels of the original SAM resolution. You can downcast it to like 640. And then it already saves you three times the encoding time. And then you can also do the compilation which I don’t understand at all.

And the model does the compilation, it gives you a hill climb a graph telling me the F1 score is not decreasimg but the speed is getting up by you know three times. That’s something beyond my capability to make it fast. And also the agent tells me you can do some cropping before you call the SAM tool to suppress the background noise, which is I think it’s very smart. And they also hand-tune a lot of parameters like which is the best cropping ratio, what’s the best place to do the cropping. And

How to ha w if SAM returns multiple candidates, how to select the best one, how to suppress the noisy, false-positive segmentation masks. The hyperparameters are tuned by the agent, which is definitely way better than me. And not to mention the acceleration part. So it’s really very astonishing that an agent at GPT-5.5 level can also do that very, very smoothly. Not to mention what Astra can do. We haven’t tried it, but

Tonghe Zhang (26:55.584)I can only suppose the Astrack is gonna be way better than this.

RJ He (27:00.842)Do you think that’s therefore the way to go for these reward functions now? We should not ever try any other approaches given the progress of these agents?

Tonghe Zhang (27:13.998)Well, I think this in this case, segmentation and geometric relationship is enough. And the development is like taking an afternoon, then you get it’s something that’s working pretty well, although not perfect, but it’s it’s already demo level. In terms of how extensible this p system is, say I give you a random task and I tell you a brief language description of what is success.

Give us some little demonstrations of a success status and a failure status. Can Astra solve everything? Well at least we tried some examples which when you can show you guys later. We also tried some new examples with Astra. Astra knows if a task is successful or not and it knows if it failed it should retry. So I suppose that inherently this is a VLM that have a general understanding of what is success means visually.

So I don’t think it’s a difficulty to scale this up to thousands of tasks as long as you have the tokens. Yeah. And I also think that it’s possible that Astra can help you train a reward model if you give it the data. It can also obviously do other research to get a better like reward model architecture or training recipe. That’s also possible.

Wenli Xiao (28:33.094)Yeah, but I do think not even code-as-policy. Maybe code-as-reward is could be a thing if we have like AGI level VLM. So because here we demonstrate that with limited data, say if you train an end-to-end neural network, you’ll definitely like be too conservative or too aggressive, not able to do RL on top of it. But if you write code to represent the reward. Like it’s explainable and as human we can tune some of the threshold to prevent RL exploit the reward. So I feel it’s much more efficient. So maybe like it worth a systematic study to compare this with the foundation reward model in the future.

Michael Cho (29:32.064)Yeah. Yeah, fascinating. I think it’s it’s it’s incredible that we have now tools like extra. It’s you know, ‘cause like RL you know, for the longest time is like how do you even like it’s so hard to hand tune the reward, right? But now you can just okay. Figure out the reward, you know, and then maybe you get the reward out in an afternoon. Yeah. This is awesome. Okay, cool.

Wenli Xiao (29:46.95)Yeah.

Wenli Xiao (29:57.061)Yeah.

RJ He (29:57.963)So then in that case, does that mean that, you know, your paper w was it was the main insight about doing the decomposition into these four parts of it and then just letting the agents run loose in there? Almost feels a little unsatisfying that it’s just agents in each of just it all you needed was the decomposition.

Tonghe Zhang (30:15.746)Yeah.

Wenli Xiao (30:23.462)Yeah.

Jia Xie (30:25.318)I think this is like one possible way of harnessimg that because the model capability is not there, that you kind of need a we can say sandbox or sort of a like assistance for the model to do the self-improvement. But as the which is a trend, the model goes to do better and better, we would eventually envision a system where

Tonghe Zhang (30:26.106)How

Jia Xie (30:53.5)You would need minimum or even zero human effort in the whole cycle.

Tonghe Zhang (31:02.254)Also it’s very important to you know build the basics of the harness because if you literally let Astra and write all the controller code, it’s gonna waste a lot of time. And there are also like safety concerns that humans should have some input at least. However a actually Astra knows a lot about safety rules than the previous models. So ENPIRE was developed like six months ago. Six months ago we started to write the first piece of code by hand and then six months later we actually don’t know how Astra can simplify the system.

Probably lots of the code of fallbacks and if/else logic, we can just remove them from our

Wenli Xiao (31:35.972)I don’t think it can simplify the system, you know, make it even more complex.

Tonghe Zhang (31:42.04)Yeah, but probably you can find some better ways to achieve it.

RJ He (31:47.659)I guess you guys have four tasks here and there are some you know, some pretty impressive tasks. But were there tasks that you all tried that you weren’t it wasn’t ready to be able to accomplish that yet or that didn’t succeed in this framework? Just wondering.

Wenli Xiao (32:05.693)Yeah, so for all these four tasks, they are relatively easy to reset and easy to write reward function. I guess for like cloth folding or like how do you say origami? I it should be very hard or like use a key to open a door, like in a in a general way, like PyDate.

I feel from the perspective of GPT-5.5 it’s really hard but maybe I it’ll become easy with GPT-7 or GPT-8.

RJ He (32:43.798)Yeah, actually maybe if you all can expand on that a little bit. What from your experience working on this, which types of tasks work well w would work well within this framework? And which ones would? You sort of alluded to it a little bit already with you know, ones that are hard to reset, but if you can expand on that a bit more.

Wenli Xiao (33:03.472)Yeah, I think because we did this from the perspective of recursive self-improvement. So eventually once we have a very good AGI model and we can use that to train skills and then we can use a train skill to reset even challenging a more challenged new skill in the future. So like you will learn like more and more complex skill.

Tonghe Zhang (33:30.182)Yeah.

Wenli Xiao (33:33.398)And but we do think the major bottleneck is that because here we only did things in the real-world. So if for long-horizon and task and for the task that need many steps to reset, it would be very challenging. So if you if you are doing a chemical experiment and you have some intermediate state, then it’s really hard to reset.

Tonghe Zhang (34:08.206)Yeah, we’re also not directly training of super long-horizon tasks too. Yeah. Go ahead, go ahead.

Chris Paxton (34:08.749)So for the for the resetting problem. Okay, but I secure it’s for the for the I just going back the resetting problem though, like I mean it can it can learn how to reset itself, right? So like I mean I understand there may be lots of steps, but like that’s is there is it just like it would take a long time to do this? What’s the okay. Just be slow. Okay. But so if you have enough robots though, all right.

Wenli Xiao (34:27.974)Yeah, yeah, I it would be it would be inefficient. Yeah, it would be

Chris Paxton (34:34.483)Like it’s not like it can save all the programs that it’s written and all the weights for the policies it’s learned. So fundamentally, okay.

Wenli Xiao (34:41.564)Yeah. Yeah. So once we have the—wait, where is it?— Yeah, once we have this harness, we have the environment, the reward, and then it’s the same as other research in the digital world. So we start with zero success rate and give it some budget and can hill-climb to get a better performance. So here because we are conducting experiments in the real-world.

Tonghe Zhang (34:58.218)Yeah.

Wenli Xiao (35:11.738)We have very limited budget and also during that time we didn’t want to deal with like the communication between local machine and the cloud cluster. So we did it in a low-data, low-compute regime. So for every robot we provide with 5090 GPU desktop. So for the model training it can only be run on

This 5090 and also for each iteration we only give like twenty ish minutes for it to explore and so like the final policy that trained by the system is like small policy or with limited capacity. So here we connect the system

To coding agent for each of the eight robots, we log in and open a Codex session, and then we let it access the same GitHub repo so that they can do like communication between each other. And here we didn’t use a simgle agent to like control all these eight robots because there’s a huge communication delay. So we found the most efficient way

During that time is to have a shared repo for them to discuss. And for each of the agents, we prompt with the same /loop, like improve the policy and don’t stop prompt. And it starts with zero and like actually we found it’s interesting that the for one of the agents at some time it’s downloaded all the

Research paper from Sergey Levine’s lab to learn what is RL and to learn the design choices of reinforcement learning. Yeah, I told this to many of my friends at Sergei’s lab. They’re happy with it.

Michael Cho (37:16.789)Okay, okay.

Chris Paxton (37:20.611)Mm-hmm.

Wenli Xiao (37:25.616)Yeah. Yeah, because they have a lot of papers about that and many of the experiments are on a like small policy. So it’s very relevant here. Yeah. And

Michael Cho (37:35.051)Well. Cross agent help.

RJ He (37:37.536)Well

Wenli Xiao (37:42.0)Yeah. Yeah.

RJ He (37:45.684)The question here is was that did that agent do better than the others after l reading all that info?

Wenli Xiao (37:52.771)I guess so. I think for this exact experiment, I think it’s better than the paper. Yeah, because it was trying to use the exact same hyperparameters from the paper appendix, but like eventually it’s tweaked a little bit. And yeah, it followed paper to add BC regularization, tried different

Measurements of distance like L2 norm, L1 norm, and cosime similarity, different stuff. And eventually we act I actually didn’t check the final code, but it tweaked it a little bit. And it even tweaked the BC term, tweaked the batch size. And so because usually when we want to train a policy, we want to

Good base model. It actually tried to download GR00T model and π0 model, but because it’s too expensive to fine-tune them, it didn’t finish a simgle iteration after twenty in a 20 minute budget. So it for like so it gave up and just trained a small policy with like a ResNet plus a MLP. It actually even tried DINOv2, but it’s still too big for this small computer.

Yeah, so this hill-climb curve is basically the optimal result under the small budget. And we think we believe that if we give more budget, like we give the whole cluster and with some simulators I it can get more like fancy methods and fancy model they can try in different cases. Yeah.

Wenli Xiao (39:50.509)And another thing very interesting is so after publishing this paper, many people ask, did it really discover some super efficient reinforcement algorithm? And our answer is no, it’s it’s actually no, it’s just doing hill climbing. So the agent is trying to find a shortcut to solve the problem. So here,

I have a slide.

Wenli Xiao (40:35.964)Okay, yeah, so here at this step it found the it developed the compensation controller. So the idea is that because this is a pin insertion task and the height of the surface is a constant, and so that the agent figure out that once the pin just reaches the hole, it cannot move in the x and y axis.

So it determined that once the height of the p of the end effector below some threshold, it will downscale the actions in the XY direction and upscale in the Z direction. So like as long as you like get in the hole a little bit, it can move very fast to reach the end and get a positive reward. And that’s a compensation controller.

Michael Cho (41:37.48)Yeah, I think I mean th like trick tricks like that sounds very reasonable to a human being, but it’s probably just so painful to like state each one of them. But if you can just Yeah.

Wenli Xiao (41:48.081)Yeah, yeah. But yeah. Yeah, but the thing is, imagine if you are like you have a robot farm of a thousand or ten thousand robots, and then this trick will not like have any improvements on the benchmark or on the score on average. So I th I still think it has this shortcut because we have very limited budget. Yeah.

Michael Cho (42:15.638)Mm, I see.

Wenli Xiao (42:18.32)Yeah, and for this system, it has another limitation is this actually a scaling law. So initially we tried only one agent to do this task and it didn’t work. So I was very mad at that time, so I actually typed in the chat box to provide some insight. But even with this

We cannot succeed in a Push-T task. But once we really actually set up the eight agent collaboration system, it just suddenly works like in two hours. So we feel like large batch size and large number of rollouts actually matters. And yeah.

RJ He (43:06.902)So just going back to earlier you were talking about this was all done in, you know, s low the sort of low compute regime, do you think that the performance would be different if you had relaxed that constraint? In and in what ways?

Wenli Xiao (43:19.722)Yeah, I think so. Because in the very beginning it tried to fine tune π0 model or GR00T model. Yeah.

Wenli Xiao (43:32.294)So we have a th like thinking experiment that’s actually every robotic startup is a is a gigantic empire system like but with a lot of human operators to con l connect the loop.

Wenli Xiao (43:53.521)Yeah.

Wenli Xiao (43:58.513)Yeah and here’s a video that’s one once we launch a system it’s it looks so fancy. Because you can see the robot like just autonomously practiced by themselves. Yeah. And also have like scaling law of the token usage. The more agent you launch, the more token you spend.

Jia Xie (44:25.425)Wenli Xiao (44:29.21)Yeah, and also one thing to mention here is so we’re not just usimg eight coding agents to do the research. So for each computer we’re prompt it to launch agent team. So it’s like eight agent teams collaborate with each other. Yeah. So in total it may have like sixty four agents like to do the search, yeah.

RJ He (44:53.504)Do you do you predefine the roles within the agent teams? Or you let that let itself define it? Okay.

Wenli Xiao (44:59.6)I don’t define the role. Yeah, I don’t define it. Yeah. So one observation is initially because there are some open source coding agent harness like open code or Pi or Hermes, right? So we initially we customize the harness a little bit, as some human insight, but later we found that w we just use the simplest codex or Claude code, it will just work.

RJ He (45:04.522)Do you of

Wenli Xiao (45:29.732)So like I think the harness here doesn’t matter much. Yeah.

Michael Cho (45:37.811)Yeah. I’m I’m curious, like you guys mentioned you guys did some like recent except experiment with Astra. How have things changed, you know, simce this paper and now with Astra. Any insights there?

Wenli Xiao (45:46.872)Yeah, yeah. Okay. Yeah.

Wenli Xiao (45:55.665)Yeah, so one funny thing is okay, I add your intro and Tonga will have more insights. So one funny thing I think is so normally if those foundation model companies re release a new model, Ph the research of PhD students will be scooped. So right. But here we found that with more powerful GPT-6, they actually like

Fit into the like animate eliminate the biggest bottleneck of our paper. So with better model the reset will get be easier, much easier. So Empire will work better. So I think so I so that’s why I’m very excited about the GPT-6 and I’d like to try Empire on like more challenging tasks with that.

RJ He (46:52.502)Well but a lot of that is because of the premise that you all have taken, which is to be almost agent first, right? In how you design your to assume that there are agents that will do a lot of the subcomponents a lot better and that actually the agents will do better w will also improve over time, right? So that’s a fundamental premise of all the work that you’re doing here.

Wenli Xiao (47:00.401)Yeah, yeah.

Wenli Xiao (47:15.326)Yeah, yeah, that’s true. Yeah, Tonghe also tried some interesting tasks where we can discuss.

Tonghe Zhang (47:24.218)Yeah, I th I think it’s something it’s on Twitter. We have made a demonstration of usimg Astra to connect with the ENPIRE harness and the a the agent is able to do a in-context learning s that I give it a demonstration of me putting a cup on another cup, flip it 180 degrees, and then Astra is able to record the MP4, read through the frames.

Jia Xie (47:28.1)Yeah, that’s

Tonghe Zhang (47:53.601)Extract my motions, understand what I was doing, summarize this, and give it a new plan for how to replicate it itself. So this is me doing the experiment and then this is what Astra tells the robot arms to move. So that it can actually mimic the entire motion sequence and not just a instant. It’s actually an entire motion sequence to finish this job. And

Surprisimgly, because our harness provides a good motion planner, so that the first time when Astra tries to mimic my motion, it only wants to use one arm, only the left arm. Astra tries to you know unflip one of the cup and use the same arm of the left arm and then place it on top of the closest cup because he thinks it’s the easiest way to do it. But unfortunately at that angle the left arm actually reaches a joint limit. So

Jia Xie (48:29.016)Yeah.

Tonghe Zhang (48:48.366)The motion planning will always fail, no matter what kind of angle or position you plan. And then Astra actually thinks that is impossible to achieve, so that he tries to select other goals. And in order to use other goals, you probably need another hand for help. So that Astra looked inside our code base and discovered that there is a handover function that we that we once left in the code base that tells you how to do a handover in general.

And then Astra calls the handover function to hand the cup from the left hand to the right hand. So that let right hand have more freedom to choose the goal that you want to place the cup to. So it literally just out of the box in one in zero-shot it knows how to do the handover and also it’s very precise. There’s some adjustments to make sure the handover is safe. Yeah, there’s there’s a first trial and it notices that it’s actually not stable.

Wenli Xiao (49:39.056)Yeah, there’s recover.

Tonghe Zhang (49:47.159)And then it does a notch in the right hand to make sure it’s aligned and then place it down. So a power model as Astra understands the harness, understands what how to deal with motion-planning errors, how to find new ways to solve a problem, even if I didn’t actually explicitly instruct him to do it. So a stronger VLM is also able to read MP4 long videos directly so that it’s possible to unlock visual in-context learning.

Which is actually a little bit difficult for previous models, which have l worse ability and video understanding. So at least from this demo we can see the Astra is more flexible, understands the harness better and exploits the harness better, and also give you better capability to do in-context learning. That’s what we have seen right now. And we are also conducting other kind of experiments, testing the boundaries of this model.

Wenli Xiao (50:44.198)Yeah.

Tonghe Zhang (50:44.258)And I think that this work, ENPIRE is model agnostic and that’s why the stronger the base model is, the more capability we can get out of it. Yeah, and that’s also why Astra has been com continuously surprisimg us instead of replacing this work. It only makes this work more valuable.

Michael Cho (51:03.598)How do you guys think about this trade off between like, you know, having stronger and stronger you know, AGI type of models, but you can only assess from the cloud? So you take a pretty bad latency hit versus something that is that really requires closed loop, you know, high control frequency. There’s this excellent article that’s going around d yesterday, I think, from semi analysis that basically talk about this trade off, right? W do you guys have any sense, like how would that impact?

Wenli Xiao (51:33.04)Yeah, I think if we only take a look at this year, that could be a problem, but if we like foresee like five years later or even like three years later, I believe like there’s a lot of engineering tricks to make the model of inference faster, like how the voice model did. Like I feel that like GPT voice too works like it’s

Michael Cho (51:34.396)Okay.

Wenli Xiao (52:02.023)Pretty smooth and that kind of frequency can already do a lot of like task for robotics. But if you’re talking about the extreme dexterous task like require reactive behavior to the tactile, to the force, yeah maybe we still need some like low-level system, low-level controller.

Yeah, so I think maybe in short term there will be revive of sim-to-real and or even VLA, because like based on my observation, like maybe like a better GPT-6 Astra can like guide a better π0 model to do a lot of impressive tasks that we never imagined before. Yeah.

RJ He (53:02.646)Do you feel like your framework needs to be the ENPIRE framework needs to be updated for these newer models or it just kinda works?

Wenli Xiao (53:12.826)I think one thing we can add is simulation. Because I used to do a lot of sim-to-real for humanoid with Tairan And I also like revisited the RoboPapers podcast that Tairan and Haoru did for VIRAL and Doorman. And they mentioned the biggest bottleneck for sim-to-real at that time is like reward function, how to construct scene and task.

And how to do long-horizon. And now I feel GPT6 Astra almost solved all of them. So it’s very easy to construct a new thing with this coding capability. And you can do autoresearch in sim to figure out the task and reward. So I think it’s very natural to incorporate simulation in ENPIRE framework. Maybe we can add one-line prompt.

That is reconstruct this scene in simulation and do domain randomization to test your ideas. I think that’s it.

Chris Paxton (54:21.059)So I mean, just to defend viral and all the work that those gu that they were doing with that, like that was they Eureka did a lot of this stuff before then, right? So like is this fundamentally has this been enabled by like would you by improvements in GPT? Yeah.

Wenli Xiao (54:29.18)Uh-huh. Yeah, yeah.

Wenli Xiao (54:34.266)I think it’s fun yeah, I think it’s may maybe fundamentally it’s not different, but in practice like it’s totally different because with Eureka y there’s still a lot of human like insight and restriction, but now with GPT, we just let it to rewrite everything. So it unlocks much more new capabilities.

Chris Paxton (55:04.621)So it’s basically so what you’re saying is basically because like Egyptic coding works now, like this the it can close the loop that previously the human was really having to do much more of. Is that fair?

Wenli Xiao (55:14.229)Yeah, yeah.

Michael Cho (55:21.721)Yeah, so I guess I mean if you know, now that we have Astra and you know y we’re gonna have even like basically AGI series models, you know. GPT-6, seven, eight or whatever.

I mean as you build an empire right now, it seems to me there’s almost like very little human in the loop anyway. So I’m just wondering it’s it’s is there like what would be the role of a robotic researcher engineer like in this in this loop. You know. I think I mean Tonher, you mentioned earlier just now that like if you just let Astra do the whole thing, it’s gonna waste a lot of tokens. So I get that, you know.

Wenli Xiao (55:53.915)Ha.

Michael Cho (56:08.647)But let’s say if you’re very well-funded lab, you know, you know, token is not a major concern.

Michael Cho (56:17.917)Like sh should human just get out of the way and just let the whole thing loop? Like

Tonghe Zhang (56:24.078)Well I think well, if you’re a very good robotic engineer, at least when we’re dealing with controllers, you still have a little more sense, philosophically. But in terms of optimizing a system, I don’t think you need to do that all the time on your own. But you can become sort of a product manager like giving GPT-6 your requirements, like latency requirement was the what is the minimum latency requirement you have to reach.

Michael Cho (56:26.269)Where do we assert some value?

Tonghe Zhang (56:54.062)Like the safety requirements and also the flexibility design of the system, y it you will more work like an advisor or a investigator instead of just the s the people implementing the details. But I think if you have a lot of experience in robotic control, you can veto some of the so-called inventions of GPT-6, although these wild ideas will gradually be become less or less, but you can help control this process and

Your prior probably can lead GPT-6 to some better outcomes, but it also may stumble the model because you know human understanding may not always be optimal. So I think if you’re very experienced, at least you can accelerate this process to veto s those significantly impossible solutions. You can also give it a better guidance to consider all the factors, and some of the factors probably are beyond the reach of GPT-6.

And I think that’s how we can work with the model together, to reach a maximum throughput of your system reliability and also very efficiency. Yeah. And anyways there has to be a human sitting there to you know, if something really bad happens, there has to be a human to reset the scene. Yeah.

Michael Cho (58:03.06)Well, I mean mm.

Yeah.

Wenli Xiao (58:10.478)I respectfully disagree. I tried something like I tried to set up a third-person view webcam when like running GPT-6 to control robots. And the funny thing is so like I connected to full setup and like a Logitech webcam and when I

Michael Cho (58:11.355)Yeah.

Tonghe Zhang (58:15.834)Yeah, I’d like to hear your thoughts.

Michael Cho (58:24.059)Mm-hmm.

Michael Cho (58:37.512)Mm-hmm.

Wenli Xiao (58:38.032)Get closer to the table to move the object and the robot just stopped. And I checked the thinking trace. It said, there’s a human. It’s unsafe. Stop it. And then I move away. It starts to move again. And the thinking trace is now it’s cleared. So if you it’s smarter than what we think.

Michael Cho (58:56.475)Yeah. Yeah, you know, just to add on to that, right, actually, I wonder if let’s say I’m a I’m a you know, I have very good like physical intuition as a human. I’m a very good you know, contr control guy, you know, like I know how to do the controller and all this understanding. But then I operate at a human speed, right? Whereas, you know, if you have a big fleet of robots that you know, like thousands of agents like, you know

How many feedback I can only attend to one feedback at one time, you know? But meanwhile there are thousands of agents talking to each other, figuring things out.

I think I’m kinda slowing things down. So you know, it’s it’s not that there’s thousands of me, you know, but there are easily thousands of them, you know, if not tens of thousands, but hundreds of thousands of agents.

Wenli Xiao (59:37.656)Yeah.

Tonghe Zhang (59:39.736)Yeah, exactly.

Michael Cho (59:51.571)I don’t know, T Tonghe, do you not agree? Like I so I agree what you say. Like, you know, I think for until we truly solve physical AGI, let’s say, you know, like we humans still have way higher physical intelligence than this specific agent. But the thing is, there you can spin up a lot more of them in parallel, especially if it’s a token budget and you have a bigger fleet of physical robots to do reset. Like me injecting myself, like how many good human robotics controllers can you go in

Tonghe Zhang (01:00:11.127)Exactly. If

Michael Cho (01:00:19.844)At the right time, at the right place to give this just additional insight, you know, to these agents when there are just so many of them, like you y we’re completely outnumbered.

Tonghe Zhang (01:00:32.152)Sure, I think yes. What I was talking about is like if you work on that on a same simgle station. But if you’re saying that there’s one day we can scale this up to like thousands of robots and thousands of GPT-7, developing some better stuff together, I think that’s definitely humans gonna be the only burden in the system. You better remove every human being from it. Yeah, but I think humans will become more like a managerial role or become the busimess guy behind this scene.

Like you can propose what’s the next goal of this month. What kind of stuff you wanna achieve. Is it a better controller? Is it a better policy? Where is it a better robot product that’s more reliable? That’s you can humans can be the guys who give it the goal. We are the guys who running the /loop or slash goal, right? That’s still be gonna be us. We gotta provide the funding, provide the robot arms, provide the product design that we wanna achieve, and then there will be a fleet of GPT-7 developing the implementations that we don’t actually care about.

Wenli Xiao (01:01:04.954)Yeah.

Tonghe Zhang (01:01:30.956)And then we will become the verification itself. We will test the robots on our own and we will say, Okay, this is too slow. You have to work harder to get it faster or this is this is too fast. You have to be more safe. So we’re gonna become out of the technical procedure, we’re gonna become the managerial roles, we’re gonna become the busimess roles, and I think that’s where humans should really shine. And I think that’s where humans have a say, at least for now. Yeah.

Michael Cho (01:01:55.889)Hm. Yeah. So I guess so what I what I’m hearing is that you are walking back a little bit from your first statement, which is that like, you know, h humans you know, could be good engineers and we can apply our engineering knowledge at the right place at the right time. But maybe not in a case where you have literally a big fleet and a lot of tokens and in fact humans probably shouldn’t do that, the engineering part and just give very high level overview.

Tonghe Zhang (01:02:21.7)Sure.

I think if you compare one station, a human engineer plus GPT-7 versus GPT-7 itself, human plus GPT-7 is gonna be faster to arrive at a solution. Probably even not be the optimal one. But if you compare one human engineer and a simgle station worth is like a hundred GPT-7 with a very robust mechanical solution that don’t break arms that easily, and you run a day through the night, then

Michael Cho (01:02:27.474)Mm-hmm.

Tonghe Zhang (01:02:50.168)Then I can’t say that there’s anybody living on the planet that’s gonna be more smarter than a thousand or a hundred GPT-7, that’s that’s for sure. Yeah. If the engineering team can scale up and everything is very stable, I think humans should gradually step down from this engineering role. Yeah.

Michael Cho (01:03:04.593)Yeah.

Chris Paxton (01:03:09.859)Then what we do?

Michael Cho (01:03:11.372)Mm. This is quite a sobering episode.

Chris Paxton (01:03:14.071)There was this thing. So like I mean I yeah, right. I mean, so like the I like the idea of the human supervisimg it as one thing, but you know, there was this there was this thing a while back. I’m curious th about when they first saw chess called Centaur chess, where you’d have a person giving high level directions to the chess-playing engines and it would beat humans. That lasted like two years. It’s like so

Do you think that’ll happen to us?

Tonghe Zhang (01:03:43.363)I think it will slowly come to that moment. It may not happen tomorrow. But I think it’s there is some economic force driving this, driving this evolution because human even humans become a burden, become the guy who’s slowing down the process, then you have to g get rid of humans. If we become the bottleneck, then we have to resolve the bottleneck itself. And it’s gradually become a truth. In at least training models and deploying out robots, I think it’s human is the bottleneck.

If that’s the case, there’s no reason we don’t get rid of humans. And the human will move to other positions to do other jobs where machines still can’t replace us. Say, busimess guy who would or verification stuffs or the top designers probably. Yeah.

Chris Paxton (01:04:16.461)The human is the bottleneck. We’ve got to get rid of humans. This is a

Michael Cho (01:04:18.138)Well

Chris Paxton (01:04:21.811)Yeah.

Wenli Xiao (01:04:32.166)Yeah, but I think that’s a good thing because like human has been long wishing we can have a very general and intelligent robot to help us do kinds of stuff. And now I think in also in other areas, like human is still the more creative than the AI itself. Like human can come up with better applications. So

Maybe we should like as robotic engineer we are well yeah, we are truly the bottleneck of like reaching this super intelligent robot t as tool. So we should like step back and think about rethink about what’s actually important for the world. Like once we have this intelligent robot, for example, we can use it to solve the science, solve materials.

Solve all the disease on Earth or explore the space. So yeah, so that’s why I’m pretty excited about the progress. Yeah. And not a although a little bit frustrating that I cannot do robotic engineer engineering anymore, but still very excited. Yeah.

Michael Cho (01:05:54.298)Yeah, well anyway, yeah, it’s this has been a very fascinating episode I think. Very, very thought provoking, you know. And I’m I’m glad that we finally get you guys on, you know. I think it’s in a way it’s good that we this episode got delayed because like Astra is now out and then now you guys have even more like I mean directionally just speed up it, but it’s it’s it’s it’s the same direction that you guys have been on, you know. And kudos of you know

Chris Paxton (01:06:04.014)Yeah.

Chris Paxton (01:06:15.691)That’s more.

Michael Cho (01:06:21.787)To you guys even attempting this like I think like half a year ago, right? When this project you got started. So yeah, really fascinating. Yeah. Anyway, I yeah, I guess, you know, to end off, like us usually we always ask the same questions. Are there any like cool papers that you guys read recently that you wanna give a shot or to? Does it have to be

You know, robotic agency thing. Could be anything, you know, robotic research. Mm-hmm.

Wenli Xiao (01:06:56.848)Yeah, yeah. I have one the Transformer paper. I really like it.

Michael Cho (01:07:02.756)I s former chat.

Wenli Xiao (01:07:05.284)It’s like co design paper from Huiha, like from [unclear].

Michael Cho (01:07:05.521)Transformers on

Tonghe Zhang (01:07:11.268)How do you get rid of mechanical engineers?

Michael Cho (01:07:11.995)Okay.

Wenli Xiao (01:07:13.788)Exactly.

Michael Cho (01:07:19.738)Yeah, that’s right. Okay, there you go.

That’s an interesting paper. Yeah, I’m looking at it now. Okay, cool. How about Jan? How about you?

Jia Xie (01:07:31.853)I wanna give a shout out to Reward AI for their awesome demo. That’s one of the most impressive demo I’ve seen for it’s so smooth and like all of the tasks, it seems to me it’s not even tedious for acquiring those data for training such a policy like that. And it’s very close to deployable policy to make good use in robotics. So it’s very, very impressive.

Michael Cho (01:07:37.524)Yeah.

Michael Cho (01:07:41.67)Mm.

Michael Cho (01:08:00.773)Yeah, very, very impressive. Like it’s it’s a one X.

Tonghe Zhang (01:08:00.919)Sure, I w I also vote for Reward AI. It’s it’s a it’s a it’s a great it’s a great demonstration. It also shows the what you can exploit from controllers and mechatronic design, also data collection device. There’s still a little corner where roboticists still have a value in and it’s already demonstrating how to exploit it to give you a Well yeah, I think yes.

Wenli Xiao (01:08:18.076)W why do you assume they don’t use AI to do this?

Michael Cho (01:08:19.033)Yeah.

Michael Cho (01:08:23.834)Yeah.

Tonghe Zhang (01:08:29.784)Anyways, I think it that’s another direction, beyond doing planning decision making, it’s more about real-time dexterous control and I think there’s still a lot of things to do there. Astra’s still not able to do it in real-time, so I think if we wanna still do robotics, that’s gonna be the direction. Yeah.

Michael Cho (01:08:50.502)Well, thank you, thank for those like words of hope for humanity. Yeah, but anyway, like thank you so much for making the time guys. I think this is a very interesting yeah, very, very interesting episode. Right, Chris Richier. Yeah, okay, well till the next time. All right. Okay. All right, ciao.

Chris Paxton (01:09:05.315)It’s really cool stuff. Thank you. Bye.

Tonghe Zhang (01:09:06.821)Wenli Xiao (01:09:07.504)Okay, thank you. Yeah, thanks for having us.

RJ He (01:09:09.428)Yep, absolutely. Thanks guys. Really appreciate it.

Tonghe Zhang (01:09:10.126)Thank you for having us.

Wenli Xiao (01:09:13.309)Okay. Okay. Bye.