Ep#106: Flex-π: A Multi-Stream World-Action Model with Compute Flexibility

World Action Models are becoming more popular in robotics, as they learn to predict the world jointly with learning how to act on it. However, these world predictions are usually purely based on reconstructing color images from video. This is limiting, because color is far from the most important quality for a robot moving around in the world — more important are qualities like 3D geometry and object semantics. In Flex-π, Ge Yan, Jesse Zhang, and team train a 6 billion parameter world action model to do exactly this, predicting 3D pointmaps and DINO features along with color. This results in a policy which is much more demonstration-efficient, generalizes well, and can perform complex long-horizon tasks.

Learn more in Episode 106 of RoboPapers, hosted by Jiafei Duan and Ruijie He!

Abstract

World-action models (WAMs) predict the future to act better, but nearly all of them predict only RGB latents, trained purely for pixel reconstruction, with no explicit signal for the 3D geometry or object semantics manipulation needs. We find a surprising free lunch: the same frozen video-generation VAE that encodes RGB also encodes 3D pointmaps almost losslessly, with no pointmap-specific training at all. This lets us supervise Flex-π, a 6B-parameter WAM, on 3D geometry and object-centric DINO semantics alongside RGB, at no cost in new sensors, new pre-training, or inference latency. Every visual signal is projected into this shared latent space and denoised jointly with actions inside a Mixture-of-Transformers backbone; per-stream dropout with cross-modality forcing then lets a single trained checkpoint run on any subset of these streams, from a fast action-only mode to full joint generation. The result is a policy that is exceptionally demonstration-efficient and generalizes well, beating the strongest baselines by up to 2-7× on dexterous, precise, real-world bimanual manipulation tasks both in and out of distribution, all while running faster than π0.5. Our project website: this https URL

Learn More

Project Page: https://flex-pi.github.io/

ArXiV: https://flex-pi.github.io/

Thread on X

Transcript

Transcript edited and prepared with automated transcription and checked by GPT 6. There may still be errors.

Jiafei Duan (00:04.231)

Okay, sure, sure, sure. Yeah, hello everyone. Welcome to a new episode of RoboPapers. We passed our hundredth paper mark and this is, I think, 102 or 103. And and today we have like two of the returning speakers from the previous episodes, Jesse and Ge Yan, to come on and really share about their new work, Flex-π. So maybe do you guys want to Reintroduce yourself a bit more and then I think for some of the audience who never heard the previous podcast before. Yeah.

Ge Yan (00:38.151)

Yeah, yeah, hi Jiafei, and thanks for having us. So I’m right now like a second year PhD student who is going to enter my third year at the University of Washington. And my PhD advisor is Dieter Fox. And now I’m working also closely with TRI and Jesse. Yeah, Jesse, it’s yours.

Jesse Zhang (01:01.599)

Yeah, I’m a postdoc at UW and I’m advised by Dieter Fox, and Abhishek. And so I started this Ge and I worked on this project for the past how long has it been? Maybe almost a year, yeah, it’s been a while. And it’s changed quite a bit since the initial version was definitely not a WAM. so it’s been very interesting how this has turned out.

Ge Yan (01:18.833)

Yeah, it’s been a long time.

Ge Yan (01:26.567)

Yeah, great.

Jiafei Duan (01:27.145)

Yeah, it’s it’s been it’s been a quite a while. I mean, I remember I was still there when all the people collecting data.

Ge Yan (01:35.464)

Yeah.

Jesse Zhang (01:35.623)

Yeah, yeah. I mean, even like Phil presented on on ideas that led to this and they were very different than what it is now.

Jiafei Duan (01:42.677)

Yeah, yeah. Yeah. I mean I mean for audience like just context, we are from the same group, so I I know exactly what this is about. But yeah, it definitely changed. But yeah, we hope to hear from Ge exactly what Flex-π is all about. So yeah, Ge, you want to take over.

Ge Yan (01:49.629)

Yeah.

Ge Yan (02:00.219)

Okay, thank you. Okay, hello everyone. Today we’re going to introduce Flex-π, a multi-stream world-action model with compute flexibility. I think we framed it as a policy that can do one checkpoint, any observed inputs, any generated features, and the speed accuracy operating point you pick and deployment. And this work is done by me, Jinghao, Yuzhi, Lori, Minwen, Jesse, and Dieter at the University of Washington. So I think recently we’ve seen a lot of work on on WAMs.

I think it starts from the trend of very good video generation model, and people start to leverage those video priors from those very large video models and use it for action prediction. And we’ve seen a lot of progress, including like Fast-WAM, LingBot-VA, DINO most recent DINOTube paper work and and a lot of others. And I think we know that the WAMs are great, but they usually only predict RGB latent and that is specifically used to for reconstruction. And the previous work like DINO or DreamZero, so they really shows us that the world modeling objective itself enables better action prediction.

And I think it is quite intuitive right now, like with with so many results show its benefits. So I think in our work we kind of want to ask a question like why not predict additional and that is especially manipulation-centric visual signals. So we really want to leverage those other signals because we want to further push the world modeling objective strengths, and we want to do some very high precision and dexterous tasks, and we want to achieve better demo efficiency and generalization. So the right is a a multi-view.

Ge Yan (04:02.367)

RGB video shown in a YAM setup that a model do this Soft-Bag Zipping task. And so right now you can see it’s purely RGB right? And so for Flex-π we focus on like two additional signals. So for better Demo efficiency and the generalization. So one is DINO. So we know the DINO had provides very good object-centric semantics that can be very good for generalization, especially in those cloud cloud environments. And also the 3D point. I think this 3D information really provides a lot of demo efficiency, lots of model to achieve sometimes even have high performance with much less data.

As as we know, there’s no like free lunch, right? as we introduce like additional signals, it can like possibly like always introduce like additional cost. So that that really comes to one problem, because the most WAMs the the inference mode is decided at training time. So like what kind of inputs the model takes and what kind of features the model needs to generate. And we know that predicting the future makes us better, but actually w when you generate those at the inference time make it pretty slow. And it’s usually we see a lot of model that joint and generate those features, it’s very hard to do real time.

And especially with our additional visual signals, that makes even makes it even harder. So we we kind of like want to do like one model that can do both. they can do like action-only prediction, and then you can also predict the futures as other WAMs. And it can be decided at the deployment time.

Ge Yan (06:00.741)

And before introducing our exact architecture, we want to share that because we build on top of the Wan video model backbone. So this the bottom video shows after training on internet-scale videos, the Wan model, this backbone has a very good physical prior. You can see it can generate this looks reasonable like videos. Although there you can see there are still some artifacts, but it really sh it shows the model can learn like the r what waterwheel looks like a reasonable behavior, even though it’s not exactly the the right. And and one thing about the one backbone is like the VAE part. So VAE can be regarded as an encoder.

So for a VLA you can have like any those image language encoder, and for these one models, we use something called a Wan 2.2 VAE. So it’s trained by you take a sequence of input videos and you have a many many convolutional networks and you compress that into latent space. So you get a more compact latent representation, and then you reconstruct RGB video from that. And that is exactly the prediction targets of most WAMs. So I think most people understand that the WAMs predict the pixels, but it’s not exactly accurate because they’re actually predicting the RGB latent. That is optimized for pixel reconstruction.

And we can also call it like appearance. One thing we found out is this VAE is is actually largely pre-trained on RGB videos, but it can surprisingly transfer pretty well to pointmaps. So pointmap, you can imagine it as like the same shape as RGB, but the pointmap is for RGB that each each pixel like is RGB three channels, and for pointmap is just XYZ. So they have they share the same format.

Ge Yan (08:08.372)

And by normalizing, they can be within the same range. It’s just quite surprising that because this VAE can transfer it pretty well, as shown in this bottom right videos, that on the left is the ground truth, on the right is VAE reconstructed. And it’s pretty much losslessly. And this actually kind of like solves a big problem about leveraging that 3D information. Because a while ago, like when we do all the 3D learning, try to leverage those information. We all we’re always bothered by what kind of 3D encoder we should use because unlike 2D models, 3D encoding is still not mature enough.

And this actually shows a very promising direction that it kind of allows to use those 2D pre-trained model applied to a 3D pointmap and it works pretty well.

RJ He (09:04.802)

So sorry, if you just go back. maybe the first question is is this therefore then limited to the one two point two model or does this observation apply to other backbones as well?

Ge Yan (09:19.378)

Yeah, that’s a great question. I think this actually comes from so actually my project. So I’m it’s not public yet, it’s called like Uniflow. So the pointmap representation is is quite good because you can actually represent this 3D as like RGB format. And so this is not only applied by like like one particular VAE. You can also try other like DINOV3 encoder and other two D model encoder, as long as it’s like with the same format and the same shape, and you can actually get very good reasonable results from that. Yeah.

RJ He (09:58.166)

I see. So your observation is that actually quite a few of these encoders are you know, the popular ones all sort of implicitly have that point cloud information as well.

Ge Yan (10:09.146)

Yeah. I think I think I think it’s more likely like by large scale pre-training the model really learns from all these priors and it can actually transfer pretty easily to geometry, even though it’s it’s never trained on that kind of data. Yeah.

RJ He (10:28.365)

And then maybe a second question is of course, you know, this this this opens up a huge possibility with this additional signal of of the RGB D. D do you think it applies to and I’m not sure if you could get to it, but do you think it applies to other signals as well beyond point cloud?

Ge Yan (10:44.592)

Yeah, that yeah that that’s correct. So it theoretically we know like because I’m yep, not sure if it’s like kinda or like a secret, but you can actually use that for tactile. So if you represent the tactile like RGB image, I think many people have done this, right? Some we call it visual tactile. So tactile you can also represent it as an image. And you can also use the Wan VAE encoder for tactile signal. So as long as as long as like you actually share the same basically similar shape and they all get they all normalize with a reasonable Range like minus one to one, then this VAE encoder can be pretty universal for different signals here.

RJ He (11:35.565)

That’s cool. So your your insight here is really about the representation more than the sort of modality per se. And so, you know, that could potentially even be applied to other sensor modalities like audio and maybe in future even smell or something that. Okay. Cool.

Ge Yan (11:50.151)

Yeah, exactly. And and I also know like some people I know they have been already starting to try using this VAE for tactile and other modality too. Yeah. Okay, I can move next. So this comes to our flex bar architecture. So the model takes RGB input with Wan VAE encoder and pointmap share the same encoder, and they are all frozen during the training. And the DINOv3 for RGB to provide the DANO semantics. And then we can get these tokens from these different modalities. And then we we input all these tokens into a Mixture-of-Transformers (MoT) structure. So MoT is like a Mixture-of-Transformers, a mixture of experts.

So basically this all this RGB, pointmap, and the DINO latent, they all process into one visual transformer backbone, that is one 2.25 billion model backbone. And then we have a action expert transformer that takes these action tokens. And for for both input and output side, we for the input side we apply something called a stream presence mask so that it can decide, okay, each modality the probability of current presence in current batch. And the output generation mask controls what the action can and cannot see. So if action can if if all these three modality output are masked out, so then action sees nothing, so it’s called action-only.

And if they are all generated, the action can see them all. So we call it a full generation. And of course, they can see anything in between. And the probability of all this different presence and the generation are 50%. And of of course.

Ge Yan (13:44.882)

for input side we did some rejection sampling to ensure that at least one modality is present so that the model can have some information about at least

Jiafei Duan (13:56.436)

I have a I have a question, okay. So what what what’s the decision factor between using like fusion first or fusion later? So like I think you’re using an encoder to sort of encode all these different modalities. There’s also people who have like multimodal encoders that try to fuse everything at the very beginning, then you skip using MoT. So like what what was the decision factor doing this?

Ge Yan (14:21.084)

D do you mean the decision for MoT or the encoder part?

Jiafei Duan (14:24.479)

Yeah, I mean like like you are doing MoT because you are encoding all these different modalities separately and then you have them being projected into the MoT, the the MoT architecture. But then there’s also a line of work where you know people have multimodal encoders that try to fill at an early stage rather than doing the MoT approach. So is that like reason behind choosing this way?

Ge Yan (14:50.246)

Yeah, it’s a point. Yeah. So MoT, why we do MoT is like the first first thing is empirically like we found out MoT is very good. Like actually there are more some more details about the MoT, like it actually We we we view MoT as like you actually to separate and integrate them together. So what I mean is like for example, like the action expert they they the interaction between action and all these modalities are not through the whole transformer. So for example, if if this transformer visual backbone have like certain layers, probably like the first like six or seven layers is each modality is specific.

And then in the middle they cross attend to each other and at the end it it just attend to itself again. So this design is actually turns out to be effective. it actually envision area like in in other multimodality generation in computation not necessary in robotics and this kind of fine findings also transfer to robotics as well so MoT it essentially is a pretty efficient architecture in our side and that’s purely experiment-driven yeah.

Jiafei Duan (16:02.411)

Yeah.

Jiafei Duan (16:08.191)

Yeah. Yeah, I I I I think I understand. But I I think maybe the question is also like, is all the input modalities time-synced?

Ge Yan (16:19.513)

you mean the time-synceded? You mean the

Jiafei Duan (16:22.357)

Yeah, like are they all in the same frequency? Or are they like different? Because like people use usually people use joint multimodal encoder if everything is time-synced and they do this approach where you have input streams that’s not time-synced. That means some could be very high frequency, some could be low frequency, you know, and then they use this encoder with MoT setup. So I’m I’m curious, is the input all time-synced?

Ge Yan (16:25.53)

Okay.

Ge Yan (16:45.146)

Yeah, so a few things. So the the data we’ve trained on is 30 FPS. So the raw data is thirty fps, the action and video are 30 FPS. And We do some video action downsampling. So for for RGB video and pointmap, though the same frequency, so we downsample by four, so sample at 5 FPS. So the visual signal is slightly slower than action. And for the dynamic part there’s another context is like because this visual input they get into a Wan VAE encoder, right? So Wan VAE has a temporal downsampling by 4×. So for example if you have eight frames input and after Wan VAE, you have two latent frames. So it is a temporal downsampling process.

And the DINOv3 here is at the same frequency as the latent frames. So it’s two instead of the raw eight frames. So is that clear?

Jiafei Duan (17:49.415)

yeah, yeah, I I see what I mean. Okay.

RJ He (17:51.99)

Well actually but all the inputs are basically from the same image, right? In the RGB and the pointmaps and all that. So time-syncedhronization is not really the issue here.

Jesse Zhang (18:00.646)

Yeah, yeah, I think Ga is referring to the outputs. So we have a separate way that we predict outputs to save on compute and and inference costs, like training and inference costs. But I think the answer is they are time-synceded, and that’s because we don’t actually need any extra modalities, right? We’re just trying to get extra priors out of the same exact RGB image that other people would train exclusively on.

RJ He (18:24.716)

Yeah, actually I thought your answer to Jiafei’s question was going to be something along the lines of by doing this sort of late fusion approach instead of early fusion, you have more of an opportunity of adding additional signals in future as well. rather than having it the early fusion then you sort of are constrained to what what you can do. I’m not sure if that was a consideration at all, but that I wonder if yeah.

Jesse Zhang (18:48.235)

It allowed us to keep everything pre-trained also. Like that’s kind of one key factor is that we don’t have to touch the weights of the Wan VAE encoder. and it because it already for free reconstructs pointmaps super well, along with like not having to touch the DINO encoder and so on.

RJ He (19:07.564)

Okay, that makes sense. Yeah, so so I think there are elements of of being able to reuse stuff by by taking this late-fusion approach. Cool.

Ge Yan (19:16.018)

Yeah. And one additional point about MoT is it’s very flexible to add other modalities. So if in the future, if we want to add more tactile and any other modalities, it’s just plug-in. So that’s another benefit about it. Okay, anyway, during this process, we’re trying to like jointly model video dental 3D and action and all then your shared latent space. And here are some visual visualization. I think one thing to notice. Here is about the video VAE latent. So as we see here, it has a lot of noise, right? So because it’s especially trained for pixel reconstruction.

So the if so the VAE latent has a lot of like unnecessary details that the model tried to optimize, but it’s not useful for manipulation. So that’s also one motivation why we want to do DINO latent additionally. Okay, so given that the whole architecture, then it comes to the world modeling objective, right? Because as we add more data, it actually comes with a cost. And we can first review like the current world action models. There are there are actually a few like very common paradigms. So Fast-WAM is the video-then-IDM. So this is something like LingBot-VA and a series of other works.

So the model takes observation and the the model then generates the videos. And they they and then they have like a model called an IDM, basically an inverse dynamics model to extract the actions from

Ge Yan (20:59.764)

So it’s basically like a two-stage process. The problem of this pipeline is it’s the slowest because it’s not end to end if we call it. And it and it also relies on the quality of the generated video. Because in many I think in robotics there are many many corner cases that it’s very hard to generate the realistic future videos. that is very accurate. So in in in order to make it work, it will need a lot of data for coverage. And th and that’s a problem about this whole pipeline. And also the reason why we eventually didn’t choose to do this kind of style. And the second paradigm is joint generation, which is like what DreamZero did.

It basically like you get the past observation and it jointly generates video plus actions. So the video and action, they are all denoised at the same time, and although they can attend to each other. And this is faster because it it doesn’t have to first generate video, right? So it’s well it’s a s is it’s it’s end to end and it’s a single step. But it’s still not real time because you generated those videos and those videos use a lot of tokens at a test time. And for DreamZero it requires like 200 GB for inference, and then there’s no way to deploy it in a like model on a GPU with 15–19 GB of memory.

And it’s pretty slow and not n not It’s very far away from real time.

RJ He (22:43.561)

What about the data data efficiency? Comparing the two? Is it also still require the same amount of you know, needing a lot of data for coverage?

Ge Yan (22:52.518)

I think think DreamZero, I think this kind of joint generation requires less data because because it it doesn’t require the model to really generate very good videos, right? Because that’s that’s the toughest one. And and the for gener joint generation for the action part, there are some tricks or techniques to do actually. You can in a training time you can add you can make the this generated video noise you can add some noise to these videos so that the action in the training time you you can you kind of force the model to predict action and those actions can big can generate reasonable actions even though the video is not perfect.

So I think this pipeline allows generated videos not that one hundred percent good. But the Flex-πpeline it it actually requires reasonable well yeah. And okay, any question?

RJ He (23:48.796)

yeah the just just to draw a little bit deeper and that that’s interesting to me because I thought they both have similar video reconstruction losses and so the requirement is should be similar of needing to recreate that both the videos pretty

Ge Yan (24:06.438)

Yeah. Yeah, these are actually two models, right? Because for like for first one is like although people right now they can unify the forward dynamics and inverse dynamics together into one model, but But when you first predict a video, that means in many cases like for example, like if you have some partial observation. For example, if if one apple is in your bag, you want to put it a bag you want to like pull an apple out of the bag, right? But When you first observe observe it, you you have no idea what what the apple will be, right? Because you you have to take it out, then you can see what it is.

So this kind of like example to show like when the model has this very partial observation and you you it has a really need to imagine what it can come out, then there’s a problem because in many cases like you c it cannot really imagine what the outcome will really be. And another example is like about the many like physical interactions. So, like if you grasp something that’s very slippery, and it there are many or you can apply a very large force to a very light object and it can have some different physical like consequence, and that’s very hard to imagine, right?

So in this case, you you really need a model to actually learn like to predict actions that is not based on like perfect video generation, but to learn a kind of Uncertainty from that video predictions. Okay. Hope yeah, I hope that answers questions.

Jiafei Duan (25:44.426)

I have another question. So I think like so you you you mentioned about the first two approach, right? The the limb ball or I think even earlier version with like Dream Gen, they do similar stuff. And then you talk about DreamZero. you you say that you say that would you agree that there’s a correspondence between the quality of the video work with the action. So if the video generation is bad, the action is not gonna be good, correct? So maybe the question I have is that for for Flex-π, when you are generating like all these different modality, do they agree?

That means does each of these modality have to be good in order for your action to be good? Or they could they disagree but yet your action could still be good.

Ge Yan (26:26.042)

Yeah, I think they agree. So I think it is a similar observation like DreamZero. So the video quality is correlated to the action quality. It’s just like but In the training, you actually what we did too is like in the training time we added noise to that video part. So sometimes that’s also part of the reason why we can do act we can do all this generation with four denoising steps. So that means even though even though it’s kind of noisy noisy, but the the action can still be reasonably good. And that’s one part.

And another part comes down to the reason why we want to do DINO and pointmap, because because the The DINO is actually much easier to predict compared to VAE latent if we go back to this visualization. So the DINO, if we see the training curve of DINO, video, and pointmap, we will see the DINO, the loss for DINO reconstruction and pointmap reconstruction, they decrease much faster than the video VAE latent because they are easier to learn. So this also shows that even though like in some cases the video is not. Perfect, but you can still have good DINO or pointmap latent. That can actually be also good enough. Yeah. Okay.

Ge Yan (27:49.809)

Yeah, the the last paradigm is action-only, which is like Fast-WAM. And so the basic idea is like it’s basically the same as DreamZero, but in the training time it has a video loss for this generated videos, but it but the action actually don’t need to see those videos at all. And it’s it’s it’s more like a training encoder because at the deployment time the model only predicts the action. So it’s it’s a fastest version and it works because works pretty similarly to a VLA model which you only predicted action, but it has the lowest performance.

So in our usage I we found out that like Fast-WAM is a r is really a bad as some like a very chin it has in real-world. So that’s also a problem we want to fix with all these different modalities.

RJ He (28:49.386)

Sorry, but doesn’t it mean action-only is really a subset of joint generation? Because it both are doing both doing training, but then action-only is only generating actions at inference, right?

Ge Yan (29:06.756)

Yeah. They I think I think the key part is that they are both fixed at the training time. So for example DreamZero the joint generation in the training time The action and the video are joint generated, so so is the in the inference time. The first one action-only is the similar. So the difference between the inference time is whether the whether the model should jointly generate those videos. And that can actually be quite different for these two models.

Ge Yan (29:40.773)

And for Flex-π is more like it’s more like a combination of both. So it’s more like it’s like one model that can do both joint generation and action-only with the presence mask and the generation mask we have shown before. And then there are some properties about this kind of design. So first thing is of course about inference mode. So because we have like multiple modality generation, so the left l left The left one is like the model can take any input and it can generate action-only. So this is one example. And on the right is full joint. So it also jointly generates RGB, DINO, 3D with action. And it can do anything in between.

So besides the action, it can select any subset of RGB, DINO, and 3D.

Jesse Zhang (30:33.699)

One thing to add here is that this relates to Jiafei’s question from before is like, you know, you kind of need all three modalities to look reasonable, maybe, for reasonable action generation, right? well because we train with random masking of essentially both inputs and outputs. the model is sometimes forced to generate like 3D from only RGB or pointmaps from only DINO, which essentially forces it to internalize the representation here, which also again ensures that like you know you can get reasonable generations no matter what.

Ge Yan (31:10.3)

Yeah. Yeah, I think I think Jesse made a very important point here. So this is something we call cross-modality forcing. So that means the output and input doesn’t does doesn’t need to be the same. So here are two examples, like given RGB input and model can predict the dynamic and a 3D. And on the right model get DINO input, they can predict RGB and the 3D. So all these things are mutually predictive. So any input and any output they can be mutually predictive. And in this way, as as Jesse said, like it we can internalize these modalities into this one backbone.

And in this way, we we can unify appearance, which is the RGB geometry and the semantics into one representation. And that representation can be quite adaptive to f to any input and any output. Okay, so with so with all this pipeline we then pre-train on AGIBOT World, with around 500 hours of data. So this is some visualization shown the model prediction on on some unseen part of the data. So the left one is prediction and the right one is ground truth. And these three different parts is RGB DINO and pointmap. So yeah, this is actually not not that difficult it is mainly like like a three three camera view prediction.

Ge Yan (32:47.502)

And I think I think one part about it is like we we found out like because AGIBOT the data released has very bad very bad depth quality, but we can use Depth Anything 3 to label this depth. And it is just a reasonable reasonable depth prediction. And and one thing to note is like in our model we don’t necessarily require metric depth. What what we mean here is like although we use a metric depth model, but because we normalize those depths or we call them pointmaps to minus one to one. So as long as it’s consistent, even though it’s not metrically accurate, then it’s fine for our model pre-training stage.

And here or here is a simulation experiment. So one one one simulator we choose is RoboTwin. So we basically frame our Flex-π as one model, and you can for action-only we kind of classify it as like VLAs because it only does action prediction and And also with a full joint mode. So all this are the same model. They have different inference mode. And they and they basically beat all these other variants and WAMs that are pretty good. And I think yeah.

Jiafei Duan (34:12.523)

I have a I have a question. So so I think I find it very interesting because the previously the there’s a slide that you you showcase the different ways of doing this full action model, right? And then you say that Flex-π sort of is an integration, or not integration, it does both. So it does what DreamZero do, and it does what Fast-WAM does, right? So will the boundary of the performance be in between? ‘Cause in machine in machine learning we always see things, right? When someone says I have I do A and I do B, somehow often the average performance falls somewhere in between of A and B.

So I I saw that there’s like fast when baseline here, but what what how would this compare to DreamZero?

Ge Yan (34:55.082)

yeah, so no there are two things. So one thing we can s we say like they can do action-only and you can do joint generation. It’s mainly like a deployment mode that can ch you can choose. So it’s not only because through that process the model can internali internalize different representations, learn to unify these different modalities. So it’s actually The I think the key part is not only different inference mode, it’s also the model can be adapted to any input and output. And that kind of representation really matters. And it it has shown improved performance.

And for for DreamZero, because it it actually didn’t show its performance in RoboTwin right now. So we didn’t include it here. Yeah.

Jiafei Duan (35:47.659)

But i then if that’s the case, I saw most of the results. It seems like the full joint prediction seems to be the best. So is the one to recommend for people to use is the full joint.

Ge Yan (35:54.652)

Yeah, exactly.

Ge Yan (35:59.345)

Yeah, I think yeah, that’s a great question. I think it really depends on the use case, right? Because in I think b why we do actually only because it’s pretty fast. So I think people like fast policies, right? Because the inference speed actually allows the model to be more reactive, especially for those dexterous tasks. So if I think full joint is aimed for like a like the highest precision, the highest performance. And action-only is like for those less difficult tasks, that you i it doesn’t need like general pick-and-place, it doesn’t need that much precision and then you can use action-only. for very fast deployment.

So that’s kind of like users their choice for for their different attacks.

Jesse Zhang (36:49.608)

And to clarify, like we’re not in between DreamZero and Fast-WAM, right? Like we’re actually ingesting and predicting far more modalities than DreamZero. so the upper bound of full joint in theory should be higher. The problem with DreamZero is actually inferencing is so slow. it’s hard to compare unless you just compare against the baselines and and benchmarks they already did themselves.

RJ He (36:50.567)

That’s what?

Ge Yan (37:14.854)

Yeah. Yeah.

Jiafei Duan (37:15.177)

I think yeah.

RJ He (37:17.807)

And if I understand correctly, y you also you don’t just have two modes, right? Of action-only and full joint. You could have anything in between as well. Right. Do you I I don’t know if you all did any analysis on which of those modes are maybe more valuable than the others, for people?

Ge Yan (37:24.368)

Yeah. Yeah.

Ge Yan (37:36.391)

Yeah, here I s yeah, we did ablation about that. So so two ablation. One is in the training time. So the training time basically to ablate whether these modalities are useful, right? They can really help the policy. So I think in training time we find that adding different modalities generally helps performance. I think it’s quite anti it’s not surprising. And for inference time comes back to your question. It’s like We do see that with more modality generation deployment time, the model gets better performance. So there are two ways, there are two things about this. So one thing is like, so you can see here we also include Fast-WAM.

So Fast-WAM, you can see that it’s action-on, but it doesn’t have those modalities. So so if we compare a Flex-π actually versus Fast-WAM action-only, that means even though the generation is the same, but the but at the training time the model can learn different modalities. Then the model can still get better performance. So that’s goes to our one motivation is like to intern internalize those orientation for better policy learning. So this is one thing. Another thing is like we found out like with adding one modality model, the model gets better performance. And this actually is in like and this this plot is about a few RoboTwin tasks.

And we also have ex experiment and the results in real-world, we we discover exactly the same thing that full joint always like has better performance than action-only. But but the gap between different modality addition is not that much. So that also That that’s also a very good signal. You can see here, adding the video, DINO, or a pointmap, the gap between each addition is smaller than action-only plus adding any any of those modalities. So that’s that observation shows like the model tends to unify different representations. So it actually when you add like any of this modality in the model can get a reasonable reasonably good.

And when you add more and more, the model

Ge Yan (39:53.189)

increase the performance and the reaches the upper bound. So that’s kind of the observation we have about this inference time modality choice.

RJ He (40:07.878)

Then sorry, and then on the other end, in terms of the latency, with action-only versus Fast-WAM, why why is action-only faster? Is that because of some optimizations that you’re all doing or is that something in the architecture that made action-only faster than Fast-WAM?

Ge Yan (40:07.931)

Yeah.

Ge Yan (40:26.416)

Yeah, we do the very heavy optimization. I think the one core reason is like we can do few step denoising. So like Fast-WAM they can like they do 10 step denoising and for all our experiments we do four step. So that that’s naturally already a two times less denoising step, meaning two times less model forward in time. And we also do additional some engineering effort to optimize the whole network. But but I think the few step denoising is the key for that Inference here. And why I think why the model can do few-step inference without degrading the performance.

I think it goes to the question like when we have this different modality, actually it helps the model to learn more efficiently. So and and the because video is so hard to optimize, so you may need a more general step. But for other modality, you don’t necessarily need to do that. And actually, this is another benefit of using different modality. Yeah.

Jiafei Duan (41:35.566)

yeah, good. I I I think do you have like some videos of the real-world task? ‘Cause I have like one some questions regarding I mean like do you have like videos cause these are just pictures. Okay, cool, cool yes, yeah yeah.

Ge Yan (41:47.17)

yeah. I know this.

Jiafei Duan (41:50.486)

Yeah, so I have a very interesting observation. So I read the paper quite thoroughly. so so in in in the Self-Repair Gripper task, you you guys collected 11 hours of data and then you collect a DAgger. Like, can you explain a bit about that? It’s like kind of the other task I can understand, but what why what’s what is this specific task?

Ge Yan (42:11.398)

Yeah, this test is the idea of Jesse. I think but it’s creative. But it’s creative to Jesse, I think. I I think it’s mainly like we want to do some long horizon task and that’s pretty required precision and that’s very

Jiafei Duan (42:14.913)

Jesse Jesse creates a problem for you.

Jesse Zhang (42:20.301)

Yeah. Sorry.

Ge Yan (42:30.524)

difficult to do for most policies. And so because we want to explore a boundary of this policy performance. So Self-Repair Gripper like kind of like a a very good idea to do this kind of task. And regarding the yeah, okay, go ahead. Okay. Regarding the data we used

Jesse Zhang (42:43.979)

Also you go.

Ge Yan (42:51.364)

Actually, yeah, we showed here, right? Because we have indeed like 10 hour data. So this this test really need a lot a lot of data. But but actually we’re not one 100% sure that if we need this much data to do to achieve this similar success. Like like this is mainly like the data we collect is mainly for baseline. So you if you see here, So the left is task completion, right? And the l and the right is a full task success. And π0.5 is always zero and it and the ManiFlow with all this data can achieve like one success out of twenty trials. So we are basically collecting this data based on the baseline’s performance.

So because if if the data is two, it’s not this quantity, then baselines are all zero. So we we we so that’s why we choose this much data. And another observation is like even though with this much data, Pi zero five still struggles. And all this including many folds is a three policy. They struggles with this task here. Because this task involves many steps of that not a pretty difficult because for full success you have to finish the all these changes steps and that that’s what makes it fail and get zero success.

Jiafei Duan (44:15.773)

I see. and and and you mentioned that, you know, because it’s like just now I think you mentioned that VLA is more fast, less precise. So will world action model does better in this task?

Ge Yan (44:29.392)

Yeah. A few a few notions. So I think I think it also depends on pre-training, right? But in general, I think π0.5 is is far better than Fast-WAM, because Fast-WAM is like it’s action pre pre action-only prediction. We also actually test that LingBot-VA is another world action model, it’s also not good. So I s I think the reason is that it they really need a lot of data, like LingBot-VA and Fast-WAM. And that they I’m not sure about their pre-training stage, like what how many data they use compared to π0.5, but in this fine-tuning stage, it shows like they need a lot of data compared to π0.5. Yeah.

Jiafei Duan (45:15.937)

Got it.

Jesse Zhang (45:19.06)

I think one thing is like we probably don’t need the eight hundred teleop demos for any of these, even for ManiFlow to get something so. I think it’s really the DAgger demos that helped a lot with such a long horizon dexterous task, like going kind of off manifold and getting a correction. that data was really helpful.

Ge Yan (45:37.168)

Yeah. Yeah, I think in simulation it also kind of reflects this. So we also did some data scaling experiment. So if you had if you only have very few data, you you can see LingBot-VA is pretty low performance. And as you add add more and more data, so basically 500 demos kind of saturated the whole benchmark. And then they can get a similar performance. And and here

RJ He (46:03.72)

Yeah, I mean on on that sorry, on that no it seems to me like the story you’re trying to tell with the on the data versus, you know, your your outputs is that initi when you have fewer demos for your training, you really need the full joint at inference in order to get to that the delta between the two is larger. But then once you have as you increase the amount of demos, the gap narrows and you can get a you can kind of get away with an action-only inference. Is that is that sort of the conclusion we’re making there?

Ge Yan (46:40.112)

Sorry, ca sorry, do you mean like when we scale the data, the gap between action-only and a footrone gets closer? Is that what you say?

RJ He (46:47.783)

Well i it it seems like the the gap between action-only and full joint narrows as you increase with the number of demos. Right. So does is the com

Ge Yan (46:55.728)

Yeah, that that’s that’s also the observation we have. Yeah.

Ge Yan (47:02.768)

Yeah. okay, so this is the real-world experiments, like we Basically, we include π0.5, ManiFlow as a 3D policy. So one thing about ManiFlow to note here is that it is not using original point cloud. It’s also using pointmap. So it’s kind of upgraded version of ManiFlow. And Fast-WAM and the Flex-π in fact action-only and full joint here of the same model with different inference mode. And this hatched part is binary success. So we report both task completion for more granular evaluation and binary success for more. fairer comparison. And we have like three easy tasks that obviously Fast-WAM is pretty bad.

And I think the ManiFlow with pointmap is slightly better than the π0.5, but they are both worse than either action mode of Flex-π. And for these two difficult tasks, we do not include Fast-WAM because it’s not competitive in these two tasks. And we see the gap is even larger on these two most difficult. tasks and and the average I think it achieves like over 2x more improvement in in terms of like task completion and success and here yeah just I show to you do you have questions

RJ He (48:23.302)

So

RJ He (48:27.59)

never, well let me finish this and then I’ll go back to the question.

Ge Yan (48:33.484)

yeah, this this are all the videos of sh some example. I’ve I’ve shown like to Jiafei before. And so here’s about the latency performance frontier. I think why we really care about latency is i for many times you you really need a model to be reactive. So if there are many WAMs that they claim they can have very good performance, but at the cost of very high latency and that’s not that useful in real-world. So we we kind of regard it as a very top priority for for our method.

Jiafei Duan (49:15.08)

I have a question. So for the latency are you measuring by per policy call? That means they are they might be at different action chunks length or are you how how are you doing it?

RJ He (49:16.123)

Yeah.

Ge Yan (49:26.842)

Here the latency is one model forward time under like under the same 5090. And and for action chunks, like we ha we have the same as Fast-WAM. So it’s like like one one prediction is around one second is like over 30 actions. Yeah.

Jiafei Duan (49:50.941)

I see. So for for your model it’s 32 steps, action chunk. But then Pi runs fifty.

Ge Yan (49:55.227)

Yeah.

Ge Yan (49:58.823)

Yeah. I think I think here like so pi can like at the same time they they do pretty good l longer action chunks. So one thing is like one thing is like We here we we we we mainly report the model for latency instead of like the whole the whole task completion time, right? Because as you say, like if if the model predicts longer action chunk, that means for one successful episode it can take potentially less time, right? So that’s also a very good point.

And another another thing about about is like We Flex-π actually actually we also tried one version of but we didn’t include in the paper is like we try to include like three or four latent frames that correspond to forty-eight actions, forty eight actions that can be roughly matched to π0.5. And that’s also And and and in that case, if you remove because it doesn’t matter because for action-only it doesn’t matter because no matter how many frames you generated, you will still remove it at the inference time.

So at in in that in in that context, it’s it’s very roughly the same as current action-only, but you can still predict f 48 to 50 actions with similar latency. Yeah, is that clear? Yeah. Okay.

Jiafei Duan (51:28.479)

Yeah, yeah, got it.

RJ He (51:30.694)

So on the performance comparison, sorry, just compare between Flex-π now and Fast-WAM, right? Because conceptually there’s they’re on a more similar spectrum of you’re only generating actions during inference, right?

Is is the intuition there in terms of you know but they have because they have pretty significant difference in per in performance right between flex Flex-π action-only and and fast when is the intuition there that if you’re only going to generate actions at inference then during training you really need need the richer representation with, you know, point cloud and DINO and all of that, to to to be able to still get your good performance. Is that is that the key difference between action-only and Fast-WAM or is there something else?

Ge Yan (52:15.974)

Yeah, that’s true. I think I think the top reason is we have We have very good optimized representation. As you said, adding DINO and the pointmap. So I think that’s key factor one. The key factor two is like you have we have those mutually predictive, right, with this masking and dropout and the cross-modality forcing. So it kind of forces the model to learn representation that’s that’s more general and unified. And that’s really good for policy learning. So I think just two reasons I think it is a key that why even with the same action-only mode is much better than Fast-WAM.

Ge Yan (53:03.046)

Yeah, yeah, another thing is we also take the out-of-distribution generalization. So on this figure we show this this shallow bar is in-distribution and the and on the right is the out-of-distribution. And here we show the the negative is the minus of out-of-distribution performance versus in-distribution performance. And on this three simple tasks and we see Flex-π either action-only or full joint they show less degradation when evaluate in out-of-distribution mode. And here, and here shows the test case. So the left is the thing basic environment and the the right is unseen. So you can see that we add a lot of distractors.

So it’s kind of crazy cluttered and yeah. And we also try something like Unseen plate, so unseen object. to really to really extremely test the model’s generalization capability under this environment.

Ge Yan (54:13.338)

And the second thing is about the demo efficiency. by dropping so this is a simpler task, Put Plate on Rack: the model puts the plate on the rack. And is we dec decrease the data from three hundred episodes to one hundred and fifty. And we observe the model performance decrease, which is a intuitive like not surprising, and we also observe π0.5 and all these baselines they decrease the performance quite a lot, while Flex-π decrease relatively less.

Okay, so yeah, maybe I can s skip this video as this video as is many the motivation why we want to design this task is many because beyond the pick-and-place tasks, we want to test the the model’s capability in terms of very long horizon and the multi-step dexterous, high-precision task. Okay, yeah, this is like both task completion and for task success. basically in this kind of difficult task like even though with this kind of teleop and DAgger demos, I think what I think the point is that why baseline for performing so bad, right? I think we can take a look at the videos. So this this video shows the failure case of π0.5 on the left.

So this step is that the model needs to insert that gripper into the holder. So because this gripper and the holder they they only have like like 0.5 millimeters of clearance. So the model really needs to like adjust the gripper to be aligned with this holder.

Ge Yan (56:07.034)

And that part is what makes π0.5 always fails. So because we see the π0.5 have 0% full success and they always fail in this step. So that’s kind of the problem here. And we I also argue that the 3D may help in this scenario because yeah it actually because the ManiFlow can actually work in this step. So that’s one observation. And the side the second observation is like about ManiFlow. So because ManiFlow is a three policy and it successfully passed the last step about inserting gripper, but it has pro but it has a limitation in the in kind of this screwing step that it always missed a few millimeters is quite close, but still not there.

And out of our 20 tries it succeeded by one time. So it’s still very low performance. But Flex-π I can actually do this quite quite consistently good. I think I think the the reason is like as Jesse said before, I think the deck data really help a lot. So no many cases the initial state is not correct enough, like like on the left. The model the initial insertion is actually not not one hundred percent right, right? It’s not fully aligned. But but the model can learn to align its gripper further and then then it can succeed. And all the right is the same.

I think I think this finds the most surprising ones because they actually did two retries. So so if we go as this one, so at first I think the the bit just glances over the socket. So okay this one it it it it it got it, but it’s not fully it’s not fully screwing. So it it actually retries the second time and then and then it’s fully screwing and then you put it back.

Ge Yan (58:14.79)

So this kind of behavior is like quite like robust and I think that that’s the key what makes you like get some a reasonable good s success for this task.

RJ He (58:28.376)

Yeah, but it sounds like beyond the depth sorry, beyond the DAgger data, it’s also the depth representation and the the objective that’s important in in getting the result the robust behavior, right? And that that’s I think the main point.

Ge Yan (58:39.963)

Yeah.

Ge Yan (58:45.05)

Another thing about is our model is not limited to depth input. It’s not so here is one example, like the model has no depth input, but it achieved a similar performance. So with depth input, I think the task completion is ninety-five, but without depths it’s getting ninety-one. So we we do see some degradation. But but it I think the further Actually like to showcase the model already learns the the internalized representation because even though it doesn’t have depth input, it doesn’t affect the model performance too much. It still pre it still shows this kind of precision and the recovery behavior.

And yeah, I I’d like to show a few more tasks, like this one is Soft-Bag Zipping task. This one is more about dexterity, like also we tried on like a few different bags with different color. And I think this task the takeaway is it shows a similar trend as Self-Repair task but but in this time I think π0.5 and ManiFlow they’re not zero anymore so they get they get some they get some non-zero success here. And also when we deal with some unseen bags they we see that they also have like a more more significant performance drop. And this is a r this is basically a non compressor in this task.

I think it’s quite funny because you we you can see the failure case of π0.5 on the on this on this task. It’s quite interesting to see because it really shows the difficulty of this for this deformable bags and it’s

Ge Yan (01:00:42.32)

the model I actually kind of learn for this kind of task, learn like you let this zip go and then retry, right, instead of keep f keep like for pushing for this actions. And yeah, f and another thing for this is the zipper is kind of small compared to the gripper. So the kinds of to show some dexterity like how to grasp it correctly. And if you it’s not grasping correctly, you need to retry. And this kind of this kind of behavior is like is also shown like because we in our tr you know because this is a pure BC data, so no DAgger data for this task.

But in our training but in our data collection, even for human, it’s sometimes very difficult to do the task from the start to the end. So the human naturally introduced some retry behavior in the tr in the in the BC data. So that the model can still can also sh imitate this kind of behavior. But the question is the baseline seems like a π0.5 not able to fully learn this behavior very accurately. Okay, so I’m going to close. the takeaways is like training WAMs with these manipulation-centric visual signals like DINO or 3D pointmaps. It enables like both generalization and demo efficiency. And that’s crucial for real-world deployment.

Ge Yan (01:02:20.792)

And it’s nearly free because you don’t have to retrain any 3D encoder. And for all the for all this data, you can also use Depth Anything 3 to label those pre-trained data that don’t have depths. So it’s there is a pretty good, like nearly a free lunch for in on top of current WAMs. And the second second takeaway is with those with this kind of mutually predictable model input and future generation. the the WIMS actually learn a unified representation. So it is one representation that can represent represent appearance, geometry and the semantics.

And this this brings better performance and also flexible latency budgets that you can cho you can choose based on your like deployment use case and any inference modes with different input and output. And I think the third takeaway is there’s no need to fix the world-modeling objective at training time. So you I think I think the key is like we want to internalize everything because either action-only full joint or first action then IDM it’s quite limiting to fix the model to one specific mode. But why not one model can do all these different things?

And it’s in this process you’re trying to internalize internalize different representation, different signals, and all into one model. And this one model can be quite useful. And another thing is like we know the action-only is actually worse than like the full joint generation. But we see that it’s getting closer at a fraction of latency. So if you are so I think I believe with more data and more pre-training, the gap between action-only and the full full joint generation can be closer and closer. And in that way, you can get a very even faster generation without sacrificing too much performance.

Ge Yan (01:04:24.436)

Yeah, I think I think that’s all. yeah, I’m free to take more questions. Yeah.

Jiafei Duan (01:04:30.229)

Cool. Yeah. Yeah, thanks thanks for thanks for sharing with us this new work that you guys have done. I think it’s pretty cool. And definitely encourage people to check out and I I g I guess like the the code is also open source. So yeah. Raj, do you have any other questions?

RJ He (01:04:50.987)

No, very cool. I guess just the way we usually end. any papers that you all wanna give a shout out to? doesn’t have to be related to to your specific work.

Ge Yan (01:05:05.222)

Yeah. Jesse, do you want to say this? Yep. go ahead.

RJ He (01:05:06.723)

Is there is

Jesse Zhang (01:05:09.835)

I think like I think Fast-WAM was one of the first to kind of show that you that you could drop some modalities at inference time. So I think definitely check that out. Like they they also have some other ablations that were useful. I know we kind of said it’s not a good baseline, at least in the real-world, but I think their performance is simulating. But I think more the takeaways from that paper are still very relevant.

RJ He (01:05:34.179)

Ge, any from here?

Ge Yan (01:05:37.228)

I I think I think we’ll we’ll also like open source a few large scale pre-trend checkpoints and we are excited like to for the community to to to test it out like zero-shot deployment. I think we’re working on that and we hope to release you know soon.

RJ He (01:05:55.319)

Yes, it would be nice for us to no longer just use π0.5 as the baseline. So very supportive of that. That’s awesome. Thanks guys.

Jiafei Duan (01:05:59.019)

Mm-hmm.

Ge Yan (01:06:00.454)

Yeah.

Ge Yan (01:06:03.826)

Thank you.

Jesse Zhang (01:06:05.397)

Thanks for having us.

Jiafei Duan (01:06:05.653)

Yeah. Cool. Yeah. If not, good to have you guys and let’s hope to see you guys next time. bye.

Jesse Zhang (01:06:12.63)

Yep, all right. All right. Thanks.

Ge Yan (01:06:12.787)

Yeah. Thanks for having us. Bye.

RJ He (01:06:14.572)

Thanks everyone. Yep.