Vollständiges Transkript von DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight

from a pretrained model called RynnBrain. 49 00:04:45,194 --> 00:04:54,434 Depth information enters through intermediates fusion blocks, and world queries are used to read the hidden states, conditioned through an action-conditioned adapter. 50 00:04:54,542 --> 00:04:57,052 Evan: Okay, let's break that down a bit. 51 00:04:57,052 --> 00:05:00,592 What kind of data does the world model use during training? 52 00:05:00,650 --> 00:05:07,230 Ashley: The training data includes multi-view RGB images, language instructions, and robot states. 53 00:05:07,250 --> 00:05:13,130 The spatial world model also receives depth and a short history of hand geometry and robot state. 54 00:05:13,210 --> 00:05:17,770 All these inputs are necessary to learn an action-conditioned spatial world model. 55 00:05:17,834 --> 00:05:22,424 Evan: How does the model predict hand motion and visual changes? 56 00:05:22,490 --> 00:05:30,650 Ashley: The model predicts 3D displacements of points attached to different URDF links of the hand through forward kinematics. 57 00:05:30,650 --> 00:05:36,850 This ensures the motion prediction includes even those parts of the hand that are occluded from the camera views. 58 00:05:36,850 --> 00:05:46,400 Additionally, visual changes are captured using feature residuals extracted from RGB images by a pretrained DINOv3 encoder. 59 00:05:46,526 --> 00:05:52,086 Evan: So it sounds like they're covering both the hidden hand dynamics and the changes we can see in the scene. 60 00:05:52,086 --> 00:05:54,806 How do these predictions help in training the policy? 61 00:05:54,866 --> 00:06:01,966 Ashley: Both hand motion and visual dynamics predictions act as conditioning inputs for policy queries during training. 62 00:06:01,966 --> 00:06:09,736 The model leverages these predictions to learn how different parts of the hand and observed scene evolve under demonstrated controls. 63 00:06:09,854 --> 00:06:13,584 Evan: And what happens to the world model after training? 64 00:06:13,634 --> 00:06:21,874 Ashley: After training, the world model is frozen, and its predictive features are transferred into policy queries through a process called query alignment. 65 00:06:21,874 --> 00:06:27,374 This eliminates the need for online depth reconstruction or future generation during deployment. 66 00:06:27,494 --> 00:06:29,484 Evan: Alright, that clears things up. 67 00:06:29,484 --> 00:06:32,094 But what does deployment look like for the policy? 68 00:06:32,168 --> 00:06:39,358 Ashley: During deployment, the policy operates on current RGB observations, language, and robot state inputs. 69 00:06:39,358 --> 00:06:46,548 It uses the aligned query features to generate actions, without the complexity of the spatial world model operation. 70 00:06:46,610 --> 00:06:48,280 Evan: Interesting approach. 71 00:06:48,280 --> 00:06:51,790 Let's talk a bit about how they organized the training data. 72 00:06:51,790 --> 00:06:54,680 What sort of tasks and categories were included? 73

00:06:54,746 --> 00:06:57,206 Ashley: The training data was quite comprehensive. 74 00:06:57,206 --> 00:07:11,906 It included around 370 hours of robot demonstrations across 60 tasks, divided into five categories: spatial arrangement, assembly and contact, rotation and articulation, deformable objects, and tool use. 75 00:07:11,906 --> 00:07:16,956 This broad range of tasks provided varied examples of hand motion and scene changes. 76 00:07:17,018 --> 00:07:19,808 Evan: Wow, that's a lot of data! 77 00:07:19,808 --> 00:07:23,338 How did they estimate the scene depth for these demonstrations? 78 00:07:23,402 --> 00:07:33,382 Ashley: For real-robot demonstrations, the scene depth was estimated offline from recorded stereo RGB sequences using a method called StereoAnyVideo. 79 00:07:33,382 --> 00:07:38,022 For simulations, depth was rendered directly from MuJoCo rollouts. 80 00:07:38,066 --> 00:07:40,146 Evan: What about the training process? 81 00:07:40,146 --> 00:07:42,876 Were there different stages or phases? 82 00:07:42,938 --> 00:07:50,238 Ashley: Yes, the training proceeded in three stages: warmup, dynamics pretraining, and depth injection. 83 00:07:50,238 --> 00:07:57,138 During the warmup stage, they jointly optimized the pretrained backbone and the introduced prediction modules. 84 00:07:57,158 --> 00:08:04,608 Then, in the dynamics pretraining stage, they trained the model to learn action-conditioned interaction dynamics. 85 00:08:04,608 --> 00:08:12,258 Finally, in the depth injection stage, depth conditioning was introduced to refine the learned dynamics representations. 86 00:08:12,374 --> 00:08:15,144 Evan: It sounds like a well-structured training process. 87 00:08:15,144 --> 00:08:18,814 How did they ensure the integration of depth information into the model? 88 00:08:18,906 --> 00:08:27,836 Ashley: Depth information was injected into the backbone through trainable fusion modules, which used token-wise routing for combining features across layers. 89 00:08:27,856 --> 00:08:36,046 This cross-layer aggregated approach allowed the model to integrate depth information effectively without perturbs the existing features. 90 00:08:36,168 --> 00:08:39,728 Evan: So what about the role of the queries in this model? 91 00:08:39,728 --> 00:08:41,188 How are they conditioned? 92 00:08:41,264 --> 00:08:46,894 Ashley: A set of N learnable world queries attends to hidden states from selected backbone layers. 93 00:08:46,894 --> 00:08:56,464 These queries are conditioned through an action-conditioned adapter, which uses demonstrated controls to decode the future evolution of hand and scene dynamics. 94 00:08:56,582 --> 00:09:00,442 Evan: And during deployment, how are these queries used by the policy? 95 00:09:00,506 --> 00:09:09,766 Ashley: During deployment, the policy uses its own set of queries to read selected hidden layers, aligning with the frozen world queries to condition the action expert. 96 00:09:09,766 --> 00:09:15,546 Essentially, these queries provide predictive supervision linked to real-time action generation. 97 00:09:15,602 --> 00:09:18,692 Evan: That really helps in grasping the whole picture, Ashley. 98 00:09:18,692 --> 00:09:21,652 Thanks for explaining the method section so thoroughly. 99 00:09:21,698 --> 00:09:23,078 Ashley: You're welcome, Evan. 100 00:09:23,078 --> 00:09:25,608 And that's the end of the Method section. 101 00:09:25,608 --> 00:09:30,998 Let's move on to discuss the experiments and results to see how DexForesight performed. 102 00:09:32,259 --> 00:09:39,699 Evan: Alright, now let's dive into the experiments and results section to see how DexForesight performed. 103 00:09:39,747 --> 00:09:43,767 Ashley: The experiments are designed to answer three key research questions. 104 00:09:43,767 --> 00:09:52,267 First, can DexForesight outperform existing baselines by retaining useful predictive structures without future decoding at deployment? 105 00:09:52,267 --> 00:09:59,587 Second, is the predictive information learned by DexForesight’s world model accessible in the deployed policy representation? 106 00:09:59,587 --> 00:10:04,617 And third, how do the components of the world model contribute to the policy's performance? 107 00:10:04,743 --> 00:10:07,233 Evan: Let's start with the baseline comparison. 108 00:10:07,233 --> 00:10:11,063 How did DexForesight perform against existing methods? 109 00:10:11,115 --> 00:10:30,985 Ashley: In simulation, they evaluated DexForesight on 11 DexJoCo tasks, comparing it against two vision-language-action (VLA) policies, π0.5 and GR00T N1.7, and two world-action models, Fast-WAM and Cosmos3. 110 00:10:31,035 --> 00:10:48,065 DexForesight reached an overall success rate of 60.5%, which is 9.6 percentage points above the strongest baseline, Fast-WAM, and 11.1 points above its own backbone, GR00T N1.7. 111 00:10:48,123 --> 00:10:49,423 Evan: That's impressive! 112 00:10:49,423 --> 00:10:52,203 Did it perform consistently across all tasks? 113 00:10:52,251 --> 00:11:03,721 Ashley: Yes, it achieved the highest success rate in both single-arm and bimanual tasks, with 76.3% in single-arm tasks and 41.6% in bimanual tasks. 114 00:11:03,721 --> 00:11:11,901 At deployment, DexForesight generates actions from policy queries and does not run the spatial world model or perform future decoding. 115 00:11:11,955 --> 00:11:17,945 Evan: Wow, it sounds like DexForesight really shines when it comes to these simulations. 116 00:11:17,945 --> 00:11:20,525 What about the real-robot evaluations? 117 00:11:20,571 --> 00:11:26,881 Ashley: In real-robot evaluations, DexForesight was instantiated on the QwenVLA framework. 118 00:11:26,901 --> 00:11:33,081 They selected three challenging tasks: Close Box, Insert Battery, and Screw On Lid. 119 00:11:33,081 --> 00:11:53,561 Across these tasks, DexForesight achieved an overall success rate of 58.3%, which compared to 50.0% for WLA, 48.3% for GR00T N1.7, 30.0% for Cosmos3, and 10.0% for Fast-WAM.

120 00:11:53,619 --> 00:11:58,119 Evan: It looks like DexForesight significantly outperforms these baselines. 121 00:11:58,119 --> 00:12:00,419 How did they measure policy inference time? 122 00:12:00,513 --> 00:12:18,013 Ashley: Under identical hardware and inference settings, the policy inference time for QwenVLA was 227.4 milliseconds, and it was 230.4 milliseconds for DexForesight, only a slight increase of 3 milliseconds, or 1.3%. 123 00:12:18,013 --> 00:12:27,173 Cosmos3, on the other hand, took 930 milliseconds to generate a 16-frame future video during its video-generation stage. 124 00:12:27,219 --> 00:12:31,879 Evan: So not only does DexForesight perform better, but it's also quite efficient. 125 00:12:31,879 --> 00:12:35,879 What about the transfer of predictive information learned by the world model? 126 00:12:35,931 --> 00:12:53,541 Ashley: They tested this with a spatial foresight probe, by freezing real-robot checkpoints and training a linear ridge probe to predict future 3D hand-point displacements at three different time horizons: 0.13, 0.27, and 0.54 seconds. 127 00:12:53,591 --> 00:13:12,301 DexForesight achieved lower probing error compared to QwenVLA action-only model and GR00T N1.7, with the error decreasing from 13.59 to 10.01 millimeters at 0.54 seconds, a 26.3% reduction. 128 00:13:12,363 --> 00:13:14,333 Evan: That’s quite a reduction. 129 00:13:14,333 --> 00:13:17,563 Did they compare this approach with any direct methods? 130 00:13:17,619 --> 00:13:30,699 Ashley: Yes, they compared DexForesight with direct additions to the GR00T policy, such as adding a DINO future-residual head, a hand-flow head, both heads together, and depth with Attention Residuals. 131 00:13:30,699 --> 00:13:36,319 The hand-flow head was the strongest direct-prediction control at 53.1% overall. 132 00:13:36,319 --> 00:13:43,109 However, DexForesight achieved 60.5%, surpassing this control by 7.4 points. 133 00:13:43,155 --> 00:13:49,555 Evan: What does this suggest about their approach of using query alignment instead of direct supervision? 134 00:13:49,611 --> 00:14:02,311 Ashley: It suggests that policy latents can retain predictive structure useful for control without future decoding, and that query alignment is more effective than directly supervising visual-change or hand-motion prediction. 135 00:14:02,355 --> 00:14:05,615 Evan: What about other aspects of the world-model components? 136 00:14:05,615 --> 00:14:11,085 How did features like offline depth and 3D hand-motion targets impact performance? 137 00:14:11,139 --> 00:14:15,939 Ashley: They conducted several ablation studies to understand the impact of these components. 138 00:14:15,939 --> 00:14:21,679 For example, without offline depth, the simulation success rate dropped by 10.8 points. 139 00:14:21,679 --> 00:14:28,199 Replacing cross-layer fusion with context depth tokens yielded a success rate of 42.0%. 140 00:14:28,199 --> 00:14:34,499 Replacing hand-point displacement with future-action targets dropped success to 48.4%. 141 00:14:34,499 --> 00:14:43,179 These results support the importance of offline depth, cross-layer depth fusion, and kinematic hand-motion supervision for effective policy performance. 142 00:14:43,227 --> 00:14:45,567 Evan: That’s quite a detailed analysis. 143 00:14:45,567 --> 00:14:50,487 It seems like each component plays a crucial role in the performance of DexForesight. 144 00:14:50,577 --> 00:14:51,437 Ashley: Evan. 145 00:14:51,437 --> 00:15:00,957 These experiments and results demonstrate how DexForesight leverages implicit spatial foresight to significantly improve dexterous manipulation performance. 146 00:15:01,011 --> 00:15:04,781 Evan: Thanks for explaining the experiments and results so clearly, Ashley. 147 00:15:04,827 --> 00:15:06,237 Ashley: You're welcome, Evan. 148 00:15:06,237 --> 00:15:08,897 And that's the end of the Experiments section. 149 00:15:08,897 --> 00:15:13,927 Let's move on to the related work to understand the broader context of this research. 150 00:15:15,233 --> 00:15:26,953 Evan: Alright, let's explore the related work section to understand how DexForesight fits within the broader landscape of robot policies and future modeling. 151 00:15:27,005 --> 00:15:34,395 Ashley: The related work section highlights several key areas in the advancing field of dexterous manipulation and robot policies. 152 00:15:34,395 --> 00:15:38,095 First, let's look at future modeling for robot policies. 153 00:15:38,141 --> 00:15:39,551 Evan: Sure, Ashley. 154 00:15:39,551 --> 00:15:43,451 How does future modeling typically interact with policy learning? 155 00:15:43,517 --> 00:15:48,697 Ashley: World-action models generally couple future prediction with policy learning. 156 00:15:48,717 --> 00:16:01,327 This can be done in two ways: by jointly modeling future observations and actions, or by incorporating predictive supervision during training without requiring future generation at deployment. 157 00:16:01,407 --> 00:16:10,507 DexForesight belongs to the latter category, leveraging predictive supervision from its world model without decoding future outputs during deployment. 158 00:16:10,595 --> 00:16:11,475 Evan: I see. 159 00:16:11,475 --> 00:16:14,245 And what are some notable approaches in this area? 160 00:16:14,309 --> 00:16:21,109 Ashley: One approach is EgoWAM, which explores 3D motion flow as a predictive target for human–robot co-training. 161 00:16:21,109 --> 00:16:26,189 Other methods focus on image prediction and predictive features to structure robot policies. 162 00:16:26,189 --> 00:16:33,029 However, these approaches often do not capture the coordinated motion of articulated fingers hidden from the cameras. 163 00:16:33,029 --> 00:16:41,909 DexForesight addresses this by combining scene geometry with kinematic hand representations, jointly predicting hand motion and visual changes. 164 00:16:41,957 --> 00:16:48,387 Evan: So, how does DexForesight's approach to

spatially grounded policies stand out compared to others? 165 00:16:48,467 --> 00:16:54,627 Ashley: DexForesight integrates geometric structure directly into the policy through its spatial world model. 166 00:16:54,627 --> 00:17:08,827 This contrasts with methods that primarily focus on scene geometry, like ReconVLA and Spatial Forcing, which use auxiliary reconstruction to improve perception or align intermediate features with pretrained 3D foundation models. 167 00:17:08,827 --> 00:17:19,137 DexForesight instead targets action-conditioned spatial dynamics, capturing hand–scene evolution beyond visible image geometry, including occluded finger motion. 168 00:17:19,261 --> 00:17:21,721 Evan: That's a significant difference. 169 00:17:21,721 --> 00:17:29,571 Prioritizing the articulated hand dynamics and scene changes seems crucial for dexterous manipulation. 170 00:17:29,621 --> 00:17:30,701 Ashley: Exactly. 171 00:17:30,701 --> 00:17:42,341 This predictive supervision, covering both scene and hand evolution, allows DexForesight to operate efficiently at deployment using only RGB, language, and robot state inputs. 172 00:17:42,389 --> 00:17:49,219 Evan: How about the broader impact and contributions of world-action models in real-world robot applications? 173 00:17:49,277 --> 00:18:00,087 Ashley: In addition to EgoWAM, approaches like Fast-WAM and Cosmos3 have significantly contributed to the field by enabling predictive features and future generation for policy learning. 174 00:18:00,087 --> 00:18:09,197 Fast-WAM specifically focuses on eliminating the need for test-time future imagination, whereas Cosmos3 generates future images and actions jointly. 175 00:18:09,197 --> 00:18:17,707 DexForesight extends these ideas by focusing on implicit spatial foresight, simplifying deployment without sacrificing predictive accuracy. 176 00:18:17,765 --> 00:18:22,895 Evan: It's interesting how DexForesight builds on and refines these existing approaches. 177 00:18:22,895 --> 00:18:28,585 What about the role of auxiliary tasks like reconstruction in enhancing spatial perception? 178 00:18:28,637 --> 00:18:33,617 Ashley: Auxiliary tasks indeed play a significant role in enhancing spatial perception. 179 00:18:33,617 --> 00:18:46,697 For instance, ReconVLA uses auxiliary reconstruction to improve spatial perception, and Spatial Forcing aligns intermediate features with pretrained 3D models to enable implicit spatial grounding. 180 00:18:46,697 --> 00:18:56,107 DexForesight leverages predictive supervision from its spatial world model to achieve similar benefits without needing explicit reconstruction tasks at deployment. 181 00:18:56,165 --> 00:18:59,145 Evan: That's a smart way to utilize spatial information. 182 00:18:59,145 --> 00:19:06,025 By aligning predictive features with action learning, DexForesight seems to streamline the whole process efficiently. 183 00:19:06,077 --> 00:19:13,287 Ashley: Yes, integrating predictive spatial dynamics directly into policy latents represents a remarkable advancement. 184 00:19:13,287 --> 00:19:20,457 This helps address challenges related to occluded finger motion and scene changes, crucial for dexterous manipulation. 185 00:19:20,501 --> 00:19:27,311 Evan: Well, that gives us a solid understanding of the related work and how DexForesight innovates within this space. 186 00:19:27,311 --> 00:19:29,681 Thanks for breaking down these comparisons, Ashley. 187 00:19:29,741 --> 00:19:31,321 Ashley: You're welcome, Evan. 188 00:19:31,321 --> 00:19:33,981 And that wraps up the Related Work section. 189 00:19:33,981 --> 00:19:39,801 Let's move on to discuss the conclusion and broader implications of DexForesight's contributions. 190 00:19:41,046 --> 00:19:50,106 Evan: Alright, Ashley, we've covered the background, methodology, experiments, and related work for DexForesight. 191 00:19:50,106 --> 00:19:54,586 Let's summarize the key contributions and takeaways from this paper. 192 00:19:54,660 --> 00:19:55,460 Ashley: Evan. 193 00:19:55,460 --> 00:20:00,480 DexForesight introduces a novel framework for learning spatial foresight implicitly. 194 00:20:00,480 --> 00:20:12,120 By grounding dexterous policies with predictive representations from a frozen action-conditioned spatial world model, it significantly enhances the efficiency and performance of robotic hand manipulation. 195 00:20:12,234 --> 00:20:16,004 Evan: And its success rate improvements are noteworthy. 196 00:20:16,004 --> 00:20:19,004 Can you recap the main experimental findings? 197 00:20:19,062 --> 00:20:20,202 Ashley: Of course. 198 00:20:20,202 --> 00:20:31,952 In simulation, DexForesight improved the overall success rate from 49.4% to 60.5%, outperforming strong baselines like Fast-WAM and Cosmos3. 199 00:20:31,952 --> 00:20:40,892 For real-robot tasks, it achieved an impressive 58.3% success rate compared to 50.0% for the strongest baseline. 200 00:20:40,892 --> 00:20:48,522 These results underscore the importance of implicitly learned spatial foresight in handling complex dexterous manipulation tasks. 201 00:20:48,642 --> 00:20:52,402 Evan: It's clear that DexForesight offers substantial advancements. 202 00:20:52,402 --> 00:20:55,362 What about the methodology's efficiency during deployment? 203 00:20:55,422 --> 00:21:05,892 Ashley: DexForesight operates efficiently at deployment using only RGB observations, language, and robot state without the need for depth input or explicit future prediction. 204 00:21:05,892 --> 00:21:10,972 This streamlined approach is key to its practical utility in real-world applications. 205 00:21:11,022 --> 00:21:13,902 Evan: That's definitely an important aspect. 206 00:21:13,902 --> 00:21:21,342 The ability to retain complex predictive knowledge without computational overhead during deployment is a big win. 207 00:21:21,390 --> 00:21:22,870 Ashley: Indeed, Evan. 208 00:21:22,870 --> 00:21:34,660 By integrating predictive hand–scene dynamics directly into policy latents, DexForesight represents a significant step forward in the field of dexterous manipulation. 209 00:21:34,660 --> 00:21:41,160 It's a testament to how spatial foresight can be leveraged without adding complexity to deployment. 210 00:21:41,274 --> 00:21:46,104 Evan: Listeners, that brings us to the end of today's episode of Daily Paper Cast. 211 00:21:46,104 --> 00:21:50,854 Thank you for joining us as we explored the innovative DexForesight framework. 212 00:21:50,910 --> 00:21:57,500 Ashley: We hope you found the discussion insightful and are inspired by the advancements in dexterous robotic manipulation. 213 00:21:57,558 --> 00:22:05,428 Evan: Be sure to tune in next time as we bring you more cutting-edge research from the world of AI, robotics, and machine learning. 214 00:22:05,478 --> 00:22:11,008 Ashley: Thank you for listening, and we look forward to having you back for our next episode.