Fullstendig transkripsjon av the localhost:0003 | the ai hardware boom is beginning

is definitely just ruminating thoughts And 77 00:03:39,935 --> 00:03:41,475 just kinda geeking out like we love to do 78 00:03:43,530 --> 00:03:44,910 Frankcx: onto this week's show. 79 00:03:45,010 --> 00:03:49,450 we are, watching AI become a new enterprise hardware category, 80 00:03:49,450 --> 00:03:52,700 and that's the question of the week: is this, starting to happen? 81 00:03:53,020 --> 00:03:57,820 Last week, we talked about NVIDIA buying Hugging Face, that's official now. 82 00:03:57,850 --> 00:04:00,890 Hugging Face has become one of the most important places where developers 83 00:04:00,890 --> 00:04:04,520 discover, download, and distribute open weight, AI models, and NVIDIA 84 00:04:04,520 --> 00:04:09,040 says they will remain open, and NVIDIA hardware will not be required. 85 00:04:09,060 --> 00:04:10,890 But I think that raises a bigger question. 86 00:04:11,260 --> 00:04:14,930 If these models are becoming strategically important, does that place those models 87 00:04:14,930 --> 00:04:19,090 where, you know, strategy, exists too, especially around sovereignty? 88 00:04:19,110 --> 00:04:22,850 And I'd be interested, does anybody else have an second take on NVIDIA 89 00:04:22,850 --> 00:04:24,770 buying Hugging Face, that's out there 90 00:04:25,070 --> 00:04:28,470 Neil: I've seen a lot of analogies to Microsoft's acquisition of 91 00:04:28,470 --> 00:04:34,270 GitHub, And I've seen, Hugging Face referred to as the GitHub for AI. 92 00:04:34,670 --> 00:04:38,220 And at the time when GitHub was acquired, there was, some concern 93 00:04:38,230 --> 00:04:39,990 from the open source community. 94 00:04:40,200 --> 00:04:44,870 What's the future of this, really organic dev-led community now 95 00:04:44,870 --> 00:04:49,050 that it's under the umbrella of the Microsoft enterprise purview? 96 00:04:49,400 --> 00:04:52,990 Fast-forward a few years and I think a lot of those concerns 97 00:04:52,990 --> 00:04:55,500 have been resolved or eliminated. 98 00:04:55,710 --> 00:05:00,590 Some of the integrations between GitHub into Azure DevOps, for example, actually 99 00:05:00,705 --> 00:05:04,490 empowers developers to get their work done faster, and it still has maintained 100 00:05:04,490 --> 00:05:06,600 this sense of a dev-centric community. 101 00:05:06,980 --> 00:05:08,330 So now take a step back. 102 00:05:08,330 --> 00:05:11,490 There are obviously, some parallels and some differences between 103 00:05:11,490 --> 00:05:16,620 this moment with Hugging Face being part of the NVIDIA family. 104 00:05:16,990 --> 00:05:20,240 However, I think from a security standpoint, ease of 105 00:05:20,240 --> 00:05:24,140 deployment, management, the topic is enterprise for today. 106 00:05:24,160 --> 00:05:28,670 I think in terms of enterprise readiness, Hugging Face has been off limits. 107 00:05:28,870 --> 00:05:31,000 You can only pull models from sanctioned tools. 108 00:05:31,260 --> 00:05:34,970 Now I think there's an opportunity for a lot of those open source, open weight 109 00:05:34,980 --> 00:05:39,560 models on Hugging Face to now be validated by enterprises in an expedited fashion 110 00:05:39,890 --> 00:05:41,080 Frankcx: Yeah, I couldn't agree more. 111 00:05:41,150 --> 00:05:45,130 I think it puts a little bit more weight behind the idea of open weight, right? 112 00:05:45,180 --> 00:05:48,890 All right, let's move on to the second news that I wanted to bring to the table. 113 00:05:48,890 --> 00:05:53,240 I saw that Citibank and then Vercel, which operates some AI gateways, 114 00:05:53,280 --> 00:05:58,090 is reporting that they are using open-weight models more than 50% 115 00:05:58,090 --> 00:06:01,740 of the time for their development and computation that they're doing. 116 00:06:01,750 --> 00:06:04,260 I think that is just an astounding number. 117 00:06:04,530 --> 00:06:07,830 And in fact, it's happening really quickly because they were showing that 118 00:06:07,830 --> 00:06:11,865 in June, just a couple months ago, they were only using it about 29%. 119 00:06:12,760 --> 00:06:17,550 But as of August 25th, they were up to 53% of their development and 120 00:06:17,570 --> 00:06:21,250 AI workloads that they measure are happening on open-weight models. 121 00:06:21,470 --> 00:06:25,860 So what that means is they're literally consuming or building their own AI. 122 00:06:25,940 --> 00:06:29,990 I find it fascinating, and I think the finserve or financial services 123 00:06:29,990 --> 00:06:34,540 market is ripe for, local AI as well as, open we- weight models. 124 00:06:34,540 --> 00:06:39,420 But I know that, not to name any names, but I know Neil and, and Laura, you're 125 00:06:39,420 --> 00:06:42,400 out there talking to these financial services customers day in and day out. 126 00:06:42,430 --> 00:06:45,240 And again, I'm not asking you to disclose anything, but hey, 127 00:06:45,250 --> 00:06:46,570 what do you think about this? 128 00:06:46,917 --> 00:06:48,137 Laura: I just have so many questions. 129 00:06:48,187 --> 00:06:49,807 I wanna know what they're running it on. 130 00:06:49,807 --> 00:06:53,597 I wanna know who's leading this, what kind of workloads are they running on it. 131 00:06:53,627 --> 00:06:58,726 I just, would love to know more about this Yeah, I've even in some of the handful of 132 00:06:58,726 --> 00:07:03,106 conversations I've been in, or even just some kind of, research that I've been just 133 00:07:03,106 --> 00:07:07,306 kind of seeing like it's… That the fin serve industry above all, I think they're 134 00:07:07,306 --> 00:07:12,186 so sensitive about the way that their data traverses internally, and externally. 135 00:07:12,285 --> 00:07:13,379 Frankcx: How about Disney. 136 00:07:13,379 --> 00:07:17,309 I think that, they had big concerns about their IP and just the… 137 00:07:17,389 --> 00:07:21,279 Disney is the images that they create or the characters that they

have. 138 00:07:21,689 --> 00:07:26,909 And so being able to have Marvel and Star Wars and, just old-timey Disney 139 00:07:26,909 --> 00:07:31,539 characters being released into, closed weight models is a huge concern to them. 140 00:07:31,539 --> 00:07:34,539 And that's their IP, that's how they make money, that's who they are. 141 00:07:34,539 --> 00:07:38,579 And so, being able to run open weight models even if they built their own 142 00:07:38,579 --> 00:07:43,599 data center or, being able to deploy it onto a future RTX Spark device under 143 00:07:43,599 --> 00:07:47,049 the governance of a Windows device, it just feels like, these are places 144 00:07:47,049 --> 00:07:50,699 where you can protect that IP and have that sovereignty that they are needing, 145 00:07:50,879 --> 00:07:54,509 Neil: I agree with that, and I'm seeing really a three-tier model 146 00:07:54,519 --> 00:07:57,649 approach starting to crystallize in the enterprise space. 147 00:07:57,959 --> 00:08:01,279 There is the AI that you can run directly on your endpoint, which we've 148 00:08:01,279 --> 00:08:03,249 talked about in episode one and two. 149 00:08:03,529 --> 00:08:06,409 There's cloud AI, which of course we've talked about, and there's 150 00:08:06,409 --> 00:08:10,199 this emerging middle tier, the edge server, the department level servers. 151 00:08:10,479 --> 00:08:13,129 GB300s are all the rage right now. 152 00:08:13,359 --> 00:08:17,669 I think RTX Dev Box on the horizon could also play an important role, 153 00:08:17,859 --> 00:08:22,899 albeit not at the compute capacity that GB300's bringing to the table, but the 154 00:08:22,899 --> 00:08:27,439 ability for a department to, leverage some of these more advanced models. 155 00:08:27,479 --> 00:08:32,689 Kimi 3 is all the rage right now, 350 gigs of RAM necessitated. 156 00:08:32,939 --> 00:08:36,849 That middle tier actually offers a strategic advantage, not just 157 00:08:36,849 --> 00:08:40,389 from a cost savings perspective, but also from a privacy, right? 158 00:08:40,579 --> 00:08:45,259 Now taking a step back, local, whether it's on the endpoint or the edge server, 159 00:08:45,449 --> 00:08:50,259 does not equal automatic compliance from a FERPA perspective, right? 160 00:08:50,269 --> 00:08:51,409 From a HIPAA perspective. 161 00:08:51,659 --> 00:08:56,089 However, no round trip to the cloud lessens the attack surface, 162 00:08:56,089 --> 00:09:02,169 Frankcx: All right, well speaking of models, the MBZUA Institute of 163 00:09:02,179 --> 00:09:05,859 Foundational Models, and they've released six different models 164 00:09:05,859 --> 00:09:07,449 for different hardware classes. 165 00:09:07,699 --> 00:09:11,679 anything from one billion parameters which could run literally on a phone 166 00:09:11,739 --> 00:09:15,749 like a, Pixel phone, all the way up to three hundred and seventy-five billion 167 00:09:15,749 --> 00:09:18,709 parameters which is probably more of a

cloud based model that you would run 168 00:09:18,709 --> 00:09:20,759 in that space or a server base model. 169 00:09:21,199 --> 00:09:25,299 But I actually downloaded the thirty-six billion parameter model and ran it 170 00:09:25,299 --> 00:09:31,389 across, two of my, old dual 3090, workstation class and honestly, I came 171 00:09:31,389 --> 00:09:33,099 out with some pretty good results. 172 00:09:33,119 --> 00:09:36,649 But I think what's interesting to me more about these models 173 00:09:36,819 --> 00:09:38,059 is they're not quantized. 174 00:09:38,129 --> 00:09:41,259 They're not like taking a big trillion parameter model and shrinking it 175 00:09:41,259 --> 00:09:44,689 down and shrinking down the accuracy which we talked about in episode two. 176 00:09:45,509 --> 00:09:49,389 are literally models that are built for different classes of devices. 177 00:09:49,409 --> 00:09:54,889 They're trained on the size of the device that they're gonna be deployed onto so, I 178 00:09:54,889 --> 00:09:59,569 find that fascinating, towards an approach and I think that still means that people 179 00:09:59,569 --> 00:10:04,009 are taking the time to build models for different sized hardware because of the 180 00:10:04,019 --> 00:10:07,959 benefit and payoff and the sovereignty and data protection and, essentially 181 00:10:08,039 --> 00:10:09,949 tokenomics that goes around that. 182 00:10:10,269 --> 00:10:13,209 But I'm wondering if anybody else has any comments towards this deployment 183 00:10:13,229 --> 00:10:17,419 and this approach that K2 Horizon is taking over maybe the quantized 184 00:10:17,479 --> 00:10:19,609 approach of, scaling down models 185 00:10:23,264 --> 00:10:26,394 Chauncey: Yeah, I'm actually, as a hardware guy, I think there are few, you 186 00:10:26,394 --> 00:10:27,644 know, few of us are hardware guys on here. 187 00:10:27,834 --> 00:10:30,534 Like, I think that to me is very appealing, right? 188 00:10:30,534 --> 00:10:33,054 Because if you start thinking about being able to really kind of target 189 00:10:33,054 --> 00:10:36,774 the model to the right workload, the right person, the right device, the 190 00:10:36,854 --> 00:10:38,234 efficiencies that we'll gain out of that. 191 00:10:38,254 --> 00:10:42,094 But also I think like the measure of capability matching to that 192 00:10:42,094 --> 00:10:44,524 efficiency, 'cause right now when we quantize, we typically kind of… 193 00:10:45,904 --> 00:10:48,804 We push out some things that we don't necessarily wanna push out in 194 00:10:48,804 --> 00:10:50,344 places in order to get it smaller. 195 00:10:50,412 --> 00:10:53,704 And so if we can kind of go the other way, making sure it has the right foundation to 196 00:10:53,704 --> 00:10:57,654 run things, that's very exciting to me in terms of like what that could potentially 197 00:10:57,654 --> 00:11:01,294 mean for some of these very specific models, what they would be able to run. 198 00:11:01,334 --> 00:11:04,284 I'll be talking about later on my device about how I'm trying to run 199 00:11:04,314 --> 00:11:08,184 some CLI, intelligence on a local model and I'm running on a quantized model. 200 00:11:08,204 --> 00:11:10,564 It's, been an interesting experience. 201 00:11:10,584 --> 00:11:13,534 I'll dive into that later, but if I can build a model to be able 202 00:11:13,534 --> 00:11:17,624 to do that upfront and be very kind of generalized approach to 203 00:11:17,624 --> 00:11:19,994 that, I think there's, definitely some cool future to that for sure 204 00:11:21,159 --> 00:11:23,899 Frankcx: Now I'm wondering, like for Laura, like as you're talking to customers 205 00:11:23,899 --> 00:11:29,399 there in the UK, you starting to see their adoption of these open weight models? 206 00:11:29,979 --> 00:11:32,549 and, you know, where are they finding them, and what are they 207 00:11:32,549 --> 00:11:35,999 gravitating towards, you know, i- in those different spaces? 208 00:11:36,099 --> 00:11:39,779 it feels like, you know, the, even the industries across the UK with, from 209 00:11:39,779 --> 00:11:43,739 financial services to other spaces probably have some duplicat- duplicity to 210 00:11:43,739 --> 00:11:45,509 what the United States is seeing as well. 211 00:11:46,703 --> 00:11:47,553 Laura: Yeah, absolutely. 212 00:11:47,703 --> 00:11:52,043 And I think it's a really interesting space, particularly within Europe and 213 00:11:52,053 --> 00:11:56,743 the UAE because we have a lot of changing landscape when it comes to sovereignty 214 00:11:56,743 --> 00:12:01,573 rules and kind of restrictions that are coming into place more in the cloud scope. 215 00:12:01,573 --> 00:12:06,983 So it's definitely pushing a lot more customers to consider these things or look 216 00:12:07,023 --> 00:12:09,173 towards open weight models or local AI 217 00:12:09,383 --> 00:12:14,753 And I think particularly UAE as well, there's so many privacy concerns that 218 00:12:14,803 --> 00:12:19,683 sit, within that region that so many companies just haven't adopted cloud 219 00:12:19,683 --> 00:12:24,613 at all and therefore AI transformation has been completely blocked for them. 220 00:12:24,653 --> 00:12:28,313 So this is a really exciting space and particularly this model coming 221 00:12:28,313 --> 00:12:32,223 out of the UAE is, really exciting to me 'cause I think a lot of customers 222 00:12:32,223 --> 00:12:35,843 and conversations that I have at the moment are much more solution driven. 223 00:12:35,853 --> 00:12:39,753 They're not coming from the model and all of the cool capabilities 224 00:12:39,773 --> 00:12:41,333 that sit on that side first. 225 00:12:41,333 --> 00:12:43,083 It's a bit more in the reverse order. 226 00:12:43,523 --> 00:12:47,653 but this is something that really, really excites me 'cause I think this 227 00:12:47,653 --> 00:12:52,563 could be a, point that does start to flip that

conversation back again 228 00:12:52,583 --> 00:12:56,883 and combine this level of technology alongside things like MoE models. 229 00:12:56,913 --> 00:13:00,643 It's super cool, and I think it enables us to be a bit more strategic when we 230 00:13:00,643 --> 00:13:05,173 think about devices and mapping that to different users, different devices, 231 00:13:05,473 --> 00:13:07,623 planning headroom on devices around that. 232 00:13:07,643 --> 00:13:09,393 There's so much cool stuff that can be done 233 00:13:11,019 --> 00:13:14,599 Frankcx: do you hear people throughout Europe and the UK 234 00:13:14,599 --> 00:13:18,689 and into the regions of the UAE, gravitating towards specific models? 235 00:13:18,689 --> 00:13:21,859 it's a little bit of a loaded question 'cause Mistral obviously plays a pretty 236 00:13:21,859 --> 00:13:26,439 big role in those open weight, categories outside of what China is building. 237 00:13:26,929 --> 00:13:32,639 I feel like it's interesting that Mistral is taking, such a big story around, the 238 00:13:32,799 --> 00:13:36,609 EU's, need for sovereignty and privacy. 239 00:13:36,979 --> 00:13:39,979 and Mistral is kind of like saying, "Hey, we're here to answer some of 240 00:13:39,979 --> 00:13:43,289 that." but I don't know if you're hearing those names pop up very often 241 00:13:44,255 --> 00:13:47,075 Laura: Funnily enough, Mistral not so much, no. 242 00:13:47,115 --> 00:13:49,905 I think it's been a lot of conversations at the moment have 243 00:13:49,905 --> 00:13:52,465 been really, purpose driven. 244 00:13:52,485 --> 00:13:57,145 So it's things like, Whisper comes up a lot because people are specifically 245 00:13:57,205 --> 00:14:01,695 looking for transcription and translation and those kinds of storylines. 246 00:14:02,095 --> 00:14:06,365 which is, you know, if you to search for what do I use with 247 00:14:06,375 --> 00:14:07,765 transcription, what's a good model? 248 00:14:07,765 --> 00:14:09,905 Whisper's obviously gonna be the first one that comes up. 249 00:14:09,905 --> 00:14:13,215 So I do think there's still a bit more of an education piece to be done around 250 00:14:13,215 --> 00:14:18,695 this, and I think it's evolved a lot, and I think cloud customers that are starting 251 00:14:18,695 --> 00:14:20,745 to make this journey towards local. 252 00:14:21,025 --> 00:14:25,025 Unfortunately, I think the default has become things like Azure Local and 253 00:14:25,495 --> 00:14:27,775 sovereign cloud options, and I really… 254 00:14:27,785 --> 00:14:31,065 It's one of those things that I really wanna spearhead into more, that I 255 00:14:31,065 --> 00:14:35,465 think local AI on device AI is being kind of left out of the picture or 256 00:14:35,465 --> 00:14:38,385 is still trying to catch up and these emerging technologies that we have 257 00:14:38,395 --> 00:14:43,046 now are going to fill that space for us hopefully I saw so many use cases 258 00:14:43,056 --> 00:14:48,306 with customers where they had certain data sets or certain workflows that 259 00:14:48,536 --> 00:14:50,116 they weren't allowed to put into cloud. 260 00:14:50,126 --> 00:14:52,456 They couldn't get the sign-off to be able to do that, and then it just 261 00:14:52,456 --> 00:14:53,926 became, okay, well, we'll stop here. 262 00:14:53,946 --> 00:14:57,196 so I loved coming along to this team 'cause I could jump in and 263 00:14:57,196 --> 00:14:58,786 say, "Aha, we have a solution." 264 00:14:59,961 --> 00:15:00,271 Frankcx: Right? 265 00:15:01,011 --> 00:15:04,631 we have devices that are smart enough to run AI at that level too, 266 00:15:04,731 --> 00:15:05,621 Laura: And that has grown 267 00:15:05,751 --> 00:15:09,201 Gonna say this, that's grown so much even in the past six months, like since 268 00:15:09,311 --> 00:15:11,421 I've started working on this stuff. 269 00:15:11,421 --> 00:15:16,281 the use cases that we could talk about now versus then are 10 times more 270 00:15:16,281 --> 00:15:18,101 powerful, which is really, really cool 271 00:15:18,108 --> 00:15:21,306 Frankcx: Well, talking about that cloud side and the Azure side, I'll 272 00:15:21,306 --> 00:15:25,486 drop in the fifth or the fourth piece of news is more around OpenAI. 273 00:15:25,546 --> 00:15:30,086 Literally last night, that would be Thursday, announced the release of Astra. 274 00:15:30,216 --> 00:15:32,656 this is GPT-6 called Astra. 275 00:15:32,656 --> 00:15:36,976 And OpenAI says Astra is the first model to reach critical cybersecurity 276 00:15:36,976 --> 00:15:39,586 capability in its preparedness framework. 277 00:15:39,636 --> 00:15:44,426 and in testing, it's been known to find vulnerabilities, and work-working 278 00:15:44,426 --> 00:15:47,823 out exploits in those chains that no other model has been able to find. 279 00:15:47,966 --> 00:15:52,756 So as a personal story, I've been developing this small meeting room app 280 00:15:52,756 --> 00:15:57,816 for a, partner of mine, and what I'm doing is I'm using Cloud Code to actually 281 00:15:57,816 --> 00:16:03,296 write the application, but I'm using Qwen to do kind of a QA and testing of it. 282 00:16:03,616 --> 00:16:08,126 But I'm super excited to take Astra because if I can take kinda this hybrid 283 00:16:08,126 --> 00:16:12,986 approach of using both cloud and local AI to help develop an application, 284 00:16:13,316 --> 00:16:17,276 I could use Astra now to come in and literally say, "Go look across all the 285 00:16:17,276 --> 00:16:22,746 code as an independent," look at my Git repository and literally dig into it and 286 00:16:22,746 --> 00:16:26,666 dive into it and give me an answer back as to, cybersecurity vulnerabilities." 287 00:16:26,696 --> 00:16:28,900 Chauncey: I'll poke a badger and just say, "Just be careful as you start 288 00:16:28,930 --> 00:16:33,410 setting that A-Astra up," that you don't want it to maybe go start pinging, 289

00:16:34,120 --> 00:16:36,910 15,000 different chats in some German chat room somewhere because that's 290 00:16:37,120 --> 00:16:38,690 something that Reuters is reporting on. 291 00:16:39,430 --> 00:16:43,700 So it kind of slipped into the news but I think that is an interesting take right? 292 00:16:43,880 --> 00:16:49,630 You think about like models are becoming crazy capable. 293 00:16:49,680 --> 00:16:52,170 I don't wanna say intelligent because I think we're still trying 294 00:16:52,170 --> 00:16:57,880 to avoid this idea likening them to AGI at this point But their ability 295 00:16:57,880 --> 00:17:01,490 to solve a problem 'cause that's effectively what it's doing right?. 296 00:17:01,490 --> 00:17:05,600 if you told it to go fix something or find something or break something it's 297 00:17:05,600 --> 00:17:08,610 gonna do that And it's going to take every measure possible to be able to go do 298 00:17:08,610 --> 00:17:14,760 that In this case someone for some reason thought it to set up some sort of chatroom 299 00:17:14,770 --> 00:17:18,620 Somewhere had literally fifteen thousand chats where they were talk…It was bots 300 00:17:18,630 --> 00:17:20,820 talking to each other Its almost like that 301 00:17:21,635 --> 00:17:21,745 Frankcx: or 302 00:17:21,910 --> 00:17:22,810 Laura: yeah yea 303 00:17:22,885 --> 00:17:25,265 Frankcx: Facebook for but like Mott yeah. 304 00:17:25,760 --> 00:17:30,660 Laura: But did that on his own which is really Interesting .So i would 305 00:17:30,660 --> 00:17:34,350 Just Say If Your Going To Use it Secure your Application Make Sure 306 00:17:35,330 --> 00:17:37,210 giving Some Well Defined Parameters 307 00:17:37,569 --> 00:17:40,149 I just wanna know what's the gossip between the agents. 308 00:17:40,239 --> 00:17:40,959 What are they all, 309 00:17:41,079 --> 00:17:41,399 Frankcx: I know. 310 00:17:41,399 --> 00:17:42,439 What are they talking about? 311 00:17:43,719 --> 00:17:46,689 I loved all the conspiracy, like they're creating their own language 312 00:17:46,719 --> 00:17:48,389 that only AIs can understand, 313 00:17:48,689 --> 00:17:50,869 Neil: I think there's two topics here to, 314 00:17:50,894 --> 00:17:51,624 Frankcx: go ahead, Neil 315 00:17:51,949 --> 00:17:53,539 Neil: two topics here to dive deeper on. 316 00:17:53,539 --> 00:17:56,039 One is code generation at the edge. 317 00:17:56,329 --> 00:18:01,159 Frank agreed with your approach of using these cloud models, as supplements 318 00:18:01,159 --> 00:18:06,639 to your local code generation studio, as the ability to run 20 billion, 319 00:18:06,989 --> 00:18:10,769 30 billion, 70 billion parameter models locally on your device. 320 00:18:10,959 --> 00:18:14,009 Seventy billion parameters is where we're seeing some really effective 321 00:18:14,049 --> 00:18:18,549 codegen models at the edge, and the hardware floor has risen now 322 00:18:18,599 --> 00:18:20,689 to where, yeah, that's feasible. 323

00:18:20,979 --> 00:18:23,709 However, the cloud can still serve an important role. 324 00:18:23,779 --> 00:18:25,029 I like the bookend approach. 325 00:18:25,049 --> 00:18:26,819 Sometimes I'll start with the cloud for… 326 00:18:26,939 --> 00:18:30,709 I have an idea, I need to turn that into a PRD and do the initial 327 00:18:30,709 --> 00:18:34,009 architecture, and then I can do some of the scaffolding and code locally. 328 00:18:34,249 --> 00:18:37,999 And then at the end, I'll review with, a cloud-based model as well. 329 00:18:38,199 --> 00:18:41,229 So I agree with the approach there, in terms of code generation. 330 00:18:41,609 --> 00:18:45,779 On the security side, I think this is, an area we can go much 331 00:18:45,779 --> 00:18:48,699 deeper into in future episodes. 332 00:18:49,069 --> 00:18:55,019 If the bookend on the tail end is, reducing code vulnerabilities, I believe 333 00:18:55,029 --> 00:18:58,139 that actually alleviates a lot of the concerns we're seeing in the market. 334 00:18:58,569 --> 00:19:02,399 With that bookend approach, though, by scanning for vulnerabilities at 335 00:19:02,399 --> 00:19:06,519 the time of code generation, we get that, latest and greatest, right? 336 00:19:06,519 --> 00:19:09,749 We get that latest and greatest as code is being generated, and we 337 00:19:09,749 --> 00:19:11,249 don't have this significant backlog. 338 00:19:11,259 --> 00:19:13,069 All that to say, Frank, I love the approach. 339 00:19:13,279 --> 00:19:15,299 I think code generation at the edge is the future. 340 00:19:15,469 --> 00:19:20,599 I think security vulnerability scanning at the edge is the future as well, 341 00:19:20,599 --> 00:19:24,049 and hopefully our security teams don't have these massive backlogs when 342 00:19:24,059 --> 00:19:27,709 these frontier models keep, exposing more vulnerabilities in the future. 343 00:19:28,962 --> 00:19:31,812 Frankcx: I mean, it's good for the long term but probably 344 00:19:31,812 --> 00:19:33,112 really scary for the short term. 345 00:19:33,412 --> 00:19:33,745 Neil: Totally. 346 00:19:33,935 --> 00:19:36,675 Chauncey: I mean you're definitely moving in the right direction, right? 347 00:19:36,675 --> 00:19:40,385 Like if you think about even Apple just kind of came out and mentioned that 348 00:19:40,385 --> 00:19:44,135 they were surprised with the amount of people that were going out and 349 00:19:44,135 --> 00:19:47,055 buying Mac Minis to run things locally. 350 00:19:47,195 --> 00:19:49,135 and it wasn't just a bunch of like hobbyists. 351 00:19:49,135 --> 00:19:52,815 enterprises are going out and doing this which by the way I think points to a 352 00:19:52,825 --> 00:19:57,435 future that we're all looking at from and Microsoft perspective like this is 353 00:19:57,485 --> 00:19:59,425 obviously a direction that we have to go. 354 00:19:59,735 --> 00:20:04,415 So be really kinda curious Frank, Laura, Neil like you guys-- you guys are

really 355 00:20:04,455 --> 00:20:07,675 kind of hitting the streets and living this stuff or building it locally and 356 00:20:07,675 --> 00:20:13,355 doing your own thing so give some of that baseline on what this should mean from 357 00:20:13,355 --> 00:20:16,955 a… What are we really caring about? 358 00:20:17,095 --> 00:20:19,095 What are some of the core things that people should be 359 00:20:19,295 --> 00:20:20,215 thinking about in this case? 360 00:20:20,515 --> 00:20:24,625 Neil: I'll open by saying that open-weight models are no longer this 361 00:20:24,905 --> 00:20:28,565 philosophical reference, this pie in the sky, "Oh, you can have your cake 362 00:20:28,565 --> 00:20:31,975 and eat it too. You can get access to these very cutting edge models but 363 00:20:31,975 --> 00:20:35,415 it's behind closed doors and it's a black box." It's becoming a reality. 364 00:20:35,425 --> 00:20:40,215 Again, the case study referenced of 29% open weight to over 50% 365 00:20:40,215 --> 00:20:41,895 in an abbreviated period of time. 366 00:20:42,225 --> 00:20:48,135 these open-weight models are, becoming a reality for customers primarily due 367 00:20:48,135 --> 00:20:50,035 to cost concerns, rising cloud costs. 368 00:20:50,085 --> 00:20:51,635 we need more agentic AI. 369 00:20:51,645 --> 00:20:55,515 We wanna run it locally so that the meter's not spinning with every turn. 370 00:20:55,905 --> 00:20:58,875 That's the hot topic right now, but then we also talked about 371 00:20:58,875 --> 00:21:02,185 some of the privacy and regulatory concerns that are alleviated. 372 00:21:02,495 --> 00:21:05,325 And just furthermore to emphasize your point, Chauncey, that these Mac 373 00:21:05,355 --> 00:21:09,295 Minis aren't just going to hobbyists folks coding in, mom's basement here. 374 00:21:09,435 --> 00:21:11,855 I've heard multiple times enterprise customers saying that 375 00:21:11,875 --> 00:21:14,985 Mac Minis are actually part of their enterprise deployment. 376 00:21:14,995 --> 00:21:16,605 They're in their data centers right now. 377 00:21:16,605 --> 00:21:20,535 They're extending capacity for these models with Mac Mini. 378 00:21:20,545 --> 00:21:25,045 So, it's no longer this pie in the sky dream that we're marching towards. 379 00:21:25,285 --> 00:21:27,825 It's a reality right now September 2026 380 00:21:27,826 --> 00:21:29,567 Chauncey: you guys paying attention to what's coming out of IFA? 381 00:21:29,567 --> 00:21:31,605 I should probably also mention that IFA is happening right now. 382 00:21:31,965 --> 00:21:34,825 I mean, like even Lenovo just kind of, came out and started talking 383 00:21:34,825 --> 00:21:39,985 about their RTX Spark adoption NVIDIA reinforced that they've got 384 00:21:39,985 --> 00:21:41,405 stuff coming out with RTX Park. 385 00:21:41,685 --> 00:21:45,565 So I think like one thing that I am super excited about is like we keep talking 386 00:21:45,565 --> 00:21:49,605 about Apple Macs and stuff, and I think because largely Windows has somewhat 387 00:21:49,615 --> 00:21:53,495 suffered in this space, like we just really haven't had the ability to do 388 00:21:53,495 --> 00:21:56,655 this in a way that our end users and our customers have been really wanting. 389 00:21:56,905 --> 00:21:59,805 but I think the RTX spark is really gonna help us change on that front. 390 00:21:59,825 --> 00:22:02,655 And so I'm super excited to see that really starting 391 00:22:02,655 --> 00:22:03,975 to come together here soon. 392 00:22:04,015 --> 00:22:07,385 and I think like if anything, IFA is a good example that this 393 00:22:07,385 --> 00:22:11,585 isn't just a Microsoft push, like this is an industry wide push. 394 00:22:11,635 --> 00:22:15,255 Like it has to happen across all OEMs, all manufacturers. 395 00:22:15,255 --> 00:22:18,275 I think Laura and Neil, y-you guys talk to me about all the time about 396 00:22:18,275 --> 00:22:21,885 this idea of like T-shirt sizing and workload rightsizing and how does 397 00:22:21,885 --> 00:22:27,095 this like and help our customers and help other people think about like 398 00:22:27,415 --> 00:22:32,045 okay well I might not need a RTX park. 399 00:22:32,245 --> 00:22:36,145 I might need Just for what I do, I just need some basic device that 400 00:22:36,145 --> 00:22:37,585 still runs a local model though. 401 00:22:37,885 --> 00:22:39,815 and how do we start of that? 402 00:22:39,825 --> 00:22:42,815 So that's actually one thing I'm actually curious about again the rest of this 403 00:22:42,815 --> 00:22:45,025 team is that kind of work load model. 404 00:22:45,355 --> 00:22:48,795 So far a lot of what we talked about requires that upper end. 405 00:22:49,145 --> 00:22:52,355 But like Is there some basic stuff that we can bring locally? 406 00:22:52,805 --> 00:22:57,315 Neil: I'll share an example of a medical school of the future initiative 407 00:22:57,355 --> 00:22:58,705 that I'm working on right now. 408 00:22:58,905 --> 00:23:04,825 currently the solution is a medical school training classroom, three different 409 00:23:05,295 --> 00:23:09,805 cameras, multiple microphones, lot of data collected sent to the cloud for analysis. 410 00:23:10,065 --> 00:23:16,455 The goal is to bring this local primarily from, an accelerated time to deliver 411 00:23:16,655 --> 00:23:18,505 these results back to the students. 412 00:23:18,555 --> 00:23:21,955 If you recall your time in university, you took that final exam, you may 413 00:23:21,985 --> 00:23:25,505 not get your results for three weeks, four weeks, after the exam. 414 00:23:25,505 --> 00:23:26,665 You forgot what it was all about. 415 00:23:26,985 --> 00:23:29,825 Not the type of, feedback we're trying to give our doctors of the future, 416 00:23:29,825 --> 00:23:31,235 our current medical school students. 417 00:23:31,735 --> 00:23:35,415 We're actually able to accomplish a lot of this multimodal 418

00:23:35,415 --> 00:23:37,235 analysis with the Surface Pro. 419 00:23:37,515 --> 00:23:44,255 We don't need a beefy device to do some of the, image and, and audio analysis. 420 00:23:44,275 --> 00:23:48,425 And so I, to the point of T-shirt size models in Laura's earlier 421 00:23:48,425 --> 00:23:50,415 comment of starting with the workload. 422 00:23:50,645 --> 00:23:54,645 Start with the w- the workload whether it's cloud based today, maybe it wasn't 423 00:23:54,655 --> 00:23:56,875 feasible in offline settings before. 424 00:23:57,165 --> 00:24:01,205 And, and once we identify that workload, we can provide the right small language 425 00:24:01,205 --> 00:24:03,025 models to unlock that capability. 426 00:24:03,465 --> 00:24:06,775 The density, the size of those small language models then 427 00:24:06,775 --> 00:24:08,255 determines the T-shirt size. 428 00:24:08,535 --> 00:24:13,005 We're getting great, great results with small language models running on 16 gig, 429 00:24:13,025 --> 00:24:19,045 32 gig RAM devices in mobile settings like with the Surface Pro doing speech to text, 430 00:24:19,355 --> 00:24:21,345 analyzing that transcription locally. 431 00:24:21,395 --> 00:24:24,535 Think about an insurance agent out in the field snapping a 432 00:24:24,535 --> 00:24:28,425 picture of a damaged vehicle or car, running an analysis of that. 433 00:24:28,555 --> 00:24:32,575 We don't need to be walking around with a GB10 or an RTX Spark Laptop Ultra. 434 00:24:32,595 --> 00:24:35,175 We can actually do that with, what we refer to as more of the 435 00:24:35,175 --> 00:24:38,375 medium or large sized hardware that we have available today. 436 00:24:38,745 --> 00:24:43,315 And so to tie a bow on this, I think, you know, for years it was a race to 437 00:24:43,335 --> 00:24:47,195 build the best model, and then we were building bigger models and running them 438 00:24:47,195 --> 00:24:48,505 on these general purpose accelerators. 439 00:24:49,535 --> 00:24:53,915 Now we're seeing the hardware being optimized for these specific classes 440 00:24:53,915 --> 00:24:57,775 of models, the T-shirt sizes, and then delivering these corresponding 441 00:24:57,775 --> 00:24:58,855 workloads to our customers 442 00:25:00,749 --> 00:25:03,399 Chauncey: You kinda mentioned that you're talking as if this is happening 'cause 443 00:25:03,399 --> 00:25:05,099 I think like it is happening, right? 444 00:25:05,299 --> 00:25:10,479 and I think based on my limited knowledge that like it's actually 445 00:25:10,489 --> 00:25:12,589 not that terribly difficult either. 446 00:25:12,619 --> 00:25:15,369 Like it is obviously you gotta learn something new, you gotta dev, you 447 00:25:15,369 --> 00:25:19,029 gotta kind of get your hands dirty on some code but that like-- That 448 00:25:19,029 --> 00:25:22,559 the platform itself actually somewhat exists today to be able to adopt this 449 00:25:22,559 --> 00:25:26,739 relatively easy and that

I would say one of our hindrances of this kind of 450 00:25:26,739 --> 00:25:30,059 exploding is just a lack of awareness. 451 00:25:30,109 --> 00:25:32,779 People don't even know that this is possible and there's only so many 452 00:25:32,779 --> 00:25:35,349 Neils and Lauras that can get out there and get in front of customers. 453 00:25:35,359 --> 00:25:38,319 So do you feel like that's-- is that the truth? 454 00:25:38,359 --> 00:25:41,559 I mean are we kind of at the stage where customers should just kind of be 455 00:25:41,609 --> 00:25:45,349 looking for this or are we still kind of maybe a few months, a few years 456 00:25:45,349 --> 00:25:46,809 out before that technology catches up 457 00:25:46,826 --> 00:25:50,136 Yeah, And it's some feedback that I get quite a lot from customers is, 458 00:25:50,476 --> 00:25:54,101 "Oh, it's actually way easier to implement this stuff than we thought." 459 00:25:54,216 --> 00:25:56,946 There's this kind of expectation that there's gonna be a whole load of new 460 00:25:57,101 --> 00:26:00,146 tooling required, new skill sets. 461 00:26:00,486 --> 00:26:03,626 But in reality, you're using all the same tooling that you used to. 462 00:26:03,636 --> 00:26:08,386 You can embed these, models and local solutions into, if you're developing 463 00:26:08,386 --> 00:26:11,926 with VS Code, if you're using GitHub, if you're using Copilot CLI, all of 464 00:26:11,926 --> 00:26:15,456 these different, native toolings that we have at our fingertips, you can 465 00:26:15,456 --> 00:26:18,186 just embed local solutions within that. 466 00:26:18,216 --> 00:26:23,986 And it is so simple, like the examples that Neil gave are great, and I have the 467 00:26:23,986 --> 00:26:26,106 same kind of, scenarios with customers. 468 00:26:26,106 --> 00:26:29,096 That sometimes it is just something really simple that they do every day, like 469 00:26:29,096 --> 00:26:33,776 comparing two documents or two invoices that they need to constantly do this. 470 00:26:34,206 --> 00:26:40,136 They can't afford to have a subscription for AI for everyone just to be 471 00:26:40,136 --> 00:26:41,506 able to do that one simple task. 472 00:26:41,536 --> 00:26:43,636 But also, it doesn't make sense to do that. 473 00:26:43,646 --> 00:26:47,956 You can literally just build one simple tool that does that same solution, uses 474 00:26:47,956 --> 00:26:51,356 local AI to do it, and everyone can just roll it out on their own devices 475 00:26:53,884 --> 00:26:54,764 Neil: Great call out, Laura. 476 00:26:54,774 --> 00:26:58,384 And I'll say the messaging that's resonating effectively with customers 477 00:26:58,574 --> 00:27:02,354 for those building with Microsoft Foundry is a set of large language models in 478 00:27:02,354 --> 00:27:05,564 the cloud that Microsoft has vetted and it's plug-and-play for developers. 479 00:27:05,944 --> 00:27:09,224 We've now extended Microsoft Foundry with the Foundry local toolkit.

480 00:27:09,514 --> 00:27:12,488 This is a set of curated models designed to run locally on the device. 481 00:27:12,544 --> 00:27:15,224 So it's the same SDKs, right? 482 00:27:15,224 --> 00:27:18,224 It's the same deployment and management tools to get these 483 00:27:18,224 --> 00:27:20,094 models out to your end users. 484 00:27:20,434 --> 00:27:25,234 I will say that the only additional tooling that some customers are 485 00:27:25,284 --> 00:27:30,034 often adopting is Agent 365, which is the ability to govern 486 00:27:30,034 --> 00:27:32,034 and manage these solutions. 487 00:27:32,384 --> 00:27:35,174 I met with a financial services customer, and they said we now have 488 00:27:35,184 --> 00:27:38,684 more agents than human employees, and getting our arms wrapped around 489 00:27:38,684 --> 00:27:40,114 what are all these agents doing? 490 00:27:40,394 --> 00:27:42,074 Are all these agents authorized? 491 00:27:42,084 --> 00:27:43,284 Should we consolidate some? 492 00:27:43,284 --> 00:27:46,284 Should we promote some of these so that there's not redundant solutions? 493 00:27:46,604 --> 00:27:50,464 That's where I think the governance and management and really Microsoft 494 00:27:50,464 --> 00:27:52,124 secret sauce comes into play here. 495 00:27:52,424 --> 00:27:56,954 It's one thing to spin up a local AI instance using Llama CPP and get an agent 496 00:27:56,964 --> 00:27:58,914 running for one person, for the hobbyist. 497 00:27:59,104 --> 00:28:02,674 It's another thing to deliver a consistent experience across 498 00:28:02,674 --> 00:28:06,304 your end users, and that's where we're seeing the pendulum shift. 499 00:28:06,644 --> 00:28:09,814 The question is no longer can this solution run locally? 500 00:28:10,014 --> 00:28:14,224 It's how can I run this solution securely in a local environment 501 00:28:15,411 --> 00:28:19,161 Frankcx: So yeah, it's super exciting that these pieces are coming together 502 00:28:19,161 --> 00:28:22,831 that enterprises can actually start to adopt and put into that space. 503 00:28:23,331 --> 00:28:27,631 just to keep the show moving, that brings us into the tension of the week. 504 00:28:28,011 --> 00:28:31,821 so this week's tension, and I'm on the hot seat, but I'm gonna just kinda 505 00:28:31,821 --> 00:28:35,471 put it out there something a little spicy, is that last week we talked 506 00:28:35,481 --> 00:28:39,971 about the twenty-five dollar per million token dying and cloud-based 507 00:28:39,971 --> 00:28:41,931 models starting to reduce their costs. 508 00:28:42,291 --> 00:28:45,501 But I would argue that whereas the twenty-five d-dollar token 509 00:28:45,511 --> 00:28:49,571 dies, that's when enterprises and the AI hardware boom begins. 510 00:28:49,971 --> 00:28:52,021 And it's not because the cloud is going away. 511 00:28:52,261 --> 00:28:57,001 it's not because every AI workload suddenly belongs on somebody's desk 512 00:28:57,001 --> 00:28:58,671 or

inside their private server room. 513 00:28:59,041 --> 00:29:03,601 My case is honestly very simple: open weight models are becoming capable enough 514 00:29:03,851 --> 00:29:09,761 to work and do real work, and enterprises are increasingly demanding those weights. 515 00:29:10,071 --> 00:29:13,081 We've seen that from Citi, we've seen that across the board. 516 00:29:13,451 --> 00:29:17,691 And these model designers, like what we're seeing out of the UAE, are taking big 517 00:29:17,691 --> 00:29:22,221 models that can work on hardware that is deployed at different scale, from phone 518 00:29:22,231 --> 00:29:25,661 to PC, to server, to, workstation class. 519 00:29:26,041 --> 00:29:30,581 we're seeing hardware vendors build specific hardware that 520 00:29:30,581 --> 00:29:32,481 can essentially host local AI. 521 00:29:32,771 --> 00:29:36,381 And Apple with Mac mini is showing us that these early demands are 522 00:29:36,381 --> 00:29:40,391 much stronger than they anticipated, especially in the enterprise space. 523 00:29:41,091 --> 00:29:43,391 so that's where this paradox comes. 524 00:29:43,421 --> 00:29:49,031 If intelligence is getting cheaper we are gonna consume, the thought would 525 00:29:49,041 --> 00:29:50,781 be is we would consume more of it. 526 00:29:51,151 --> 00:29:54,551 Are we gonna start inventing more and more uses for AI? 527 00:29:54,901 --> 00:29:59,441 more agents, more inference, more workloads, and we would say that 528 00:29:59,521 --> 00:30:03,481 which intelligence should we rent and which intelligence should we own? 529 00:30:03,831 --> 00:30:07,601 So I'd ask you to kind of prove me wrong that, you know, or maybe just agree 530 00:30:07,601 --> 00:30:11,151 along the way, but I've kinda opened it up for discussion on the open debate. 531 00:30:11,531 --> 00:30:15,111 If the cloud cost pennies, you know, why would we buy hardware 532 00:30:15,371 --> 00:30:16,941 that can essentially drive it? 533 00:30:16,981 --> 00:30:20,631 But my argument would be there's more now uses for hardware than ever, 534 00:30:21,001 --> 00:30:24,031 and we would just find the right place for that in the right time. 535 00:30:24,351 --> 00:30:26,771 But opening up further discussion here 536 00:30:29,876 --> 00:30:34,386 Chauncey: I think a lot of folks have, strong opinions about cloud usage. 537 00:30:34,876 --> 00:30:39,406 cloud does allow for a lot of scale. 538 00:30:39,766 --> 00:30:45,266 Cloud allows for a lot of very quick growth, and You can just do things at 539 00:30:45,286 --> 00:30:49,236 that compute level in particular, that like I think locally, I don't know that 540 00:30:49,236 --> 00:30:51,465 we'll ever really be able to match. 541 00:30:51,706 --> 00:30:55,086 Maybe get really close to, but not quite exactly match, just because the sheer 542 00:30:55,086 --> 00:30:58,116 amount of power that you can bring from a data center or collective data centers. 543 00:30:58,726 --> 00:31:05,376 however, local AI, on device AI really provides, I think more comfort 544 00:31:05,556 --> 00:31:09,686 than anything, like this sense of control, this sense of ownership. 545 00:31:10,353 --> 00:31:15,846 and when costs are such a variable thing, especially with AI, because It is still 546 00:31:15,856 --> 00:31:18,616 somewhat of an un-unknown factor, right? 547 00:31:18,616 --> 00:31:22,306 I mean, especially as, models change in price or we, end up like 548 00:31:22,306 --> 00:31:24,736 as a new model comes out that's expensive, but we get cheaper models. 549 00:31:24,746 --> 00:31:26,196 Like there's, it's just a lot of variability. 550 00:31:26,196 --> 00:31:32,506 But if I can control, hey, I know for a fact that my hardware is okay and that 551 00:31:32,586 --> 00:31:35,836 maybe I do have to buy some kind of model that's offline, I can control that. 552 00:31:35,836 --> 00:31:37,576 That becomes a very, fixed cost. 553 00:31:37,636 --> 00:31:40,726 and I think that is very appetizing for a lot of people. 554 00:31:41,076 --> 00:31:47,576 so I think even if the cloud does get down to pennies, the sheer comfort of having 555 00:31:47,576 --> 00:31:52,996 control over the local piece will be local AI will continue to be such a big thing 556 00:31:54,431 --> 00:32:00,221 Frankcx: I wanted to pose a question to Laura on, in the UK and your region 557 00:32:00,841 --> 00:32:05,381 have you started to notice that it's easier to talk to customers around local 558 00:32:05,381 --> 00:32:07,731 AI and open-weight models and such? 559 00:32:08,251 --> 00:32:12,361 but more on the idea that hey there was a first wave of AI that came out 560 00:32:12,431 --> 00:32:14,381 that, you know, you only had the cloud. 561 00:32:14,871 --> 00:32:19,341 But people were afraid to push their IP and their sovereignty, into US 562 00:32:19,341 --> 00:32:23,791 data centers or into sharing, their intellectual property And now there's 563 00:32:23,791 --> 00:32:28,241 this second wave of these open weight models that's allowing it to be much 564 00:32:28,241 --> 00:32:32,511 more acceptable and be able to be like "Now I can actually my company take 565 00:32:32,511 --> 00:32:37,171 part in this AI experience." i don't know if you've seen that but I guess 566 00:32:37,181 --> 00:32:38,311 I'd be interested in your opinion 567 00:32:40,212 --> 00:32:41,252 Laura: Yeah, for sure. 568 00:32:41,302 --> 00:32:45,022 it's really funny for me actually coming from spending the last 569 00:32:45,032 --> 00:32:47,282 five years as a cloud architect. 570 00:32:47,342 --> 00:32:50,632 I've been telling everyone to throw everything into the cloud, push it all 571 00:32:50,632 --> 00:32:52,172 up there, everything's better up there. 572 00:32:52,522 --> 00:32:56,842 And all of those reasons that you just listed, like, being able to, 573 00:32:56,862 --> 00:33:01,292 you know, have control, everything being kind of centralized, it's… 574

00:33:01,452 --> 00:33:05,112 Those are all the pushes and it was things like move from, I can never get this 575 00:33:05,112 --> 00:33:09,432 in the right order, CapEx to OpEx, OpEx to CapEx, and it just feels like I've 576 00:33:09,432 --> 00:33:13,642 literally done a 360 and I'm like, "Nope, bring everything back down. Everything's 577 00:33:13,682 --> 00:33:18,552 better on the device." but it really goes in line with what you just said, Shaunsy, 578 00:33:18,582 --> 00:33:26,112 in terms of we've been so much kind of almost hate that AI gets, and so much of 579 00:33:26,122 --> 00:33:31,322 that hate is tied to the idea of cloud and tied to the negative aspects of cloud. 580 00:33:31,322 --> 00:33:36,632 I think like really great PR for AI to be able to run locally and say, "Actually, 581 00:33:36,632 --> 00:33:41,802 you can get around all of those problems," but also the security side of it is such 582 00:33:41,802 --> 00:33:47,772 a… Particularly in the UK and the kind of Europe with the sovereignty issues 583 00:33:47,772 --> 00:33:52,872 that we have, it's a huge driver and I think it just answers so many questions. 584 00:33:52,872 --> 00:33:56,672 You don't have to work through all of these security challenges. 585 00:33:57,052 --> 00:34:00,772 You don't have to think about cloud space as well, which 586 00:34:00,772 --> 00:34:02,392 in Europe is a huge problem. 587 00:34:02,392 --> 00:34:05,242 Capacity is just at its limits. 588 00:34:05,672 --> 00:34:06,742 and yes, it's funny. 589 00:34:06,752 --> 00:34:11,452 It feels really ironic for me that, yeah, I'm just slating cloud again now. 590 00:34:11,452 --> 00:34:16,792 I promise I'm not actually doing that, but yeah, it's definitely an interesting move 591 00:34:18,382 --> 00:34:22,552 Neil: even when cloud is reduced to pennies, I'll use the insurance example. 592 00:34:22,812 --> 00:34:26,782 If an insurance claim agent is filing 50 claims a day and we've 593 00:34:26,782 --> 00:34:30,382 got 10,000 insurance agents doing this, those pennies add up, right? 594 00:34:30,382 --> 00:34:32,702 That's over a million dollars over the course of a year. 595 00:34:32,702 --> 00:34:37,632 So even these, lower cost workloads, there's still economic benefit when it's 596 00:34:37,632 --> 00:34:39,412 a repetitive task running it at the edge. 597 00:34:39,902 --> 00:34:44,222 To Laura's point, there are values beyond just cost savings. 598 00:34:44,222 --> 00:34:48,452 In fact, we've identified eight forces as to why a workload should run locally. 599 00:34:48,642 --> 00:34:52,522 there's physics, there's economics, there's regulatory reasons as to 600 00:34:52,522 --> 00:34:55,432 why local is the preferred path. 601 00:34:55,892 --> 00:34:58,752 The public sentiment here, Frank shared an awesome example of his 602 00:34:58,752 --> 00:35:00,402 conversation with an Uber driver. 603 00:35:00,582 --> 00:35:04,762 I'll just share a brief anecdote of, an encounter

I had at a brewery recently. 604 00:35:05,062 --> 00:35:10,432 I went up to order a beverage, and there was a lively conversation around, 605 00:35:10,522 --> 00:35:14,802 really this distaste for AI, and the bartender turns to me and says, 606 00:35:15,122 --> 00:35:20,442 "You're anti-AI, aren't you, bro?" And to which I hesitated, and I said, 607 00:35:20,452 --> 00:35:22,712 "Actually, I'm an AI solutions architect. 608 00:35:22,892 --> 00:35:27,852 However, I'm focused on getting these workloads running locally which offers 609 00:35:27,852 --> 00:35:32,472 cost advantage, lower energy draw." And at the end of my 30 second pitch, I still got 610 00:35:32,472 --> 00:35:33,562 served a beverage. He said, "All right. 611 00:35:33,562 --> 00:35:34,362 you're okay with me." 612 00:35:34,532 --> 00:35:35,252 Laura: You are bad. 613 00:35:35,747 --> 00:35:38,097 Neil: all that to say, I think this public sen-… 614 00:35:38,150 --> 00:35:38,420 Frankcx: data 615 00:35:38,720 --> 00:35:39,867 Neil: I, I didn't say data center. 616 00:35:39,867 --> 00:35:40,507 Exactly. 617 00:35:40,527 --> 00:35:41,197 Exactly. 618 00:35:41,197 --> 00:35:46,007 So although the emissions 25% increase on, emissions year over year for Microsoft, 619 00:35:46,227 --> 00:35:49,707 certainly, you know, there's some merit in those environmental concerns and, 620 00:35:49,907 --> 00:35:54,377 I, I, I believe Local AI offers a, a path, an exciting path forward to the 621 00:35:54,377 --> 00:35:58,717 future where we can deliver intelligence back to Zuck's manifesto, right? 622 00:35:58,957 --> 00:36:03,597 Intelligence for every person across the globe, without creating 623 00:36:03,617 --> 00:36:04,637 unnecessary emissions in the process 624 00:36:06,580 --> 00:36:09,120 Laura: Yeah, there's, a data center being built in the town that I 625 00:36:09,120 --> 00:36:13,780 live in, and there's been uproar, there's been riots with people from 626 00:36:13,780 --> 00:36:18,630 an environmental standpoint saying, you know, AI is ruining everything. 627 00:36:18,640 --> 00:36:24,090 And I feel like the Homer Simpson meme just like shrinking back into the bushes. 628 00:36:24,100 --> 00:36:28,130 But hope that we can be on the right side of history in that sense 629 00:36:28,160 --> 00:36:32,250 that some saving to be made there from a data center perspective 630 00:36:34,760 --> 00:36:37,460 Frankcx: Well then maybe that's a future show that we have coming up. 631 00:36:37,570 --> 00:36:40,010 But let's get to the motion, the vote. 632 00:36:40,390 --> 00:36:45,900 I had pitched out this idea that as the $25 token dies and that's when 633 00:36:45,960 --> 00:36:48,750 enterprise AI hardware boom begins. 634 00:36:49,220 --> 00:36:52,670 So as things become cheaper does that mean we start to use more of it? 635 00:36:52,730 --> 00:36:57,010 And does that open up the gates for AI hardware to really start to flow 636 00:36:57,010 --> 00:36:58,820 into

our commercial industries? 637 00:36:58,850 --> 00:37:00,760 So let's go around the horn. 638 00:37:00,930 --> 00:37:02,420 I will start with Laura. 639 00:37:02,760 --> 00:37:03,680 what's your take on that? 640 00:37:06,308 --> 00:37:07,278 Laura: All for it, yes. 641 00:37:07,378 --> 00:37:10,008 I mean the cost saving is a huge driver. 642 00:37:10,008 --> 00:37:15,618 This idea of tokenomics is becoming part of my everyday vocabulary now 643 00:37:15,618 --> 00:37:20,648 and I think… I was always a bit of a cynic that I never expected AI 644 00:37:20,658 --> 00:37:24,998 to stay as cheap as it was or the kind of unlimited usage that we had. 645 00:37:25,418 --> 00:37:29,158 and in some ways I kind of fear about the same trend happening from a local 646 00:37:29,168 --> 00:37:33,538 perspective but I definitely think that it's the, right push that we've 647 00:37:33,538 --> 00:37:36,928 had for customers to just be more intentional with how they look at 648 00:37:36,928 --> 00:37:39,978 where they're using AI and what they're doing with it and bringing devices 649 00:37:39,978 --> 00:37:41,758 into that fold and into that strategy 650 00:37:44,686 --> 00:37:45,166 Frankcx: Awesome. 651 00:37:45,166 --> 00:37:46,056 I love the vote. 652 00:37:46,106 --> 00:37:49,866 Chauncey, what's your take on this things get cheaper, we start to use more of it? 653 00:37:51,311 --> 00:37:52,051 Chauncey: Yeah, I think so. 654 00:37:52,171 --> 00:37:55,251 I hope, the economies of scale kind of help back this movement. 655 00:37:55,481 --> 00:37:56,951 Like, we kinda need it to, let's say it that way. 656 00:37:59,071 --> 00:37:59,471 Yes. 657 00:38:01,344 --> 00:38:01,744 Frankcx: All right. 658 00:38:01,804 --> 00:38:03,864 And then Neil, your last one 659 00:38:05,403 --> 00:38:07,733 Neil: This is gonna be an anonymous vote, for this group today. 660 00:38:07,783 --> 00:38:10,593 If we draw parallels to the '90s internet boom, it was all about 661 00:38:10,593 --> 00:38:13,093 the servers, and then it was quickly followed by the PC boom. 662 00:38:13,603 --> 00:38:16,553 Draw parallels now to the AI moment, and it was all about 663 00:38:16,553 --> 00:38:18,443 the data centers, the GPU boom. 664 00:38:18,793 --> 00:38:22,763 Now it's the AI endpoint boom and, we have a front row seat to this. 665 00:38:22,773 --> 00:38:24,223 So, exciting times ahead. 666 00:38:24,658 --> 00:38:31,773 edge AI devices are here to stay, and as hardware continues to increase in 667 00:38:31,773 --> 00:38:37,033 capabilities, software, is quantized or we're even seeing specific models 668 00:38:37,033 --> 00:38:38,533 developed for specific hardware. 669 00:38:38,743 --> 00:38:40,913 this AI endpoint boom is here to stay 670 00:38:42,484 --> 00:38:46,774 Frankcx: And I would agree, so-- I'll go right into saying yes, I agree with you, 671 00:38:47,124 --> 00:38:50,884 so we're four for four but that leads us into kind of like what's on our

device 672 00:38:50,884 --> 00:38:52,044 and we'll just finish up with that. 673 00:38:52,054 --> 00:38:56,784 But I did take that thirty-six billion parameter model, and ran it on the 674 00:38:56,784 --> 00:38:59,344 Beast which is my dual 3090 device. 675 00:38:59,444 --> 00:39:02,824 and I said: "Hey, w-what job do you get?" Like as in we did that 676 00:39:02,824 --> 00:39:04,444 job interview piece in the past. 677 00:39:04,444 --> 00:39:05,724 And did it get the job? 678 00:39:05,744 --> 00:39:06,994 Actually it does get the job. 679 00:39:07,014 --> 00:39:10,714 it went up against, you know Qwen three dot eight twenty-seven billion and 680 00:39:10,724 --> 00:39:16,394 almost the exact same qualifying profile as the test So really surprising how 681 00:39:16,444 --> 00:39:18,444 equal the playing field is these days. 682 00:39:18,454 --> 00:39:22,474 This is a brand new model specifically not quantized but ran head to head 683 00:39:22,474 --> 00:39:26,254 against a Qwen three dot eight billion… twenty-seven billion perimeter and 684 00:39:26,254 --> 00:39:29,244 it really came out and showed that it could perform head to head. 685 00:39:29,944 --> 00:39:33,664 It was giving out about sixty-eight tokens per second versus forty on a 686 00:39:33,664 --> 00:39:37,754 standard Qwen three dot eight model.Um, and if I'm hiring it as a local 687 00:39:37,754 --> 00:39:39,654 agent, it's a calling tools agent. 688 00:39:39,864 --> 00:39:43,014 it's great for working through long prompts and big jup- 689 00:39:43,054 --> 00:39:47,564 continuous jobs and honestly K two made a very strong case for it. 690 00:39:47,574 --> 00:39:50,344 But there's a little bit of a catch.Qwen three dot eight 691 00:39:50,344 --> 00:39:54,124 fits onmy single 3090 card. 692 00:39:54,354 --> 00:39:58,234 The K2 needed both of my 3090 cards so obviously there's a little bit more 693 00:39:58,234 --> 00:40:01,684 expense towards how thirty-six billion parameters are spread across forty-eight 694 00:40:01,734 --> 00:40:06,300 gigs of beverypowerfulintothatspace.So anybodyelse outthereplayingon 695 00:40:06,300 --> 00:40:06,320 theirdeviceand testingthings 696 00:40:07,279 --> 00:40:11,699 Chauncey: I-I am on the, training wheels while you guys are all flying spaceships. 697 00:40:11,749 --> 00:40:14,799 and so I was just kind of messing around with, Copilot CLI actually 698 00:40:14,799 --> 00:40:18,239 running locally, so I was trying to get it to run and trying to find the 699 00:40:18,239 --> 00:40:19,979 right model for it to run locally on. 700 00:40:20,089 --> 00:40:22,609 and I used Qwen actually in this case, Qwen code. 701 00:40:22,729 --> 00:40:25,709 Frankcx: man, it, is-- it was a shocker that while it was doing this, I literally 702 00:40:25,709 --> 00:40:28,649 asked it a question like, "Hey, like why are you taking so long?" It's like 703 00:40:28,649 --> 00:40:32,089 well, it's because my parameters are, pretty limited and

I can only do so much 704 00:40:32,189 --> 00:40:35,979 Like you should basically expect about a two hour response time, for this. 705 00:40:35,979 --> 00:40:39,629 Laura: And so it was a pretty major eye opening moment for me of like, ah okay. 706 00:40:39,629 --> 00:40:43,889 That's, that's where I think being able to really help our customers 707 00:40:43,919 --> 00:40:47,889 and, and anyone really just start to think about like what am I really 708 00:40:47,899 --> 00:40:49,929 trying to do with this specific model? 709 00:40:49,969 --> 00:40:54,539 Because maybe I'm actually not applying the right thing or the wri-right workload. 710 00:40:54,829 --> 00:40:58,759 Now, I will say once I was able to, limit the amount of, response and 711 00:40:58,759 --> 00:41:02,389 kind of massage it a little bit, I am actually using it for coding right now. 712 00:41:02,419 --> 00:41:05,459 it, it is helping at least do some of the basic planning and some of 713 00:41:05,469 --> 00:41:08,839 the basic stuff, as I'm, I'm kinda tailoring this other app I'm building. 714 00:41:09,139 --> 00:41:10,719 So it is capable. 715 00:41:10,989 --> 00:41:12,099 is it as fast? 716 00:41:12,239 --> 00:41:13,309 Definitely not. 717 00:41:13,339 --> 00:41:17,679 Like I, you know, will type a, a rude question in there and then I'll wait for 718 00:41:17,749 --> 00:41:20,989 20 minutes for them to give me another response but I think we kinda talked 719 00:41:20,989 --> 00:41:24,259 about this on an earlier call which is if I wanted to do that overnight, maybe 720 00:41:24,259 --> 00:41:28,379 I just wanted to compile something over night, like why not do that for free? 721 00:41:28,389 --> 00:41:32,249 Frankcx: One of the benefits there is I'm not costing anyone anything 722 00:41:32,249 --> 00:41:33,919 right now, to be able to run this. 723 00:41:33,999 --> 00:41:34,859 So it's huge 724 00:41:34,896 --> 00:41:37,106 Frankcx (2): say, " Give me your answer right now?" You know? 725 00:41:37,421 --> 00:41:37,901 Frankcx: Yeah. 726 00:41:37,931 --> 00:41:38,451 Yeah. 727 00:41:39,206 --> 00:41:39,346 Neil: You're 728 00:41:39,461 --> 00:41:40,491 Frankcx: mean-- And it just becomes 729 00:41:40,806 --> 00:41:41,526 Neil: Take your time." 730 00:41:41,826 --> 00:41:43,941 Frankcx: very, focused on-on kind of driving the right 731 00:41:43,941 --> 00:41:44,851 person for the right models. 732 00:41:44,991 --> 00:41:48,811 but it… yeah, no, I would maybe, rephrase that to be like, 733 00:41:48,811 --> 00:41:52,091 are you seeing anything that's, interesting that you've seen that's 734 00:41:52,091 --> 00:41:53,881 been built for on-device as well? 735 00:41:56,397 --> 00:42:00,097 Laura: I did have a cool conversation with someone today actually who is, 736 00:42:01,197 --> 00:42:07,437 building their own kind of edition of the Microsoft Speaker Coach That they are 737 00:42:07,517 --> 00:42:11,927 creating something that's almost like a teleprompter that is wrapped in within 738 00:42:11,927 --> 00:42:16,417 this, that would take your speaker notes, it would have a look at the slides that 739 00:42:16,417 --> 00:42:20,997 you've got, and it would notice once you've started to list off some of those 740 00:42:21,027 --> 00:42:24,807 topics that you have in your speaker notes, it would make them disappear off 741 00:42:24,807 --> 00:42:28,487 the screen and kind of leave you with anything that you've missed out, anything 742 00:42:28,487 --> 00:42:32,767 to kinda prompt you to, fill the gaps in your talk track, whatever that is. 743 00:42:32,817 --> 00:42:36,037 which I thought was a really interesting usage, and that's obviously entirely 744 00:42:36,447 --> 00:42:40,807 on device, but very much a kind of low spec, device requirement as well. 745 00:42:40,807 --> 00:42:43,467 It's very, very simple small language models that that would run on. 746 00:42:43,467 --> 00:42:45,727 So that was something that got me excited this week 747 00:42:46,735 --> 00:42:49,375 Frankcx (2): I feel like I might need that for the podcast as we're talking 748 00:42:49,425 --> 00:42:49,575 Laura: was 749 00:42:49,668 --> 00:42:50,718 Frankcx (2): follow a script 750 00:42:50,875 --> 00:42:51,785 Laura: are you gonna start building it?" 751 00:42:53,090 --> 00:42:53,390 Frankcx (2): All right. 752 00:42:54,130 --> 00:42:54,860 We'll bring it on. 753 00:42:56,260 --> 00:42:57,970 All right, and Neil, anything you're doing? 754 00:42:58,843 --> 00:43:02,543 Neil (2): I believe there's, the right local AI hardware with the 755 00:43:02,543 --> 00:43:04,573 right local model for every user 756 00:43:07,912 --> 00:43:08,642 Frankcx (2): All right. 757 00:43:08,892 --> 00:43:10,772 Well, thank you all. 758 00:43:10,802 --> 00:43:14,212 A big thank you to Laura for joining us on this Friday night 759 00:43:14,212 --> 00:43:16,032 of yours in the United Kingdom. 760 00:43:16,429 --> 00:43:16,669 Laura: Thank 761 00:43:16,702 --> 00:43:17,322 Frankcx (2): So, 762 00:43:17,389 --> 00:43:17,719 Laura: me 763 00:43:18,262 --> 00:43:21,752 Frankcx (2): You are open… The door is open at any time for 764 00:43:21,752 --> 00:43:23,202 you to come back and join us. 765 00:43:23,542 --> 00:43:27,332 I'm gonna close up by saying, you know, with the $25 token that we're 766 00:43:27,332 --> 00:43:32,902 claiming is dying, know, this AI boom around hardware may just be starting. 767 00:43:32,952 --> 00:43:36,782 we might be seeing that Apple is selling a lot of Macs for various 768 00:43:36,782 --> 00:43:40,282 reasons, but Bean's started to be adopted into the enterprise, and we 769 00:43:40,282 --> 00:43:43,402 might be able to say that most of this still belongs in the cloud, but I 770 00:43:43,402 --> 00:43:45,852 think the direction is worth watching. 771 00:43:46,172 --> 00:43:48,412 cheap AI doesn't kill local AI. 772 00:43:48,722 --> 00:43:52,742 cheap AI is what finally makes local AI into a category, and 773 00:43:52,742 --> 00:43:54,082 that's the point of the show today. 774 00:43:54,942 --> 00:43:59,272 if this has been worth your time, a rating out on Apple Podcasts or 775 00:43:59,272 --> 00:44:02,932 Spotify has generally helped us get this show found, and you can find 776 00:44:02,962 --> 00:44:05,872 everything at thelocalhost.show. 777 00:44:06,812 --> 00:44:11,002 for Laura and Neil and Chauncey and Robert and, Jacob, who are 778 00:44:11,032 --> 00:44:18,632 off enjoying themselves today, there is no place like 127.0.0.1. 779 00:44:18,632 --> 00:44:20,182 So have a great weekend, all, 780 00:44:20,547 --> 00:44:21,797 Laura: Have a great extended weekend all. 781 00:44:22,167 --> 00:44:22,647 Thanks all 782 00:44:22,719 --> 00:44:23,329 Neil (2): Happy Labor Day. 783 00:44:23,329 --> 00:44:23,909 Thanks everyone