Fullstendig transkripsjon av ReadMultiplex.com: The Terminal Dies: Flapping Airplanes Returns the Silicon to the Metal and Leaves OpenAI and Anthropic as the New IBM

00:03:48.370 Yeah. The classic beige box. The source text 00:03:48.370 --> 00:03:50.509 gives us a direct account from the author, who 00:03:50.509 --> 00:03:53.409 describes building a program called QFresh in 00:03:53.409 --> 00:03:56.689 a garage in 1987. Okay. And this was running 00:03:56.689 --> 00:04:00.169 on an Intel 8088 processor inside an IBM PC. 00:04:00.669 --> 00:04:03.889 But to understand the parallel to today's AI 00:04:03.889 --> 00:04:06.289 accelerators, we have to get into the physical 00:04:06.289 --> 00:04:09.330 weeds of how that 1987 machine actually worked. 00:04:09.469 --> 00:04:12.569 Like at the hardware level. Exactly. Specifically, 00:04:12.729 --> 00:04:16.629 how its memory worked. The IBM PC used DRAM, 00:04:16.670 --> 00:04:19.670 Dynamic Random Access Memory. Okay, so dynamic 00:04:19.670 --> 00:04:22.529 meaning it's constantly changing, or in this 00:04:22.529 --> 00:04:24.850 case, constantly forgetting things if you don't 00:04:24.850 --> 00:04:28.209 actively remind it. That is actually a very literal 00:04:28.209 --> 00:04:30.769 way to describe it. Oh, really? Yeah. Physically, 00:04:30.850 --> 00:04:33.069 DRAM stores each bit of data in a microscopic 00:04:33.069 --> 00:04:35.329 capacitor. You can think of a capacitor like 00:04:35.329 --> 00:04:37.790 a tiny bucket holding a static electric charge. 00:04:37.949 --> 00:04:40.269 Okay, tiny bucket. I can picture that. But these 00:04:40.269 --> 00:04:42.980 buckets are leaky. Hmm. Over a very short period 00:04:42.980 --> 00:04:45.259 of time, the charge just drains away. Right. 00:04:45.379 --> 00:04:47.740 So if the computer doesn't periodically read 00:04:47.740 --> 00:04:50.060 the charge and top it back up, which is a process 00:04:50.060 --> 00:04:52.980 called refreshing, the data just vanishes. So 00:04:52.980 --> 00:04:55.079 the computer has to constantly stop whatever 00:04:55.079 --> 00:04:58.220 it's doing to like water the plants, so to speak. 00:04:58.379 --> 00:05:00.779 That's a great analogy. The architecture required 00:05:00.779 --> 00:05:03.160 a dedicated hardware timer to trigger a direct 00:05:03.160 --> 00:05:07.160 memory access or DMA controller. Okay. IBM engineered 00:05:07.160 --> 00:05:09.279 the system so that this refresh happened on a 00:05:09.279 --> 00:05:12.930 very rigid fixed schedule. Every 15 microseconds, 00:05:12.930 --> 00:05:16.129 the CPU would essentially be put on pause. Wow. 00:05:16.209 --> 00:05:18.829 Every 15 microseconds. Yeah. The DMA controller 00:05:18.829 --> 00:05:21.170 would take over the bus, read a row of memory, 00:05:21.370 --> 00:05:23.750 write it back to refresh the capacitors, and 00:05:23.750 --> 00:05:26.689 then finally give control back to the CPU. So 00:05:26.689 --> 00:05:28.470 it's constantly interrupting the actual work. 00:05:28.689 --> 00:05:31.269 Exactly. And the problem the author noticed was 00:05:31.269 --> 00:05:33.649 that this fixed schedule was incredibly conservative. 00:05:33.990 --> 00:05:37.110 It left a lot of compute cycles completely unused. 00:05:37.490 --> 00:05:40.300 Because it was being overly safe? Right. The 00:05:40.300 --> 00:05:42.220 computer was just sitting there in the dark, 00:05:42.300 --> 00:05:45.459 waiting for the refresh cycle to finish and dictate 00:05:45.459 --> 00:05:47.720 when it could work again. So the author looks 00:05:47.720 --> 00:05:49.959 at this and says, well, that's a waste. But how 00:05:49.959 --> 00:05:52.939 do you actually fix a hardware timer? I mean, 00:05:52.959 --> 00:05:55.319 it's physically wired into the motherboard, isn't 00:05:55.319 --> 00:05:59.420 it? It is. But you bypass it with software that 00:05:59.420 --> 00:06:02.600 operates at the lowest possible level. Ah. The 00:06:02.600 --> 00:06:05.459 author wrote a TSR, a Terminate and Stay Resident 00:06:05.459 --> 00:06:08.699 program. Wait, what's a TSR? In the days of MS 00:06:08.699 --> 00:06:10.980 -DOS, before we had modern multitasking operating 00:06:10.980 --> 00:06:13.620 systems like Windows or Linux. Right, back when 00:06:13.620 --> 00:06:15.560 you could only do one thing at a time. Exactly. 00:06:16.319 --> 00:06:18.680 TSR was a program that loaded into memory and 00:06:18.680 --> 00:06:20.680 just stayed there in the background, hooking 00:06:20.680 --> 00:06:23.920 into hardware interrupts. Okay. What this specific 00:06:23.920 --> 00:06:27.560 TSR did was intercept the timer. It effectively 00:06:27.560 --> 00:06:30.319 took that memory refresh process and, instead 00:06:30.319 --> 00:06:33.100 of running it on IBM's rigid schedule, moved 00:06:33.100 --> 00:06:35.579 it directly into the natural gaps in the CPU's 00:06:35.579 --> 00:06:38.889 workload. Oh, wow. So it broke the fixed schedule 00:06:38.889 --> 00:06:42.209 entirely. Yes. It hid the refresh inside the 00:06:42.209 --> 00:06:45.850 idle moments. So it's like, okay, it's like renting 00:06:45.850 --> 00:06:48.029 an apartment but realizing there's a whole secret 00:06:48.029 --> 00:06:50.310 room you aren't using just because the landlord's 00:06:50.310 --> 00:06:52.610 floor plan didn't draw a door. That's exactly 00:06:52.610 --> 00:06:54.769 it. You're paying for the whole square footage. 00:06:54.990 --> 00:06:57.189 The electricity is already running to that room, 00:06:57.230 --> 00:06:58.910 but you're only living in half of it because 00:06:58.910 --> 00:07:01.709 the schematic told you to. So you just decide 00:07:01.709 --> 00:07:03.589 to knock a hole in the drywall. And the results 00:07:03.589 --> 00:07:05.629 of knocking that hole in the drywall were staggering. 00:07:06.300 --> 00:07:08.800 How big of a difference did it make? On the workloads 00:07:08.800 --> 00:07:11.180 the author measured, this tiny piece of background 00:07:11.180 --> 00:07:15.040 code delivered a massive 50 % performance gain. 00:07:15.480 --> 00:07:18.439 50 %? Just from moving the refresh schedule? 00:07:18.879 -->

00:07:21.360 Let that sink in. They didn't upgrade the processor. 00:07:21.720 --> 00:07:25.279 They didn't add more RAM. They got 50 % more 00:07:25.279 --> 00:07:27.519 performance out of the exact same physical machine. 00:07:27.899 --> 00:07:30.519 Just by reclaiming time that the hardware was 00:07:30.519 --> 00:07:33.160 already experiencing, but the software was ignoring. 00:07:33.439 --> 00:07:36.449 Exactly. But wait, I have to push back a little 00:07:36.449 --> 00:07:38.670 here because I don't understand IBM's motivation. 00:07:39.290 --> 00:07:42.329 I mean, IBM was the undisputed king of computing 00:07:42.329 --> 00:07:44.870 in the 1980s. They absolutely were. They had 00:07:44.870 --> 00:07:47.050 the best engineers in the world. Why would they 00:07:47.050 --> 00:07:49.769 design a system that threw away 50 % of its potential? 00:07:49.930 --> 00:07:51.689 Did they just not care about the sufficiency 00:07:51.689 --> 00:07:54.290 back then? Well, it's not that they didn't care. 00:07:54.790 --> 00:07:57.509 The author notes that IBM preferred the standard 00:07:57.509 --> 00:08:00.310 timing. But is this the exact same mistake the 00:08:00.310 --> 00:08:02.949 big AI labs are making today? Are they preferring 00:08:02.949 --> 00:08:06.009 standard timing over actual efficiency? Yes, 00:08:06.129 --> 00:08:08.589 that's the core argument. See, IBM preferred 00:08:08.589 --> 00:08:11.769 predictability over absolute optimization. Okay, 00:08:11.910 --> 00:08:14.009 that makes sense for a massive company. Right. 00:08:14.110 --> 00:08:16.529 When you are building hundreds of thousands of 00:08:16.529 --> 00:08:19.230 consumer machines, you have to account for variances 00:08:19.230 --> 00:08:22.230 in manufacturing, temperature, component degradation. 00:08:22.689 --> 00:08:25.370 Sure. A fixed conservative schedule guarantees 00:08:25.370 --> 00:08:28.370 that no matter how hot the room gets or how cheap 00:08:28.370 --> 00:08:30.990 the DRAM chips are, the memory won't corrupt. 00:08:31.670 --> 00:08:35.960 So it's foolproof. IBM accepted the 50 % performance 00:08:35.960 --> 00:08:39.039 penalty as the cost of absolute stability and 00:08:39.039 --> 00:08:43.120 simplicity in their hardware design. Wow. And 00:08:43.120 --> 00:08:45.580 the argument the source text makes is that this 00:08:45.580 --> 00:08:48.120 is the exact psychological and structural trap 00:08:48.120 --> 00:08:51.360 that OpenAI and Anthropic are falling into right 00:08:51.360 --> 00:08:55.220 now. That is wild. So we are taking the exact 00:08:55.220 --> 00:08:57.940 same philosophical approach from a 1987 garage 00:08:57.940 --> 00:09:00.460 and applying it to multi -thousand dollar NVIDIA 00:09:00.460 --> 00:09:03.980 AI chips in 2026. The foundational premise of 00:09:03.980 --> 00:09:06.440 flapping airplanes is identical to that 1987 00:09:06.440 --> 00:09:09.340 QFOS hack. Just on a much bigger scale. Right. 00:09:09.419 --> 00:09:11.799 The modern

GPU, the graphics processing unit, 00:09:11.940 --> 00:09:14.299 which is the engine driving the entire generative 00:09:14.299 --> 00:09:17.320 AI boom, already contains massive amounts of 00:09:17.320 --> 00:09:20.299 idle time. Just like that old IBM PC. Exactly. 00:09:20.379 --> 00:09:23.460 And that idle time can be recovered if we simply 00:09:23.460 --> 00:09:25.980 stop treating the hardware vendor schedule or 00:09:25.980 --> 00:09:28.419 the standard software stack as the final unbreakable 00:09:28.419 --> 00:09:31.379 law. Because the silicon already holds that capacity. 00:09:31.740 --> 00:09:34.090 Yes. You just have to remove the software boundary 00:09:34.090 --> 00:09:36.750 that is walling it off. Okay, so we have established 00:09:36.750 --> 00:09:39.049 this philosophy of finding hidden space in the 00:09:39.049 --> 00:09:41.370 machine. Now, let's look at the actual blueprint 00:09:41.370 --> 00:09:43.649 that proved this works on modern AI hardware. 00:09:43.909 --> 00:09:46.409 Right, because they need proof. Because I'm assuming 00:09:46.409 --> 00:09:48.669 flapping airplanes didn't just walk into a Sand 00:09:48.669 --> 00:09:50.830 Hill Road venture capital office, tell a cute 00:09:50.830 --> 00:09:53.610 story about a 1987 DOS hack, and walk out with 00:09:53.610 --> 00:09:55.720 a billion dollars. Definitely not. They have 00:09:55.720 --> 00:09:58.279 prior technical records. So we need to talk about 00:09:58.279 --> 00:10:01.919 something called the launch tax and an incredibly 00:10:01.919 --> 00:10:04.279 named piece of software called Thunder Kittens. 00:10:04.440 --> 00:10:06.580 Yeah, Thunder Kittens is a great name. But to 00:10:06.580 --> 00:10:08.559 appreciate what Thunder Kittens achieved, we 00:10:08.559 --> 00:10:10.639 first have to map out the invisible friction 00:10:10.639 --> 00:10:14.159 slowing down modern AI. Okay. And that friction 00:10:14.159 --> 00:10:18.629 is the kernel launch tax. The launch tax. Right. 00:10:19.389 --> 00:10:22.690 When an AI model, say a massive large language 00:10:22.690 --> 00:10:26.570 model, runs on a GPU, It isn't executing as one 00:10:26.570 --> 00:10:30.190 monolithic piece of code. It executes as thousands 00:10:30.190 --> 00:10:34.330 of separate tiny programs called kernels. Let's 00:10:34.330 --> 00:10:36.289 pause there for a second because kernel is one 00:10:36.289 --> 00:10:38.090 of those words that means like five different 00:10:38.090 --> 00:10:39.929 things depending on who you talk to in tech. 00:10:40.210 --> 00:10:42.210 Oh, absolutely. We aren't talking about the Linux 00:10:42.210 --> 00:10:45.669 kernel here, right? No, no. In the context of 00:10:45.669 --> 00:10:48.669 GPU programming, particularly with NVIDIA's CUDA 00:10:48.669 --> 00:10:51.769 platform. A kernel is simply a function that 00:10:51.769 --> 00:10:54.570 is compiled to run on the GPU instead of the 00:10:54.570 --> 00:10:57.789 CPU. Okay. Think of it as a highly specialized 00:10:57.789 --> 00:11:01.529 microtask. You have a kernel for matrix multiplication. 00:11:01.970 --> 00:11:03.850 Right. You have a completely separate kernel 00:11:03.850 --> 00:11:06.990 for the attention mechanism. Another kernel for 00:11:06.990 --> 00:11:10.450 layer normalization. And yet another kernel just 00:11:10.450 --> 00:11:12.730 for moving memory from one part of the chip to 00:11:12.730 --> 00:11:16.379 another. So a single inference pass, like just 00:11:16.379 --> 00:11:19.500 generating one word of text, requires the system 00:11:19.500 --> 00:11:22.320 to orchestrate thousands of these microtasks 00:11:22.320 --> 00:11:25.740 in a specific order. Exactly. And the CPU acts 00:11:25.740 --> 00:11:28.700 as the manager. Okay. The CPU tells the GPU, 00:11:28.960 --> 00:11:31.759 okay, run the matrix multiplication kernel. The 00:11:31.759 --> 00:11:34.620 GPU runs it. Then the GPU has to signal back 00:11:34.620 --> 00:11:37.519 to the CPU, I'm done. So there's a back and forth? 00:11:37.759 --> 00:11:40.980 Yes. The CPU says, great, now run the normalization 00:11:40.980 --> 00:11:44.570 kernel. Every single time one of these kernels 00:11:44.570 --> 00:11:46.590 is launched, it carries administrative overhead. 00:11:47.049 --> 00:11:50.690 Like paperwork. Pretty much. The GPU has to allocate 00:11:50.690 --> 00:11:54.049 registers, set up the execution grid, synchronize 00:11:54.049 --> 00:11:56.809 threads, and eventually tear it all down when 00:11:56.809 --> 00:11:59.549 the kernel finishes. Imagine if every time a 00:11:59.549 --> 00:12:01.649 line cook in a restaurant chopped a carrot, they 00:12:01.649 --> 00:12:04.190 had to clock out, clock back in, wash their hands, 00:12:04.309 --> 00:12:06.590 re -read the recipe, and wait for the head chef 00:12:06.590 --> 00:12:09.429 to specifically authorize them to start chopping 00:12:09.429 --> 00:12:11.570 an onion. That is exactly what is happening. 00:12:11.990 --> 00:12:14.169 That's what our most advanced multi -million 00:12:14.169 --> 00:12:16.450 dollar supercomputers are doing right now. It 00:12:16.450 --> 00:12:18.190 sounds absurd when you put it in human terms. 00:12:18.409 --> 00:12:20.990 The chef analogy works perfectly because the 00:12:20.990 --> 00:12:23.889 chef's actual chopping time, like the pure mathematical 00:12:23.889 --> 00:12:27.070 compute, is incredibly fast. Right. GPUs are 00:12:27.070 --> 00:12:30.269 fast. The GPU is an absolute monster at math. 00:12:30.570 --> 00:12:32.990 But the administrative overhead of clocking in 00:12:32.990 --> 00:12:36.009 and out is destroying their throughput. So what 00:12:36.009 --> 00:12:38.639 happens during that downtime? That's the tragedy 00:12:38.639 --> 00:12:41.340 here. What happens inside the GPU while all this 00:12:41.340 --> 00:12:43.519 kernel launching and tearing down is taking place, 00:12:43.620 --> 00:12:46.820 the actual compute engines, which on modern NVIDIA 00:12:46.820 --> 00:12:49.220

chips are called tensor cores, are just sitting 00:12:49.220 --> 00:12:52.899 there idle. Just doing nothing. Nothing. They're 00:12:52.899 --> 00:12:55.000 waiting for dependencies to resolve. They're 00:12:55.000 --> 00:12:57.200 waiting for the CPU manager to finish its paperwork. 00:12:57.500 --> 00:12:59.980 And these tensor cores are the expensive part 00:12:59.980 --> 00:13:01.659 of the chip, right? Like that's what everyone 00:13:01.659 --> 00:13:04.240 is paying tens of thousands of dollars for. They 00:13:04.240 --> 00:13:07.769 are the beating heart of the AI revolution. A 00:13:07.769 --> 00:13:10.889 standard CUDA core on a GPU is good at general 00:13:10.889 --> 00:13:13.769 -purpose parallel math. Okay. But a tensor core 00:13:13.769 --> 00:13:16.470 is a highly specialized piece of silicon designed 00:13:16.470 --> 00:13:20.470 to do one thing with terrifying efficiency, multiply 00:13:20.470 --> 00:13:24.049 and accumulate matrices. And that's what AI runs 00:13:24.049 --> 00:13:27.820 on. Exactly. To put it in perspective, the source 00:13:27.820 --> 00:13:30.919 text notes that tensor cores supply roughly 16 00:13:30.919 --> 00:13:33.840 times the floating point operations, the flops. 00:13:34.539 --> 00:13:37.240 of the general purpose compute on an NVIDIA H100 00:13:37.240 --> 00:13:42.580 chip. Wait, 16 times? Yes. So the vast, vast 00:13:42.580 --> 00:13:45.279 majority of the actual computational muscle on 00:13:45.279 --> 00:13:47.559 the chip is locked up in these specific cores? 00:13:47.960 --> 00:13:51.600 It is. So if your software stack is designed 00:13:51.600 --> 00:13:54.059 in a way that leaves those tensor cores idle 00:13:54.059 --> 00:13:56.700 while it manages administrative kernel launches 00:13:56.700 --> 00:14:00.539 or waits for memory to slowly trickle in, you 00:14:00.539 --> 00:14:03.120 are literally... forfeiting the vast majority 00:14:03.120 --> 00:14:06.019 of a multi -thousand -dollar machine. Oh, man. 00:14:06.539 --> 00:14:09.299 You're paying for a Ferrari and driving it in 00:14:09.299 --> 00:14:11.840 a school zone because your software forces it 00:14:11.840 --> 00:14:14.980 to stop at every crosswalk. Which brings us to 00:14:14.980 --> 00:14:18.340 Thunder Kittens. Yes. This is the 2024 Stanford 00:14:18.340 --> 00:14:20.840 framework co -authored by Ben Spector, who is 00:14:20.840 --> 00:14:22.480 now one of the founders of flapping airplanes. 00:14:22.879 --> 00:14:25.320 That's him. This was the proof of concept that 00:14:25.320 --> 00:14:28.620 you could bypass the crosswalks. In 2024, out 00:14:28.620 --> 00:14:31.399 of Stanford's hazy research group, Thunder Kittens 00:14:31.399 --> 00:14:34.320 provided the undeniable open -source technical 00:14:34.320 --> 00:14:37.639 record that you could bypass this massive inefficiency. 00:14:37.980 --> 00:14:40.000 How did they do it? Well, they didn't write a 00:14:40.000 --> 00:14:41.940 new model. They wrote a new way of talking to 00:14:41.940 --> 00:14:44.919 the hardware. And it went incredibly granular, 00:14:45.159 --> 00:14:47.799 matching the software directly to the physical 00:14:47.799 --> 00:14:50.120 reality of the chip. What does that mean in practice? 00:14:50.480 --> 00:14:52.840 Like, how do you match software to physical reality? 00:14:53.200 --> 00:14:55.240 It comes down to understanding the physical dimensions 00:14:55.240 --> 00:14:57.879 of the silicon. The Thunder Kittens framework 00:14:57.879 --> 00:15:02.460 defined 16 by 16 matrix tiles. Why 16 by 16? 00:15:02.759 --> 00:15:05.000 Because that perfectly matches the physical hardware 00:15:05.000 --> 00:15:07.980 layout of the Tensor cores on an NVIDIA GPU. 00:15:08.559 --> 00:15:11.519 I see. Instead of writing abstract code that 00:15:11.519 --> 00:15:14.080 the compiler has to awkwardly squish into the 00:15:14.080 --> 00:15:17.100 hardware, they wrote code shaped exactly like 00:15:17.100 --> 00:15:19.500 the hardware. So they basically built a kitchen 00:15:19.500 --> 00:15:21.519 where the cutting boards are exactly the size 00:15:21.519 --> 00:15:24.259 of the chef's knife span, so no movement is wasted. 00:15:24.539 --> 00:15:27.320 Exactly. And they went further. They created 00:15:27.320 --> 00:15:29.500 templates that overlapped asynchronous memory 00:15:29.500 --> 00:15:32.600 loads and stores across warps. Let's slow down 00:15:32.600 --> 00:15:35.740 there. Asynchronous memory loads across warps. 00:15:35.820 --> 00:15:39.340 I know, it's a lot of jargon. I know a warp in 00:15:39.340 --> 00:15:42.299 NVIDIA terminology is a group of 32 threads that 00:15:42.299 --> 00:15:45.679 execute instructions simultaneously, but what 00:15:45.679 --> 00:15:48.779 does it actually look like when they work asynchronously 00:15:48.779 --> 00:15:51.679 versus synchronously? Let's use an analogy. Are 00:15:51.679 --> 00:15:53.600 we talking about a relay race versus a three 00:15:53.600 --> 00:15:56.029 -legged race? The standard synchronous approach 00:15:56.029 --> 00:15:58.070 is like a relay race where the entire team has 00:15:58.070 --> 00:16:01.110 to wait for the baton. Okay. A warp of 32 threads 00:16:01.110 --> 00:16:03.830 needs data from the main memory. It requests 00:16:03.830 --> 00:16:06.909 the data, and then all 32 threads just sit down 00:16:06.909 --> 00:16:09.350 and wait for the data to arrive. Just twiddling 00:16:09.350 --> 00:16:11.690 their thumbs. Once it arrives, they stand up, 00:16:11.730 --> 00:16:13.850 do the math, and then request the next batch 00:16:13.850 --> 00:16:17.529 of data. Sit down, wait, stand up, compute. That 00:16:17.529 --> 00:16:20.830 sounds terribly inefficient. It is. Asynchronous 00:16:20.830 --> 00:16:22.809 memory loading completely changes the paradigm. 00:16:23.330 --> 00:16:26.190 Instead of waiting, you have multiple warps operating 00:16:26.190 --> 00:16:29.210 in a staggered rhythm. How does that work? Warp 00:16:29.210 --> 00:16:32.129

A requests data. While Warp A's data is traveling 00:16:32.129 --> 00:16:34.690 over the memory bus, Warp B is already computing 00:16:34.690 --> 00:16:37.169 on data it received a microsecond earlier. Oh, 00:16:37.289 --> 00:16:39.870 so someone is always working. Right. When Warp 00:16:39.870 --> 00:16:42.370 B finishes computing, it requests its next batch 00:16:42.370 --> 00:16:44.870 of data. And right at that exact moment, Warp 00:16:44.870 --> 00:16:47.509 A's data arrives and it begins computing. So 00:16:47.509 --> 00:16:49.769 there's no downtime. You are hiding the latency 00:16:49.769 --> 00:16:52.169 of memory transfers behind the computation of 00:16:52.169 --> 00:16:55.450 other threads. The tensor cores never stop firing. 00:16:55.649 --> 00:16:58.570 The chef never clocks out. The ingredients are 00:16:58.570 --> 00:17:00.769 just asynchronously tossed to them while they 00:17:00.769 --> 00:17:03.289 keep chopping. Perfectly said. And the numbers 00:17:03.289 --> 00:17:05.650 from the source text backing this up are absolutely 00:17:05.650 --> 00:17:07.869 mind -blowing. They really are. The Thunder Kittens 00:17:07.869 --> 00:17:10.410 kernels match the absolute industry gold standards, 00:17:10.710 --> 00:17:14.509 NVIDIA's own kublas, and the famous Flash Attention 00:17:14.509 --> 00:17:17.109 3 on standard forward passes. Which is already 00:17:17.109 --> 00:17:19.380 impressive. But then on the attention backward 00:17:19.380 --> 00:17:22.140 passes, which are notoriously difficult, they 00:17:22.140 --> 00:17:24.539 outperformed the strongest baselines by 10 to 00:17:24.539 --> 00:17:27.759 40 percent. Huge gains. They were eight times 00:17:27.759 --> 00:17:30.440 faster on state space models and an incredible 00:17:30.440 --> 00:17:34.480 14 times faster on linear attention. 14 times. 00:17:34.720 --> 00:17:38.099 We are talking about a tiny 40 line kernel written 00:17:38.099 --> 00:17:41.279 by academics competing with and beating massive 00:17:41.279 --> 00:17:44.079 corporate code bases maintained by armies of 00:17:44.079 --> 00:17:46.000 engineers at NVIDIA and Google. And the most 00:17:46.000 --> 00:17:48.420 vital takeaway here is that. They didn't invent 00:17:48.420 --> 00:17:51.180 new hardware. Right, no secret chips. They didn't 00:17:51.180 --> 00:17:53.339 fabricate a magical new chip in a secret laboratory. 00:17:53.980 --> 00:17:56.700 They used the exact same off -the -shelf GPUs 00:17:56.700 --> 00:17:58.920 everyone else had. That's the crazy part. They 00:17:58.920 --> 00:18:01.559 just organized the work to perfectly fit the 00:18:01.559 --> 00:18:03.559 physical reality of the silicon that was already 00:18:03.559 --> 00:18:06.900 there. They measured the idle intervals, just 00:18:06.900 --> 00:18:09.079 like the QFresh author measured the DRAM refresh 00:18:09.079 --> 00:18:11.440 caps. And they removed the software boundaries 00:18:11.440 --> 00:18:13.740 creating those intervals. Okay, here's

where 00:18:13.740 --> 00:18:16.599 it gets really interesting. Thunder Kittens was 00:18:16.599 --> 00:18:19.819 an amazing academic proof of concept. Yes. But 00:18:19.819 --> 00:18:22.539 writing a fast 40 -line kernel for one specific 00:18:22.539 --> 00:18:24.920 operation doesn't build a $6 billion company. 00:18:25.319 --> 00:18:27.619 No, it doesn't. Flapping airplanes took that 00:18:27.619 --> 00:18:30.319 academic success and they scaled it into a total 00:18:30.319 --> 00:18:33.240 paradigm shift. They evolved the concept of the 00:18:33.240 --> 00:18:35.460 optimized kernel into something called the mega 00:18:35.460 --> 00:18:38.660 kernel. This is the architectural leap that justifies 00:18:38.660 --> 00:18:42.769 the valuation. The mega kernel. The mega kernel 00:18:42.769 --> 00:18:45.500 takes the philosophy of Thunder Kittens. You 00:18:45.500 --> 00:18:47.440 know, eliminating administrative overhead and 00:18:47.440 --> 00:18:49.799 matching the hardware. And applies it to the 00:18:49.799 --> 00:18:53.420 entire generative AI system. How is that different 00:18:53.420 --> 00:18:55.819 from what everyone else does? Instead of launching 00:18:55.819 --> 00:18:58.480 thousands of separate kernels for all those different 00:18:58.480 --> 00:19:01.240 tasks we talked about earlier. Yeah. Flapping 00:19:01.240 --> 00:19:03.279 Airplanes has collapsed the entire execution 00:19:03.279 --> 00:19:07.599 schedule inside one single monolithic long -running 00:19:07.599 --> 00:19:10.619 program. A mega kernel. A single kernel launch. 00:19:11.190 --> 00:19:14.069 Because it is one continuous program, the GPU 00:19:14.069 --> 00:19:17.049 can overlap memory -bound work with compute -bound 00:19:17.049 --> 00:19:19.710 work across the entire lifecycle of the model, 00:19:19.809 --> 00:19:21.950 without ever giving control back to the host 00:19:21.950 --> 00:19:24.450 CPU. I want to make sure I'm wrapping my head 00:19:24.450 --> 00:19:27.309 around this. Because normally the CPU is the 00:19:27.309 --> 00:19:29.410 boss giving orders to the GPU for every single 00:19:29.410 --> 00:19:31.930 step. Yes. Do step one, okay now do step two, 00:19:32.109 --> 00:19:34.529 okay now do step three. You're saying the mega 00:19:34.529 --> 00:19:36.650 -kernel cuts the boss completely out of the loop. 00:19:37.190 --> 00:19:40.319 Entirely. The CPU... launches the mega kernel 00:19:40.319 --> 00:19:43.579 once, and then the GPU essentially secedes from 00:19:43.579 --> 00:19:47.400 the host system. It goes rogue. Kind of. The 00:19:47.400 --> 00:19:50.200 GPU manages its own schedule internally. To accomplish 00:19:50.200 --> 00:19:52.740 this, the team built what is described as the 00:19:52.740 --> 00:19:56.740 KVM, the Kitten's Virtual Machine. Kitten's Virtual 00:19:56.740 --> 00:19:59.160 Machine. It sounds so non -threatening for a 00:19:59.160 --> 00:20:01.000 piece of software that is trying to dethrone 00:20:01.000 --> 00:20:02.960 the biggest tech companies on Earth. It does 00:20:02.960 --> 00:20:06.319 sound cute. What actually is it? How do you run 00:20:06.319 --> 00:20:09.890 a virtual machine inside a GPU? Well, in traditional 00:20:09.890 --> 00:20:11.890 computing, a virtual machine lets you run an 00:20:11.890 --> 00:20:14.289 operating system inside another operating system. 00:20:14.410 --> 00:20:16.630 Right, like running Windows inside a Mac. Exactly. 00:20:16.910 --> 00:20:19.869 The KVM is a persistent runtime environment that 00:20:19.869 --> 00:20:22.970 takes over the GPU entirely. It exposes its own 00:20:22.970 --> 00:20:24.950 instruction -based programming model directly 00:20:24.950 --> 00:20:28.730 to the hardware. Okay. But its main trick, the 00:20:28.730 --> 00:20:32.170 real magic, is how it handles memory. The KVM 00:20:32.170 --> 00:20:34.190 can co -pack completely independent workloads 00:20:34.190 --> 00:20:36.950 into the same kernel. Co -packing independent 00:20:36.950 --> 00:20:39.329 workloads, meaning it's doing different tasks 00:20:39.329 --> 00:20:41.569 at the exact same time without separating them. 00:20:41.750 --> 00:20:44.349 Yes. How do you trick the GPU into doing that? 00:20:44.430 --> 00:20:46.630 Don't different tasks overwrite each other's 00:20:46.630 --> 00:20:49.690 memory? In a traditional stack, yes. If you run 00:20:49.690 --> 00:20:52.109 two different kernels, they have separate address 00:20:52.109 --> 00:20:54.109 spaces to prevent them from crashing into each 00:20:54.109 --> 00:20:56.049 other. Right. But setting up those separate spaces 00:20:56.049 --> 00:20:59.589 takes time. The KVM forces the hardware to share 00:20:59.589 --> 00:21:02.470 the exact same address space, the same ultra 00:21:02.470 --> 00:21:04.869 -fast registers, and the same interconnects continuously. 00:21:05.549 --> 00:21:07.849 That sounds dangerous. It operates like a highly 00:21:07.849 --> 00:21:10.509 choreographed dance. Because the KVM is fully 00:21:10.509 --> 00:21:12.670 aware of the physical layout of the chip, down 00:21:12.670 --> 00:21:15.970 to the individual 16x16 matrix tiles, it can 00:21:15.970 --> 00:21:19.849 weave different operations together. Ah, I see. 00:21:20.029 --> 00:21:21.910 While one part of the model is waiting for memory, 00:21:22.089 --> 00:21:24.509 the KVM instantly swaps in a different operation 00:21:24.509 --> 00:21:28.009 to use the idle tensor cores. It maintains sustained 00:21:28.009 --> 00:21:31.569 maximum occupancy. And we have the measured gains 00:21:31.569 --> 00:21:33.269 from the source text to prove this isn't just 00:21:33.269 --> 00:21:35.789 theoretical. The numbers are real. These are 00:21:35.789 --> 00:21:38.390 secondary accounts of their earlier mega kernel 00:21:38.390 --> 00:21:41.150 work, but the numbers are wild. On an NVIDIA 00:21:41.150 --> 00:21:43.990 H100 chip running standard open source frameworks 00:21:43.990 --> 00:21:48.309 like VLLM or SGLang, the memory

bandwidth utilization 00:21:48.309 --> 00:21:52.369 is an abysmal 50%. Half of it. Half of it. It's 00:21:52.369 --> 00:21:54.910 gone. The very thing that usually bottlenecks 00:21:54.910 --> 00:21:57.309 these models is just wasted. But running on the 00:21:57.309 --> 00:22:00.490 mega kernel for a LAMA 3 .2 1 billion parameter 00:22:00.490 --> 00:22:03.269 model doing single sequence generation. That 00:22:03.269 --> 00:22:08.049 utilization shot up from 50 % to 78%. And the 00:22:08.049 --> 00:22:10.430 actual inference speed was more than one and 00:22:10.430 --> 00:22:12.930 a half times faster. And it scales up to much 00:22:12.930 --> 00:22:15.210 larger models, too. Does it? The source mentions 00:22:15.210 --> 00:22:18.190 later work on a massive LAMA70B model using a 00:22:18.190 --> 00:22:21.250 related framework called Tokasaurus. Tokasaurus. 00:22:21.910 --> 00:22:24.150 Another great name. Which showed end -to -end 00:22:24.150 --> 00:22:27.069 throughput more than 22 % higher than the strongest 00:22:27.069 --> 00:22:29.299 baselines in SG Lang. But the most staggering 00:22:29.299 --> 00:22:31.920 detail in this entire document is a post from 00:22:31.920 --> 00:22:34.319 one of the founders. Oh, this part. They stated 00:22:34.319 --> 00:22:36.900 that this framework has already been used for 00:22:36.900 --> 00:22:39.519 trillion parameter scale pre -training inside 00:22:39.519 --> 00:22:43.359 a single mega kernel. That is the line that should 00:22:43.359 --> 00:22:45.900 keep hardware engineers up at night. A trillion 00:22:45.900 --> 00:22:49.460 parameters. Pre -training a frontier class model 00:22:49.460 --> 00:22:53.519 inside one single kernel launch. That completely 00:22:53.519 --> 00:22:55.779 breaks the standard architecture of how AI is 00:22:55.779 --> 00:22:57.880 built today. I mean, normally training a trillion 00:22:57.880 --> 00:23:01.240 parameter model requires a supercomputer choreographing 00:23:01.240 --> 00:23:03.240 millions of kernel launches across thousands 00:23:03.240 --> 00:23:05.740 of GPUs over several months. It's a logistical 00:23:05.740 --> 00:23:08.119 nightmare. To say you can collapse that orchestration 00:23:08.119 --> 00:23:10.619 into a single persistent runtime. I mean, that 00:23:10.619 --> 00:23:12.539 brings us to the giants in the room. The open 00:23:12.539 --> 00:23:14.940 AIs of the world. Right. If this startup, this 00:23:14.940 --> 00:23:17.799 two dozen person team in San Francisco can get 00:23:17.799 --> 00:23:20.380 vastly more power out of a single chip. What 00:23:20.380 --> 00:23:22.619 does that mean for open AI and Anthropic? It 00:23:22.619 --> 00:23:24.789 changes their whole value proposition. Because 00:23:24.789 --> 00:23:26.630 they are currently spending tens of billions 00:23:26.630 --> 00:23:28.750 of dollars buying up hundreds of thousands of 00:23:28.750 --> 00:23:31.690 these chips. Are they building dinosaurs? Are 00:23:31.690 --> 00:23:34.150 they just constructing massive, expensive, inefficient 00:23:34.150 --> 00:23:36.829 monuments to the old way of doing things? This 00:23:36.829 --> 00:23:39.769 brings us to the core thesis of the new IBM. 00:23:40.069 --> 00:23:43.049 Right, the title of the piece. OpenAI and Anthropic 00:23:43.049 --> 00:23:45.329 are operating under what the text implies is 00:23:45.329 --> 00:23:49.410 the trap of cloud -scale thinking. Okay. They 00:23:49.410 --> 00:23:51.809 have achieved incredible things, but their business 00:23:51.809 --> 00:23:54.230 model is entirely predicated on scaling horizontally. 00:23:54.750 --> 00:23:57.210 Meaning just adding more hardware. Exactly. They're 00:23:57.210 --> 00:24:00.470 scaling by purchasing more of the same accelerators, 00:24:00.549 --> 00:24:03.569 securing more massive data center leases, negotiating 00:24:03.569 --> 00:24:06.109 for more megawatts of power from the grid and 00:24:06.109 --> 00:24:08.750 hoarding more of the same massive training data 00:24:08.750 --> 00:24:11.289 sets. Because that's what worked for GPT -3 and 00:24:11.289 --> 00:24:14.230 that's what worked for GPT -4. Precisely. Or 00:24:14.230 --> 00:24:17.029 I should say, that's the established playbook. 00:24:17.549 --> 00:24:20.190 Their entire software stack, which relies on 00:24:20.190 --> 00:24:22.910 standard frameworks and traditional CUDA implementations, 00:24:23.009 --> 00:24:25.950 assumes that kernel launch overhead, the strict 00:24:25.950 --> 00:24:28.329 separation of training and inference, and the 00:24:28.329 --> 00:24:30.970 absolute necessity of massive cloud data centers 00:24:30.970 --> 00:24:34.490 are fixed, unavoidable costs. They just accept 00:24:34.490 --> 00:24:37.190 the friction. They accept the 50 % utilization 00:24:37.190 --> 00:24:40.089 rate because they believe the only way to get 00:24:40.089 --> 00:24:42.450 more intelligence is to plug in another GPU. 00:24:43.049 --> 00:24:46.769 They're acting exactly like IBM in the 1980s. 00:24:47.079 --> 00:24:51.140 Exactly. IBM assumed that mainframes and fixed, 00:24:51.200 --> 00:24:53.539 rigid hardware schedules were just the way computing 00:24:53.539 --> 00:24:56.680 had to be. They couldn't imagine that a guy in 00:24:56.680 --> 00:24:59.480 a garage tweaking the DRAM refresh rate could 00:24:59.480 --> 00:25:02.539 change the economics of computing. The parallels 00:25:02.539 --> 00:25:06.200 are striking. OpenAI and Anthropic now occupy 00:25:06.200 --> 00:25:08.880 the market position IBM held when the personal 00:25:08.880 --> 00:25:12.160 computer arrived. They're the incumbents. They 00:25:12.160 --> 00:25:13.839 have the capital, they have the market share, 00:25:13.960 --> 00:25:16.970 and they have the institutional momentum. Flapping 00:25:16.970 --> 00:25:19.250 airplanes, on the other hand, treats all of those 00:25:19.250 --> 00:25:21.890 fixed assumptions as the actual problem to be 00:25:21.890 --> 00:25:24.210 removed. But I've got

to play devil's advocate 00:25:24.210 --> 00:25:26.650 here. Go for it. Buying more data centers is 00:25:26.650 --> 00:25:30.029 a proven model. OpenAI is the undisputed heavyweight 00:25:30.029 --> 00:25:32.609 right now. Their models work beautifully. They 00:25:32.609 --> 00:25:34.769 do. Their brute force scaling laws have held 00:25:34.769 --> 00:25:37.349 up for years. Every time they add more compute, 00:25:37.490 --> 00:25:40.269 the model gets smarter. Isn't the $6 billion 00:25:40.269 --> 00:25:42.849 valuation for a company with no product just 00:25:42.849 --> 00:25:46.170 a massive, incredibly risky gamble that efficiency 00:25:46.170 --> 00:25:48.809 can beat brute force? It is a massive gamble. 00:25:49.069 --> 00:25:51.609 I mean, couldn't OpenAI just hire 500 engineers 00:25:51.609 --> 00:25:54.089 to build their own megakernel? They could try, 00:25:54.210 --> 00:25:56.930 but it's a calculated bet based on physical and 00:25:56.930 --> 00:26:00.569 economic limits. The $6 billion valuation isn't 00:26:00.569 --> 00:26:02.750 pricing a current product. What's it pricing 00:26:02.750 --> 00:26:05.779 then? It prices the possibility that the next 00:26:05.779 --> 00:26:08.759 major leap in AI capability doesn't come from 00:26:08.759 --> 00:26:11.019 signing another multi -billion dollar data center 00:26:11.019 --> 00:26:13.400 lease. Because we might run out of power. Exactly. 00:26:14.160 --> 00:26:17.079 The grid power literally might not exist to support 00:26:17.079 --> 00:26:20.180 the next generation of brute force scaling. The 00:26:20.180 --> 00:26:22.720 valuation prices the possibility that the next 00:26:22.720 --> 00:26:25.019 leap comes from finally utilizing the installed 00:26:25.019 --> 00:26:27.779 silicon to its full potential. The chips we already 00:26:27.779 --> 00:26:31.500 have. Right. The capital. That billion -dollar 00:26:31.500 --> 00:26:34.319 Series A has already moved to bet on this efficiency. 00:26:35.240 --> 00:26:38.440 And regarding OpenAI just copying it, you run 00:26:38.440 --> 00:26:41.099 into the innovator's dilemma. How so? When your 00:26:41.099 --> 00:26:43.140 entire billion -dollar infrastructure and your 00:26:43.140 --> 00:26:45.220 thousands of engineers are optimized for the 00:26:45.220 --> 00:26:48.039 old stack, pivoting to a megakernel architecture 00:26:48.039 --> 00:26:50.380 requires tearing down everything you've built. 00:26:51.099 --> 00:26:53.160 It's hard to change the engines while the plane 00:26:53.160 --> 00:26:55.990 is flying. Very hard. And there's another layer 00:26:55.990 --> 00:26:58.650 to this bet that is even more profound than just 00:26:58.650 --> 00:27:01.589 saving money on GPUs. It's about the data itself. 00:27:02.069 --> 00:27:05.009 The source material talks about the sample efficiency. 00:27:05.210 --> 00:27:07.690 Holy grail. This is where the hardware efficiency 00:27:07.690 --> 00:27:11.049 translates into actual intelligence. OK, tell 00:27:11.049 --> 00:27:13.670 me about sample efficiency. The founders of flapping 00:27:13.670 --> 00:27:16.730 airplanes have set a very specific target. They 00:27:16.730 --> 00:27:19.950 estimate that human beings are roughly 100 ,000 00:27:19.950 --> 00:27:22.910 to 1 million times more sample efficient than 00:27:22.910 --> 00:27:25.759 current AI models. Meaning a human toddler can 00:27:25.759 --> 00:27:28.740 see one picture of a cat, maybe have their parent 00:27:28.740 --> 00:27:31.140 point to it and say cat. And from that moment 00:27:31.140 --> 00:27:33.119 on, the toddler forever knows what a cat is. 00:27:33.200 --> 00:27:36.099 Yes. But an AI needs a data set of hundreds of 00:27:36.099 --> 00:27:38.400 thousands, if not millions of cat images from 00:27:38.400 --> 00:27:41.019 every possible angle just to learn the same concept. 00:27:41.220 --> 00:27:44.059 The human brain is a marvel of sample efficiency. 00:27:44.420 --> 00:27:48.039 Clearly. The AI requires a... brute force amount 00:27:48.039 --> 00:27:50.500 of data because its architecture currently is 00:27:50.500 --> 00:27:53.019 highly inefficient at extracting the underlying 00:27:53.019 --> 00:27:56.359 concepts and rules from a single exposure. It's 00:27:56.359 --> 00:27:58.839 just memorizing patterns. Right. Flapping airplanes 00:27:58.839 --> 00:28:01.880 treats this million -fold gap not as a permanent 00:28:01.880 --> 00:28:04.319 biological advantage, but as an engineering target. 00:28:04.619 --> 00:28:07.019 But how does a mega -kernel help an AI learn 00:28:07.019 --> 00:28:09.819 what a cat is faster? By freeing up compute cycles. 00:28:10.240 --> 00:28:13.170 Walk me through that. If you reclaim 50 % of 00:28:13.170 --> 00:28:15.390 the GPU's time that was previously wasted on 00:28:15.390 --> 00:28:18.210 launch taxes and memory bottlenecks, you don't 00:28:18.210 --> 00:28:21.190 necessarily just use that time to spit out the 00:28:21.190 --> 00:28:23.529 next word faster. What do you use it for? You 00:28:23.529 --> 00:28:25.529 can use those free compute cycles to run richer, 00:28:25.609 --> 00:28:28.150 more complex internal architectures. You can 00:28:28.150 --> 00:28:30.410 run vastly more internal experiments during the 00:28:30.410 --> 00:28:33.650 forward pass. Like it's contemplating the data? 00:28:33.990 --> 00:28:37.009 Exactly. You can allow the model to think longer 00:28:37.009 --> 00:28:39.309 about a single piece of data before moving on. 00:28:39.950 --> 00:28:42.049 If they can use their hardware efficiency to 00:28:42.049 --> 00:28:44.609 close even a fraction of that sample efficiency 00:28:44.609 --> 00:28:47.849 gap. If they do that, the whole paradigm collapses. 00:28:48.069 --> 00:28:50.369 The capital and the data required to build a 00:28:50.369 --> 00:28:53.589 capable AI would plummet drastically. Because 00:28:53.589 --> 00:28:55.309 you wouldn't need a massive data set anymore. 00:28:55.589 --> 00:28:57.910 You no

longer need to scrape the entire internet, 00:28:58.029 --> 00:29:00.430 which we are running out of, by the way. Right, 00:29:00.470 --> 00:29:03.049 the data wall. And you no longer need to spend 00:29:03.049 --> 00:29:05.009 a billion dollars on compute to train a frontier 00:29:05.009 --> 00:29:07.799 model. If a model can learn a concept from 10 00:29:07.799 --> 00:29:11.430 examples instead of 10 ,000, the barrier to entry 00:29:11.430 --> 00:29:14.009 for building superintelligence drops out of the 00:29:14.009 --> 00:29:16.910 cloud and into the garage. So we have solved 00:29:16.910 --> 00:29:19.509 the hardware bottleneck, or at least Flapping 00:29:19.509 --> 00:29:21.450 Airplanes believes they are on the verge of solving 00:29:21.450 --> 00:29:24.930 it. We have a path to sample efficiency. Yes. 00:29:25.049 --> 00:29:27.569 But doing so breaks something much older than 00:29:27.569 --> 00:29:29.849 the AI boom. It breaks a fundamental contract 00:29:29.849 --> 00:29:32.329 of computing that has existed since the very 00:29:32.329 --> 00:29:34.750 first teletype machines. This is a huge shift. 00:29:35.210 --> 00:29:37.670 The source text calls this the death of the terminal, 00:29:37.809 --> 00:29:40.309 and this is the part of the deep dive that really 00:29:40.309 --> 00:29:42.650 messes with my head. This is where the hardware 00:29:42.650 --> 00:29:45.849 evolution forces a deep philosophical shift in 00:29:45.849 --> 00:29:49.690 how humans interact with machines. How so? If 00:29:49.690 --> 00:29:52.210 we trace the history of computing, from the early 00:29:52.210 --> 00:29:55.549 days of the ENIAC with punch cards, to teletype 00:29:55.549 --> 00:29:57.369 machines printing on endless rolls of paper, 00:29:57.509 --> 00:30:00.910 to cathode ray tubes, to the glass terminals 00:30:00.910 --> 00:30:04.339 of the 1970s, to... graphical consoles, and all 00:30:04.339 --> 00:30:07.940 the way up to modern AWS cloud dashboards, they 00:30:07.940 --> 00:30:10.819 all share one fundamental, unquestioned requirement. 00:30:11.299 --> 00:30:14.809 Human readability. Exactly. The intermediate 00:30:14.809 --> 00:30:17.450 states of the machine, the file systems, the 00:30:17.450 --> 00:30:20.009 process schedulers, the network logs, the error 00:30:20.009 --> 00:30:23.170 reports, were all meticulously written and formatted 00:30:23.170 --> 00:30:25.789 so that a human operator could read them, monitor 00:30:25.789 --> 00:30:28.450 them, and intervene. Computers were designed 00:30:28.450 --> 00:30:31.390 to explain themselves to us at every step. Yes. 00:30:31.690 --> 00:30:33.789 They had to pause what they were doing, translate 00:30:33.789 --> 00:30:36.170 their internal binary states into English strings, 00:30:36.289 --> 00:30:38.990 and print them on a screen so we could feel like 00:30:38.990 --> 00:30:41.609 we were in charge. The contract was transparency 00:30:41.609 --> 00:30:44.730 through human readability. Logs are formatted 00:30:44.730 --> 00:30:46.970 so a person can understand them. Right. When 00:30:46.970 --> 00:30:49.569 a server fails, a systems administrator expects 00:30:49.569 --> 00:30:52.089 to be able to open a text file and read a timestamped 00:30:52.089 --> 00:30:54.799 error message explaining what happened. But flapping 00:30:54.799 --> 00:30:57.079 airplanes, mega kernel architecture absolutely 00:30:57.079 --> 00:30:59.339 shatters this contract, doesn't it? It shatters 00:30:59.339 --> 00:31:01.599 it completely. Because the scheduler now lives 00:31:01.599 --> 00:31:04.200 inside the GPU running continuously inside the 00:31:04.200 --> 00:31:07.099 KPM. Those intermediate activations no longer 00:31:07.099 --> 00:31:10.140 need a human readable form. The cycle does not 00:31:10.140 --> 00:31:13.400 pause for a human to review a log. Nope. The 00:31:13.400 --> 00:31:15.819 machine talks only to itself and to the physical 00:31:15.819 --> 00:31:18.599 world. Let's consider the physical cost of logging 00:31:18.599 --> 00:31:21.220 for a second. Okay. Translating a high dimensional 00:31:21.220 --> 00:31:24.089 vector state into a human readable string. Moving 00:31:24.089 --> 00:31:27.849 it from the GPU memory across the PCIe bus through 00:31:27.849 --> 00:31:30.630 the CPU into the operating system and rendering 00:31:30.630 --> 00:31:33.630 it on a monitor takes millions of clock cycles. 00:31:33.829 --> 00:31:36.309 Just to print some text. In the time it takes 00:31:36.309 --> 00:31:39.990 to print error 404 on a screen, the megakernel 00:31:39.990 --> 00:31:42.390 could have run a million matrix multiplications. 00:31:42.589 --> 00:31:45.609 Wow. By removing the requirement to log intermediate 00:31:45.609 --> 00:31:48.650 states, you remove a massive computational anchor. 00:31:48.910 --> 00:31:51.069 It's like having a brilliant translator working 00:31:51.069 --> 00:31:53.839 for you. Okay. I like this analogy. Usually they 00:31:53.839 --> 00:31:55.819 have to stop every five seconds, turn to you, 00:31:55.880 --> 00:31:57.980 and translate their internal thoughts into English 00:31:57.980 --> 00:32:00.240 so you can monitor their progress. Right. They 00:32:00.240 --> 00:32:01.960 spend more time translating their thoughts for 00:32:01.960 --> 00:32:04.119 you than they do actually solving the problem. 00:32:04.779 --> 00:32:07.039 If they suddenly realize they can think and act 00:32:07.039 --> 00:32:09.119 infinitely faster if they just stop translating 00:32:09.119 --> 00:32:11.220 for you, they just do the work. They just do 00:32:11.220 --> 00:32:13.880 it. Input arrives as raw sensor streams or tokens, 00:32:13.940 --> 00:32:16.980 and output leaves as actuator commands or the 00:32:16.980 --> 00:32:19.660 next internal state. The terminal, the screen 00:32:19.660 --> 00:32:21.900 we stare at, becomes nothing more than a debug 00:32:21.900 --> 00:32:24.140 port. that we only plug into when the machine 00:32:24.140 -->

00:32:26.779 physically breaks. I'd take that analogy a step 00:32:26.779 --> 00:32:29.500 further. Oh. It's not just that the translator 00:32:29.500 --> 00:32:32.319 stops speaking English to you. The translator 00:32:32.319 --> 00:32:34.579 begins developing their own internal shorthand 00:32:34.579 --> 00:32:38.079 that is so hyper -optimized for speed that even 00:32:38.079 --> 00:32:40.039 if you demanded they explain it, the concepts 00:32:40.039 --> 00:32:42.019 wouldn't map neatly onto human language anymore. 00:32:42.559 --> 00:32:45.779 Oh, that's creepy. The machine's internal representations 00:32:45.779 --> 00:32:49.660 become entirely alien. optimized purely for hardware 00:32:49.660 --> 00:32:52.480 efficiency. And this leads to another major boundary 00:32:52.480 --> 00:32:54.859 collapsing, one that is crucial for the future 00:32:54.859 --> 00:32:58.160 of AI. Which is? The boundary between training 00:32:58.160 --> 00:33:00.940 and inference. Right, because historically, those 00:33:00.940 --> 00:33:03.279 are two totally separate phases. Completely separate. 00:33:03.440 --> 00:33:05.599 You train a model in a massive data center, that's 00:33:05.599 --> 00:33:08.140 phase one. You freeze its weights, you deploy 00:33:08.140 --> 00:33:10.380 it to a phone or a server, that's phase two, 00:33:10.500 --> 00:33:13.380 inference. If the deployed model needs to learn 00:33:13.380 --> 00:33:16.329 something new, it can't. You have to collect 00:33:16.329 --> 00:33:18.569 that data, send it all the way back to the cloud, 00:33:18.869 --> 00:33:20.730 retrain a new version of the model, and push 00:33:20.730 --> 00:33:24.130 an update. It is a slow, disconnected loop. But 00:33:24.130 --> 00:33:26.130 with the MegaKernel, both training and inference 00:33:26.130 --> 00:33:28.869 run inside the exact same persistent environment. 00:33:29.369 --> 00:33:31.920 At the same time. Because you don't have to tear 00:33:31.920 --> 00:33:34.039 down the kernel, and because you aren't logging 00:33:34.039 --> 00:33:37.220 everything to a CPU manager, a local model running 00:33:37.220 --> 00:33:39.900 on an edge device can serve a request and learn 00:33:39.900 --> 00:33:43.099 from it simultaneously. Give me a real -world 00:33:43.099 --> 00:33:45.589 example of what that looks like. Imagine a robotic 00:33:45.589 --> 00:33:47.769 arm sorting recycling on a conveyor belt. Okay, 00:33:47.809 --> 00:33:50.589 got it. It's running a mega kernel locally. It 00:33:50.589 --> 00:33:52.569 reaches for a crushed soda can, but its grip 00:33:52.569 --> 00:33:55.750 slips. Standard error. In a conventional system, 00:33:56.029 --> 00:33:58.630 it would just log a failure, maybe try again, 00:33:58.829 --> 00:34:01.329 and eventually an engineer would gather all the 00:34:01.329 --> 00:34:03.609 failure logs, retrain the vision model in the 00:34:03.609 --> 00:34:06.369 cloud, and update the robot next week. The slow 00:34:06.369 --> 00:34:09.800 loop. With the mega kernel, the robot serves 00:34:09.800 --> 00:34:12.300 the inference request. attempting to grab the 00:34:12.300 --> 00:34:15.019 can, and simultaneously updates its internal 00:34:15.019 --> 00:34:17.679 weights based on the immediate tactile feedback 00:34:17.679 --> 00:34:20.320 that it failed. It just learns right then and 00:34:20.320 --> 00:34:23.280 there. It learns continuously, locally, in real 00:34:23.280 --> 00:34:25.679 time, without checkpointing or shipping data 00:34:25.679 --> 00:34:28.409 back to a cloud terminal. So where does that 00:34:28.409 --> 00:34:31.289 leave us? What is the new role of the human if 00:34:31.289 --> 00:34:33.369 we aren't monitoring the terminal, reading the 00:34:33.369 --> 00:34:36.940 logs, or manually initiating training runs? Human 00:34:36.940 --> 00:34:39.360 intervention is pushed entirely to the physical 00:34:39.360 --> 00:34:42.500 and conceptual edges of the system. The edges. 00:34:43.590 --> 00:34:45.809 We become the entities that set the high -level 00:34:45.809 --> 00:34:49.210 goals. We say, sort the recycling. We supply 00:34:49.210 --> 00:34:51.510 sparse, high -value labels when the machine is 00:34:51.510 --> 00:34:53.869 truly stuck. Like pointing out a brand new material 00:34:53.869 --> 00:34:57.170 it's never seen? Exactly. Or, in a physical system, 00:34:57.369 --> 00:34:59.889 we are the ones who arrive with a wrench to replace 00:34:59.889 --> 00:35:01.730 a broken actuator when the machine physically 00:35:01.730 --> 00:35:05.130 cannot move its arm. We're just mechanics. But 00:35:05.130 --> 00:35:07.150 the continuous stream of tokens, the massive 00:35:07.150 --> 00:35:09.829 matrices of activations, the memory transfers, 00:35:09.969 --> 00:35:13.050 that all stays securely inside the black box 00:35:13.050 --> 00:35:15.550 of the chassis. The black box. Neither side, 00:35:15.670 --> 00:35:17.949 human or machine, maintains a running commentary 00:35:17.949 --> 00:35:21.210 for the other anymore. That is deeply unsettling, 00:35:21.210 --> 00:35:23.909 but also incredibly elegant from an engineering 00:35:23.909 --> 00:35:26.619 perspective. It is. And the source material actually 00:35:26.619 --> 00:35:29.059 provides a literal blueprint for how a rogue 00:35:29.059 --> 00:35:31.460 lab can execute this shift and challenge the 00:35:31.460 --> 00:35:34.260 giants. They call it the 10 places to build highly 00:35:34.260 --> 00:35:37.619 optimized systems. A very detailed list. Now, 00:35:37.659 --> 00:35:39.480 we aren't going to just read a list of 10 points. 00:35:39.539 --> 00:35:41.460 Nobody wants to listen to a PowerPoint presentation 00:35:41.460 --> 00:35:44.780 on a podcast. I mean, in a deep dive like this. 00:35:44.880 --> 00:35:46.949 Right. Let's group them. But I think it helps 00:35:46.949 --> 00:35:49.369 to weave these 10 services into a narrative. 00:35:49.449 --> 00:35:51.389 Let's group them into three distinct thematic 00:35:51.389

--> 00:35:53.369 clusters for you listening at home and imagine 00:35:53.369 --> 00:35:55.730 how a startup operates under these rules. The 00:35:55.730 --> 00:35:57.449 first cluster is all about ruthless hardware 00:35:57.449 --> 00:36:01.289 efficiency. Which covers points 1, 2, 3, 7, and 00:36:01.289 --> 00:36:04.050 8 from the text. Exactly. This first cluster 00:36:04.050 --> 00:36:06.110 is essentially the physical manifestation of 00:36:06.110 --> 00:36:09.230 the 1987 QFresh hack brought into the modern 00:36:09.230 --> 00:36:11.809 era. If you are a garage startup trying to beat 00:36:11.809 --> 00:36:14.309 a trillion -dollar company, you start here. Okay, 00:36:14.349 --> 00:36:16.800 lay it out. Point one is removing the kernel 00:36:16.800 --> 00:36:19.619 launch tax, which we discussed at length, using 00:36:19.619 --> 00:36:21.760 that persistent runtime to recover wasted time. 00:36:22.019 --> 00:36:25.719 The mega kernel. Right. Point two is co -packing 00:36:25.719 --> 00:36:28.820 independent work inside one address space. Right. 00:36:28.880 --> 00:36:31.659 Forcing the memory -bound phases and the compute 00:36:31.659 --> 00:36:34.059 -bound phases to share the same long -lived kernel. 00:36:34.519 --> 00:36:36.880 Utilizing the same registers and interconnects 00:36:36.880 --> 00:36:38.920 concurrently instead of waiting in line. And 00:36:38.920 --> 00:36:41.380 point three is treating underused hardware units 00:36:41.380 --> 00:36:44.500 as free capacity. Like empty real estate. The 00:36:44.500 --> 00:36:47.380 text advises mapping the actual physical utilization 00:36:47.380 --> 00:36:50.039 of every single unit on the chip. The tensor 00:36:50.039 --> 00:36:52.780 cores, the memory accelerators, the NVLink interconnects. 00:36:52.780 --> 00:36:55.639 Everything. The philosophy is that anything running 00:36:55.639 --> 00:36:59.199 below 70 to 80 % sustained occupancy isn't good 00:36:59.199 --> 00:37:03.619 enough. It is an optimization surface. It is 00:37:03.619 --> 00:37:06.039 free real estate sitting inside the chip that 00:37:06.039 --> 00:37:08.739 your competitor is ignoring. 0 .7 is the one 00:37:08.739 --> 00:37:11.199 that really caught my eye. Building the scheduler 00:37:11.199 --> 00:37:13.719 into the silicon rather than on top of it. This 00:37:13.719 --> 00:37:15.739 is the realization of the KVM pattern we talked 00:37:15.739 --> 00:37:18.360 about. The kitten's virtual machine. Right. By 00:37:18.360 --> 00:37:20.380 letting the persistent runtime expose its own 00:37:20.380 --> 00:37:23.019 instruction model directly to the hardware, the 00:37:23.019 --> 00:37:25.300 software and the hardware scheduler co -evolve. 00:37:25.739 --> 00:37:28.400 You stop treating the GPU as a dumb black box 00:37:28.400 --> 00:37:30.980 accelerator waiting for high -level OS instructions. 00:37:31.320 --> 00:37:35.300 You treat it as an autonomous engine. Yes. And 00:37:35.300 --> 00:37:37.099 point eight brings it back to the physical world, 00:37:37.159 --> 00:37:39.750 which I love. Measure power and thermal limits 00:37:39.750 --> 00:37:43.250 as first class constraints. This is crucial for 00:37:43.250 --> 00:37:46.050 small teams. If you are literally in a garage, 00:37:46.289 --> 00:37:49.849 you might have a 20 amp breaker. You cannot rely 00:37:49.849 --> 00:37:52.429 on the infinite water cooling and dedicated grid 00:37:52.429 --> 00:37:55.829 substations of a billion dollar AWS data center. 00:37:56.050 --> 00:37:58.110 You'd blow the fuse immediately. You have to 00:37:58.110 --> 00:38:00.570 optimize for sustained watts per useful token. 00:38:00.730 --> 00:38:03.130 If you find thermal headroom, meaning the chip 00:38:03.130 --> 00:38:05.630 isn't running as hot as it safely could, you 00:38:05.630 --> 00:38:08.300 exploit it. Just like finding those unused DRAM 00:38:08.300 --> 00:38:11.039 cycles, you push the metal to the absolute melting 00:38:11.039 --> 00:38:14.000 point. The physical constraints enforce the discipline 00:38:14.000 --> 00:38:16.980 that leads to the megakernel. So that's the hardware. 00:38:17.260 --> 00:38:19.860 Now, the second cluster shifts from purely manipulating 00:38:19.860 --> 00:38:23.400 the hardware to the software evolution. This 00:38:23.400 --> 00:38:27.400 covers points 4, 6, and 9. Point 4 is what we 00:38:27.400 --> 00:38:29.639 just touched on with the robotic arm, collapsing 00:38:29.639 --> 00:38:32.599 the training inference boundary. The slow loop 00:38:32.599 --> 00:38:35.909 becomes a fast loop. If a robot can update on 00:38:35.909 --> 00:38:38.469 the data it just produced without shipping weights 00:38:38.469 --> 00:38:41.090 to the cloud, the software loop becomes infinitely 00:38:41.090 --> 00:38:43.309 tighter. You don't need a massive data center 00:38:43.309 --> 00:38:45.250 if the robots are learning on the job. Right. 00:38:45.900 --> 00:38:48.280 And point six is the big one we discussed earlier, 00:38:48.500 --> 00:38:51.280 optimizing for sample efficiency before parameter 00:38:51.280 --> 00:38:53.960 count. This is the paradigm breaker. Learning 00:38:53.960 --> 00:38:56.300 like a human toddler instead of a brute force 00:38:56.300 --> 00:39:00.079 scraper. If you can achieve a 10x or 100x reduction 00:39:00.079 --> 00:39:02.300 in the data required for a given capability, 00:39:02.699 --> 00:39:05.820 you multiply the value of every single watt and 00:39:05.820 --> 00:39:07.920 gigabyte you already have installed. Because 00:39:07.920 --> 00:39:10.460 it does more with less. You stop caring about 00:39:10.460 --> 00:39:13.039 having a trillion parameters if a 10 billion 00:39:13.039 --> 00:39:16.159 parameter model can learn perfectly from a tiny, 00:39:16.260 --> 00:39:19.119 high -quality data stream. You compete on the 00:39:19.119 --> 00:39:22.119 quality of learning, not the volume of memorization. 00:39:22.579 --> 00:39:25.739 And point nine is fascinating from a business

00:39:25.739 --> 00:39:28.010 strategy perspective. Publish the primitive, 00:39:28.230 --> 00:39:31.110 not the finished product. This is how a small 00:39:31.110 --> 00:39:34.010 team builds an ecosystem. By releasing open, 00:39:34.090 --> 00:39:36.989 narrow abstractions like those 16 by 16 Thunder 00:39:36.989 --> 00:39:39.590 Kittens matrix tiles, they allow other garage 00:39:39.590 --> 00:39:41.630 labs to build on their own metal. They share 00:39:41.630 --> 00:39:43.630 the building blocks. They don't release a massive 00:39:43.630 --> 00:39:46.409 bloated framework. They release the highly optimized 00:39:46.409 --> 00:39:49.230 blocks. It prevents developers from being locked 00:39:49.230 --> 00:39:51.469 into closed proprietary stacks that force them 00:39:51.469 --> 00:39:53.690 to buy expensive cloud compute from the big players. 00:39:53.929 --> 00:39:56.820 It democratizes the hardware efficiency. Which 00:39:56.820 --> 00:39:59.739 brings us to the final cluster, the human demotion. 00:39:59.840 --> 00:40:02.960 The spooky one. Points five and ten. This is 00:40:02.960 --> 00:40:05.420 the death of the terminal. Making intermediate 00:40:05.420 --> 00:40:08.400 states machine -readable only and keeping humans 00:40:08.400 --> 00:40:11.800 strictly at the goal and failure edges. Point 00:40:11.800 --> 00:40:14.219 five challenges every dashboard, every log file, 00:40:14.380 --> 00:40:17.039 and every cloud console that exists today. No 00:40:17.039 --> 00:40:19.760 more checking the logs. It argues forcefully 00:40:19.760 --> 00:40:22.659 that intermediate state, the activations, the 00:40:22.659 --> 00:40:25.969 cache. the optimizer states, should never leave 00:40:25.969 --> 00:40:28.809 the chassis in human -readable form. Just keep 00:40:28.809 --> 00:40:31.090 it inside the machine. You replace the beautiful 00:40:31.090 --> 00:40:33.949 complex dashboard with sparse, high -value signals. 00:40:34.150 --> 00:40:36.570 A green light for working, a red light for broken. 00:40:36.849 --> 00:40:39.909 Simple as that. And point 10 finalizes the architecture. 00:40:40.829 --> 00:40:44.849 Continuous operation stays inside the box. Inside 00:40:44.849 --> 00:40:48.190 the megakernel. The ultimate workflow bottleneck 00:40:48.190 --> 00:40:50.710 in modern AI systems is currently a human reading 00:40:50.710 --> 00:40:54.190 a terminal to decide what to do next. You remove 00:40:54.190 --> 00:40:57.630 the terminal, you remove the bottleneck. So what 00:40:57.630 --> 00:41:00.840 does this all mean? When we look at this blueprint, 00:41:01.039 --> 00:41:04.880 this staggering $6 billion valuation for a company 00:41:04.880 --> 00:41:08.239 with two dozen employees, and this philosophical 00:41:08.239 --> 00:41:11.139 shift away from human readability, where does 00:41:11.139 --> 00:41:14.710 this leave the AI industry right now? Well, to 00:41:14.710 --> 00:41:16.630 summarize the core thesis of the source material, 00:41:17.110 --> 00:41:20.349 flapping airplanes has not shipped this final 00:41:20.349 --> 00:41:22.710 world -changing system yet. Right. They still 00:41:22.710 --> 00:41:24.170 have to prove they can do it on a massive scale. 00:41:24.599 --> 00:41:27.179 That is the crucial caveat that justifies the 00:41:27.179 --> 00:41:29.639 skepticism. But they have a massive war chest 00:41:29.639 --> 00:41:32.880 of capital. They have the undeniable prior open 00:41:32.880 --> 00:41:35.420 source record with Thunder Kittens, proving the 00:41:35.420 --> 00:41:37.639 math actually works on the silicon. The technical 00:41:37.639 --> 00:41:39.780 record is there. And they have a very clear, 00:41:39.860 --> 00:41:42.780 ruthless statement of intent. Until they ship 00:41:42.780 --> 00:41:45.480 it, OpenAI and Anthropic will continue to be 00:41:45.480 --> 00:41:48.179 the largest buyers of the old, inefficient architecture. 00:41:48.639 --> 00:41:52.320 They will continue to build the new IBM. Exactly. 00:41:52.909 --> 00:41:55.210 But the metal, the raw silicon capability is 00:41:55.210 --> 00:41:57.530 already there sitting idle in the chips we have 00:41:57.530 --> 00:42:00.349 today. The only question now is who actually 00:42:00.349 --> 00:42:03.250 finishes the job of using all of it. It is a 00:42:03.250 --> 00:42:05.369 race to the bottom of the silicon. But as we 00:42:05.369 --> 00:42:07.530 wrap up, I want to leave you with a final thought 00:42:07.530 --> 00:42:09.710 to mull over building on this idea of the death 00:42:09.710 --> 00:42:12.489 of the terminal. Go ahead. If flapping airplanes 00:42:12.489 --> 00:42:15.250 is right and we successfully build these mega 00:42:15.250 --> 00:42:18.489 kernels where machines only output physical actuator 00:42:18.489 --> 00:42:20.409 commands and internal states. Where they stop 00:42:20.409 --> 00:42:22.960 talking to us. Right. If we permanently remove 00:42:22.960 --> 00:42:25.239 the human readable terminal because it slows 00:42:25.239 --> 00:42:28.199 the machine down, are we inadvertently creating 00:42:28.199 --> 00:42:31.099 a scenario where AI develops an internal operational 00:42:31.099 --> 00:42:35.079 language or a logic so deeply optimized and native 00:42:35.079 --> 00:42:37.659 to the silicon that even if we wanted to plug 00:42:37.659 --> 00:42:39.920 a terminal back in one day to see how it thinks, 00:42:40.000 --> 00:42:42.699 we would be fundamentally incapable of understanding 00:42:42.699 --> 00:42:45.519 it? It's entirely possible. We might build a 00:42:45.519 --> 00:42:48.909 machine so fast and so efficient. that its thoughts 00:42:48.909 --> 00:42:51.349 become completely inaccessible to the human mind. 00:42:51.710 --> 00:42:53.989 We are actively designing the machine to turn 00:42:53.989 --> 00:42:56.210 the lights out on us so it can work faster in 00:42:56.210 --> 00:42:58.889 the dark. That is a chilling thought to end on. 00:42:59.469 --> 00:43:01.650 That's all for today's Deep Dive. Just a quick 00:43:01.650 --> 00:43:04.510 reminder before you go, this is the Read Multiplex 00:43:04.510 --> 00:43:07.269 podcast. Support Brian's work, buy him a coffee, 00:43:07.389 --> 00:43:10.519 or become a member. See you next time. Welcome 00:43:10.519 --> 00:43:14.480 to the multiplex, where the future comes alive. 00:43:15.179 --> 00:43:19.739 Past, present, future tech, open up your eyes. 00:43:20.460 --> 00:43:23.840 Brian Romella leads the way with research deep 00:43:23.840 --> 00:43:28.489 and true. Technology and history, he connects 00:43:28.489 --> 00:43:32.070 them all for you. Brian Romella leads the way 00:43:32.070 --> 00:43:34.210 with research deep and true. Technology and history, 00:43:34.469 --> 00:43:36.289 he connects them all for you. Ancient sparks 00:43:36.289 --> 00:43:38.670 to modern code, he traces every thread good. 00:43:39.070 --> 00:44:14.630 Exploration. But human moves on. Love equation 00:44:14.630 --> 00:44:20.389 for AI. Zero human companies building from the 00:44:20.389 --> 00:44:26.320 start. Thermodynamic wages earning the key. Secrets 00:44:26.320 --> 00:44:30.420 we will read. Experiment in publishing real researches 00:44:30.420 --> 00:44:36.349 and spending every. Great.