WEBVTT
00:00:00.160 --> 00:00:12.000
Well, like, I think my take on events these days is just like the agentic era has made open-ended judging a bit cooked because it's like, how do you tell whether something is amazing versus not?
00:00:12.160 --> 00:00:14.480
If it's just like if Claude one-shot it.
00:00:14.800 --> 00:00:26.160
But if someone did a hill climb on a task that we can all collectively agree on is hard and made like a breakthrough, that's better than what I would do myself as like an advanced user.
00:00:26.399 --> 00:00:29.359
Like, if they can just do that during the hackathon, like I'm very impressed.
00:00:29.440 --> 00:00:32.960
And so, like, it just doesn't matter how they did it, it would matter is the output.
00:00:33.119 --> 00:00:40.960
So, I think that I'm very open to, and I think it's gonna become a lot more interesting doing it with open models this year, because like the open models are good finally at these tests.
00:00:41.119 --> 00:00:43.840
I think it's it looks like they've been RL'd on kernel data.
00:00:43.920 --> 00:00:47.840
Yeah, and so I think it's gonna be a lot more exciting than it was like a year ago.
00:00:57.280 --> 00:01:02.799
Welcome to ADSP the podcast, episode 304, recorded on July 20th, 2026.
00:01:02.880 --> 00:01:08.719
My name is Connor, and today with my co-host Bryce, we finish part three of our three-part conversation with Mark Seraphim.
00:01:08.799 --> 00:01:31.519
In this episode, we continue our conversation about GPU mode, auto research, Mark's favorite books, and more.7, which feels like a long time ago, was like auto research on Cubs a histogram.
00:01:31.680 --> 00:01:40.799
And that auto research, I didn't give it any steering, and it basically that auto research popped out six different relatively novel algorithms.
00:01:41.040 --> 00:01:47.840
Two of them were like kind of existing, a twist on existing techniques, but the rest were like really kind of I think novel.
00:01:48.000 --> 00:01:51.439
They did build on like existing things from literature.
00:01:51.519 --> 00:01:55.439
This is the SMIM cache that Connor that we talked about in the last episode.
00:01:55.599 --> 00:01:59.920
Like, certainly like caches, hash tables are like a thing that are in literature.
00:02:00.079 --> 00:02:06.400
And I think they're you know, histograms are very similar to to to to to a hash table in some way.
00:02:06.560 --> 00:02:23.280
So, you know, maybe there is actually some literature out there on this particular technique, but it really did feel like it was it was a novel, it wasn't a completely novel idea, but it was a novel combination of ideas, and there was no chance I would have ever come up with that idea on my own.
00:02:23.520 --> 00:02:23.840
Yeah.
00:02:24.159 --> 00:02:31.759
I I I guess Bryce, I'm curious to ask you on the specific example, like what was what was it like trying to merge it back into Cub?
00:02:31.840 --> 00:02:33.039
I don't know because I don't know if you did or something.
00:02:33.439 --> 00:02:34.800
I'm in the process of explaining that right now.
00:02:35.120 --> 00:02:46.960
So so the the interesting thing with that, with the cub histogram, was the key like auto research run was done in like I think the last auto research run I did for that was like end of May.
00:02:47.120 --> 00:02:52.879
And so I I came out of that with like the the the candidate, you know, code in kernels.
00:02:53.039 --> 00:03:01.599
And what I've had to do since then is I was running auto research with Cub's existing benchmark, which handles which has basically like two different distributions.
00:03:02.000 --> 00:03:08.080
And you know, I knew I needed to do a much wider input characterization.
00:03:08.319 --> 00:03:14.159
So I constructed a whole lot more inputs, I constructed adversarial inputs.
00:03:14.400 --> 00:03:22.240
Me and the agent learned that in fact the way that Cobb generated its histogram inputs for the benchmark have always been broken since the beginning of time.
00:03:22.639 --> 00:03:31.120
And the the the full like input characterization r takes like 30 hours on a B200 to like really get good data.
00:03:31.280 --> 00:03:39.840
And with Cobb, there's the constraint that we really can't ship six different algorithms, one for this particular dynamic shape, one for that particular dynamic shape.
00:03:40.000 --> 00:03:43.120
And and so I've done more like manual work.
00:03:43.439 --> 00:03:44.240
How come, by the way?
00:03:44.639 --> 00:03:47.120
Like, was it because it's sort of uh binary size?
00:03:49.759 --> 00:03:50.000
Yeah.
00:03:50.159 --> 00:03:57.599
So so I needed to really to pick like I was gonna have to do go with two algorithms, one for the small bin case and one for the large bin case.
00:03:57.680 --> 00:04:04.639
And there are a couple of like additional post-auto research optimizations that me and the agent did.
00:04:04.879 --> 00:04:18.959
And the the those things were basically they weren't new algorithms, it was just like the right combination of things, like combining the SMEM cache with the existing privatized, you know, histograms and global memory.
00:04:19.120 --> 00:04:32.720
And then like at one point we we turned off warp coalescing before you hit the SMEM cache was like a big loss, but doing coalescing on spills from the cache was a big win.
00:04:32.879 --> 00:04:38.399
And like I just got it sort of nailed down, and I I've looked at the code a few times throughout this process.
00:04:38.639 --> 00:04:40.160
I think there's two things I can try.
00:04:40.240 --> 00:04:50.399
The first thing is I can try to run my simplify scale, and the simplify scale is basically minimize the bytes of code difference, of non-common code difference from upstream.
00:04:50.959 --> 00:04:59.279
The alternative I think would be to identify like a series of commits of like, okay, this is like the here's like four PRs.
00:04:59.360 --> 00:05:06.720
We could break this down into four PRs, and let's just let's just do the four PRs as like from fresh.
00:05:06.800 --> 00:05:16.160
Like, don't try to take the existing code and like shape reshape it into a PR, but just like using the existing code as a reference point, generate the PRs.
00:05:16.319 --> 00:05:21.360
And I think those PRs are gonna go up uh sometime this month.
00:05:21.600 --> 00:05:26.319
And my guess is gonna be that it'll take two to two months or so to merge.
00:05:26.480 --> 00:05:29.519
I think it's in a shape now where it will be acceptable.
00:05:29.839 --> 00:05:33.759
The yeah, I mean the current the current code is it's a little bit of a mess.
00:05:33.920 --> 00:05:38.560
Like the biggest problem, the bit the single biggest problem was not following conventions in the code base.
00:05:38.639 --> 00:05:42.399
So like there was a convention for how tuning got done and it wasn't followed.
00:05:42.560 --> 00:05:47.759
And the a good part of it was things that the model never could have known.
00:05:48.079 --> 00:06:02.240
For example, like, oh, you know, this is the way that we used to do the tuning in this part of code, but you know, there's a new there's a new policy that's not documented anywhere for how we would do this, but the PR probably won't be accepted if it doesn't follow this new convention.
00:06:02.480 --> 00:06:18.079
Or like there's you know, there's this convention that we if you want to have an algorithm that dynamically dispatches to two, if you want to have a cub routine that can dynamically dispatches to two algorithms, like you need we need to get a special sign-off from someone.
00:06:18.160 --> 00:06:19.839
And that's not documented in the code.
00:06:19.920 --> 00:06:22.000
So how could the model have ever known that?
00:06:22.319 --> 00:06:23.279
This is this is interesting.
00:06:23.360 --> 00:06:42.639
Like basically, the this sort of makes sense if you sort of assume, like this is my guess, by the way, that like the cub team has been working on it for a while, that the original team hasn't disbanded much, and so a lot of these sort of like unwritten rules are actually in people's heads, and they all have internalized them, and there's no reason to write docs because maybe the team doesn't grow very quickly.
00:06:42.800 --> 00:06:47.680
But then, like in an age where you're forcing them to verbalize everything, they're like, Oh, goddammit, like this is so much work.
00:06:47.759 --> 00:06:49.600
I have to write docs now and I have to write tests.
00:06:49.759 --> 00:06:51.199
Yeah, yeah, I see interesting.
00:06:51.360 --> 00:07:10.800
Yeah, but I think it will be very interesting to see whether we can get the get this landed because Cobb in particular is a really high barrier to land because you need to be good across a really wide range of inputs, and you know, you you you you can't like ignore edge cases.
00:07:10.959 --> 00:07:25.360
And so it'll be I I I I'm hopeful because I do think it's a it's like a pretty big win uh for the for the cases that it like now I've gotten it to the point where essentially there's there's no adversarial inputs that are really bad for it.
00:07:25.439 --> 00:07:26.399
There may be one left.
00:07:26.480 --> 00:07:27.439
I gotta try that.
00:07:27.600 --> 00:07:28.000
Yeah.
00:07:28.160 --> 00:07:37.920
Have you have you have you done any auto research work that has landed upstream PRs in like big projects that have a high bar for quality?
00:07:38.319 --> 00:07:39.199
Personally, no.
00:07:39.360 --> 00:07:48.560
Like I think like around the time I left Meta, there was like I think an R like a fused RMS norm, which was like the first AI generated kernel that was merged, but that was like not my work.
00:07:48.639 --> 00:07:52.240
That was like Laura Laura's work on the kernel agent team.
00:07:52.480 --> 00:07:59.839
I do remember though, the main experience we had at the time was that like the kernel was quite long, and so I think it was important to deslopify it.
00:08:00.000 --> 00:08:10.800
I think there was sort of sometimes a bit of an ideological debate between like what's the point of deslopifying it, like just don't read it, to which like the PyTorch core maintainers were like, What are you talking about?
00:08:10.879 --> 00:08:14.079
Because this is maintenance debt, I need to understand every single line.
00:08:14.240 --> 00:08:24.240
I'm not sure which of those two stances is correct because like some of those PyTorch people felt very strongly about these things, are just like no longer working on the team, and so like I actually don't know.
00:08:24.319 --> 00:08:29.040
Like, like, but kind of like one thing that stands out to me at least was like this one.
00:08:29.120 --> 00:08:39.679
Um I think like this is sort of a paraphrase from a talk by Terence Stowl, so it's not exactly what he said, but it's like what I came away with, which is that like he was sort of talking about like I mean, let me just not just talk about kernels.
00:08:39.759 --> 00:08:54.879
Like, basically, like my my hypothesis is that typically by the time you're done writing a complicated kernel, you already have simplified it multiple times because as you're writing something, you're like evolving it, you're fixing it, and you've also like understood it.
00:08:55.039 --> 00:09:04.639
Like by the end of it, if you sort of like artisanally crafted something, basically uh generating, understanding, simplifying is like in some sense the same act.
00:09:04.799 --> 00:09:15.600
But like if we're in an age where people are only doing generation, it becomes like I think a bit of a problem because like let's say your kernel introduces a race condition, but it's like non-deterministic.
00:09:15.840 --> 00:09:18.320
Well, you know, that really sucks for a model training run.
00:09:18.399 --> 00:09:23.759
But maybe it's forgivable for like a tiny little inference service that like runs some sort of speculative optimization.
00:09:23.919 --> 00:09:29.360
So there's like a risk-reward thing where like maybe there's some areas where it need to be really paranoid and this is not acceptable.
00:09:29.519 --> 00:09:42.320
But generally, like when I broadly look at this, I just don't buy the people don't care because you know, as we talked at the beginning of this episode on like if people are freaked out about non-determinism, like boy are they gonna be freaked out about like straight up incorrectness.
00:09:42.399 --> 00:09:44.799
Um bugs that are like non-deterministic as well.
00:09:44.960 --> 00:09:49.120
Like, I think this is bound to cause conspiracy theories, uh, especially at scale, right?
00:09:49.200 --> 00:09:51.519
Like, if let's say 1% of your users only see something.
00:09:51.600 --> 00:09:51.919
Yeah, yeah.
00:09:52.399 --> 00:09:56.000
So, my broad take is like I think we should solve the hard problem.
00:09:56.080 --> 00:09:59.360
I don't think the pro the solution is people are reading the code.
00:09:59.519 --> 00:10:04.879
Like that that to me feels like uh like no, like, because it's let's say the QR problem.
00:10:05.039 --> 00:10:13.840
You know, at first I was like, hey, I'm gonna sort of synthesize all the kernels by all 170 users, and then I'm gonna try to come up with like a beautiful QR kernel.
00:10:14.000 --> 00:10:19.840
Uh, you know, like I've I've yet to do this, it feels onerous every time I think about like sitting down and doing this exercise.
00:10:20.080 --> 00:10:26.720
But at the same time, if I just ask it like Godex and I'm like, hey, Godex, go cook, like, look at all these kernels and tell me what comes up.
00:10:26.879 --> 00:10:28.240
In some sense, I don't even know.
00:10:28.320 --> 00:10:29.279
Did it read everything?
00:10:29.440 --> 00:10:31.519
Like, did it like really understand like everything?
00:10:31.679 --> 00:10:33.200
Did it capture the key ideas?
00:10:33.360 --> 00:10:40.320
Like, maybe there's a slow kernel that had more elegant ideas, will it really capture them or will it really just prioritize the first like dozen kernels that are fast?
00:10:40.480 --> 00:10:41.279
Like, I'm not sure.
00:10:41.440 --> 00:10:45.039
Like, in some sense, like I don't know what the model is doing, like I can't inspect it.
00:10:45.200 --> 00:10:47.600
Like, so it does feel like it's an alien artifact.
00:10:47.679 --> 00:10:48.879
I tell it go cook.
00:10:49.120 --> 00:10:59.360
But my hope is that like we can sort of figure out like how to come up with beautiful kernels from a large sum of fast but slop kernels.
00:10:59.600 --> 00:11:07.440
I just think like it's maybe in the beginning there's a bit more manual than we'd like, but I'm I'm fairly confident we can figure out like good tooling for for problems like this.
00:11:07.759 --> 00:11:24.720
So, first of all, I I gotta say, I I f I find I find it very interesting the proposition that a like a uh a race condition, slightly buggy kernel that's like good 99% of the time but and really fast, maybe it's good enough in some cases.
00:11:24.799 --> 00:11:26.000
I hadn't considered that before.
00:11:26.159 --> 00:11:27.120
Maybe maybe it is.
00:11:27.360 --> 00:11:28.399
Um for inferences.
00:11:29.279 --> 00:11:35.440
So my goal in auto research here is to think about what's the process that I as a human would do.
00:11:35.600 --> 00:11:40.799
And I think that both for humans and agents, it's very hard to optimize for multiple things at once.
00:11:40.960 --> 00:11:52.480
And every time that I've had to myself do like deep like kernel performance work, it's almost always where I have a phase where I'm trying to reach speed of light.
00:11:52.559 --> 00:12:05.759
And during that phase, you know, I'm adding restrict places, I'm adding in assumes, I'm adding in this and that, I'm rebenchmarking, I'm trying things, I'm if-defing stuff out, you know, I'm I'm I've got some dead code branches, etc.
00:12:06.159 --> 00:12:09.039
And then like, and then I reach performance.
00:12:09.120 --> 00:12:10.639
And I'm like, ah, great.
00:12:10.799 --> 00:12:11.840
This is the right thing.
00:12:12.000 --> 00:12:22.799
And then I've got this mess, and then what I do is I go back and I'm like, okay, okay, I'm gonna delete all the if-defs, I'm gonna, I'm gonna, you know, take out those like if I take out this assume, is it fine?
00:12:23.039 --> 00:12:26.879
Okay, and you know, and then I've got I've always got like four versions of this code.
00:12:27.039 --> 00:12:31.759
And it's like, okay, well, like I can get I can delete this variant that I don't need anymore, and I can delete that variant.
00:12:31.919 --> 00:12:43.120
And it's like I always do this process where I've first spent a bunch of time trying to find speed of light, and then once you've found speed of light, then like you have an Oracle answer.
00:12:43.279 --> 00:12:46.960
And then I can try to write better, cleaner versions.
00:12:47.200 --> 00:13:03.840
And so I think it's it's to me, it is like a natural process to do this V-shaped thing, or maybe not V or upside down V, where like you're you're climbing up to find the fast thing, and then like you're climbing down to make it not so ugly.
00:13:04.000 --> 00:13:11.679
And I think one of the, you know, maybe maybe I'm wrong about this idea that you or I said that like you can't always optimize for two things at once.
00:13:11.919 --> 00:13:16.000
Like it's harder to optimize for multiple things, harder to to optimize for multiple things.
00:13:16.159 --> 00:13:23.519
And I think that's true, but I think part of that is just that we don't have the right reward metrics and safeguards in place.
00:13:23.600 --> 00:13:35.840
That like the to me, if if with my auto research process always generating these like 10,000 line, you know, slop things that's got all this dead code over, that to me means I I haven't put the right guardrails in place.
00:13:35.919 --> 00:13:39.919
I haven't put the right linters and the right things to like for code quality.
00:13:40.080 --> 00:13:46.240
And if I did more of that, then like I could probably generate a better kernels.
00:13:46.320 --> 00:13:48.720
And also I think that would help with the stalls.
00:13:48.879 --> 00:13:53.440
And yeah, not not everything can be like deterministically checked or machine checked.
00:13:53.519 --> 00:14:05.919
The other thing is like I so right now I have in my framework I I uh you know the model can go off, I I let it it can do a macro optimization, a micro optimization, and then a combine.
00:14:06.000 --> 00:14:08.720
And I do have a fourth type called simplify.
00:14:08.879 --> 00:14:13.279
And usually I don't let it run simplify during the main auto research loop.
00:14:13.440 --> 00:14:14.240
But maybe I should.
00:14:14.320 --> 00:14:16.399
Maybe it should be simplifying as it goes.
00:14:16.559 --> 00:14:25.200
I just the reason I set it up this way is that I've I guess I've always done some of the simplification as I go, but usually I make a mess of things until I find speed of light.
00:14:25.519 --> 00:14:29.120
Yeah, it's an like your harness is like reflecting like your own process.
00:14:29.519 --> 00:14:29.759
Yeah.
00:14:30.000 --> 00:14:30.320
Yeah.
00:14:30.480 --> 00:14:31.039
I'm not sure.
00:14:31.360 --> 00:14:33.360
Like I think it's hard for me to say this is right or wrong.
00:14:33.519 --> 00:14:36.320
Like it's this seems like much more of a like personal choice.
00:14:36.399 --> 00:14:39.039
So I can see argument good arguments being made either way.
00:14:39.279 --> 00:14:39.679
Yeah.
00:14:39.840 --> 00:14:40.240
Yeah.
00:14:40.480 --> 00:15:02.080
So yeah, it's I I I find I find the the the the mission that that you guys have at core automation is very compelling to me because like it seems it seems very evident that like if you can make the model really good at automating the development of the model, then like you can like then it can all it can do all the rest, right?
00:15:02.240 --> 00:15:04.399
You can just sit back and sip tea.
00:15:05.440 --> 00:15:16.080
I mean what's cool about it too is that like I uh again, because like for me this is all new, but like by virtue of focusing on like on a few problems, I think it simplifies like a lot of things.
00:15:16.240 --> 00:15:24.559
Like uh, you know, we don't have to have like a model that's like say necessarily generally capable of all sorts of things, but it it primarily needs to be good at things that are tasks in the lab.
00:15:24.799 --> 00:15:28.879
So like you can think of every individual person in the lab as like an eval in some sense.
00:15:29.039 --> 00:15:29.279
Yeah.
00:15:29.679 --> 00:15:31.279
And I think this is like very helpful for me.
00:15:31.440 --> 00:15:37.360
And you know, when you look at like the kinds of tasks people do in the lab, like it's just it becomes like very self-evident, like how you should do the rest.
00:15:37.519 --> 00:15:46.159
That like, let's say for kernels at least, like none of the evals I know of open source, and I, you know, I know this because I've also created a couple of them, are like sufficient for what we want.
00:15:46.320 --> 00:15:56.799
And then it's like very cool thinking through like basically like how do you like basically what would you know the way I explain this to my colleagues is like what evals would help us like aura farm really hard?
00:15:57.039 --> 00:16:01.200
You know, and then these tend to be like capabilities that you yourself think are really impressive.
00:16:01.360 --> 00:16:01.519
Yeah.
00:16:01.679 --> 00:16:04.320
Like like what do I think is at the limits of my capabilities?
00:16:04.559 --> 00:16:10.159
And if I can get like our next release to do those things, you know, that then I think I'd be really, really proud of the work we did.
00:16:10.399 --> 00:16:10.799
Yeah.
00:16:11.039 --> 00:16:16.559
You know, and the the another the last point I'll touch on before I let everybody go and I'll go to sleep.
00:16:16.720 --> 00:16:23.840
You know, Connor, you mentioned earlier that like like the stuff you were doing, you know, like I stopped it because it it wasn't valuable.
00:16:24.080 --> 00:16:39.120
And I I just keep coming across places in which the usage data, the like the experimentation, like the the history of what has been tried is just so valuable.
00:16:39.360 --> 00:16:53.440
It's valuable for like one learning like an you can analyze it, you can improve things, you can analyze how particular tools or frameworks were used, and like once you have data at scale, like you can learn a lot from that.
00:16:53.600 --> 00:17:07.759
It it's it's out it's valuable for like simulating how you know how these long horizon tasks work, which you know can tell us a lot about how various like frameworks and hardware might perform.
00:17:07.920 --> 00:17:13.920
And then it can be valuable for training models for for doing post-training and RL.
00:17:14.000 --> 00:17:21.519
And it can be valuable for like extracting, like, oh, like maybe this is something that would be a good, you know, a good benchmark.
00:17:21.599 --> 00:17:24.640
And I I just I think there's so much value in all this data.
00:17:24.720 --> 00:17:39.440
And like that's one of the things I think is great about GPU mode is it's just it's this, you know, it's this farm that's producing all of this super high value data, like open data that we can all benefit from.
00:17:39.759 --> 00:17:51.200
I mean, Bryce, honestly, to me, it seems like the the more the more you're talking, like the more I get excited about this idea of like, why don't we just make it easier for people to compete in our competitions using open model tokens?
00:17:51.440 --> 00:17:57.359
We can subsidize the tokens by picking the problems to be the kernels that show up in the open models.
00:17:57.519 --> 00:18:05.920
And we get this like nice recursive self-improvement loop with people doing search over the best prompts collectively, because I think that's like highly non-obvious.
00:18:06.400 --> 00:18:11.359
For this, for these sprints over the weekend, like I and the Connor's definitely gonna have to cut this part out.
00:18:16.799 --> 00:18:26.480
Because it was like advertised as like, come, like we're gonna do agentic engineering on this hard problem, like they didn't have to learn anything about CUDA first.
00:18:26.640 --> 00:18:30.240
What they did was they had to learn how to set up auto research.
00:18:30.400 --> 00:18:43.359
And then, like on the second day, once they'd produced some auto research results, then like we like sat down and I talked to them about like the problem and taught them a bit about GPU architecture, and like they were able to learn more because they'd already already accomplished something.
00:18:43.440 --> 00:18:46.559
They they didn't know what they'd accomplished yet, because the agent had accomplished something.
00:18:47.759 --> 00:18:50.880
Yeah, but but but but they got like that sort of first like dopamine rush, right?
00:18:50.960 --> 00:18:51.920
Like I made something faster.
00:18:52.400 --> 00:19:21.839
Definitely like one thing I I I have this vision for like in the future of like doing, I guess the way I I know you said you weren't such a fan of like in-person events, but I'm imagining like GTC Auto Research Academy, where like you just like you have two hours every day where you come you give everybody unlimited inference for the week, you come in for two hours every day, and it's like a study group because you know you're like in between the study group meetings, people are going and running stuff and producing results, and then you come back and you you check in.
00:19:21.920 --> 00:19:38.720
And yeah, I think it'd be I I think it'd be very valuable to have open tokens, and I think people position open tokens as a public good as like this idea of like, oh, it's like it's goodwill, but for me it seems like it's insanely valuable to NVIDIA to do this.
00:19:38.880 --> 00:19:39.440
Yeah, yeah.
00:19:40.160 --> 00:19:57.519
Like, but basically, I just to sort of be specific on the like I think my take on events these days is just like the agentic era has made open-ended judging a bit cooked because it's like how do you tell whether something is amazing versus not if it's just like if Claude one-shot it?
00:19:58.079 --> 00:20:09.440
But if someone did a hill climb on a task that we can all collectively agree on is hard and made like a breakthrough, that's better than what I would do myself as like an advanced user.
00:20:09.599 --> 00:20:12.640
Like, if they can just do that during the hackathon, like I'm very impressed.
00:20:12.720 --> 00:20:16.160
And so, like, it just doesn't matter how they did it, it matters is the the output.
00:20:16.400 --> 00:20:24.319
So I think that I'm very open to, and I think it's gonna become a lot more interesting doing it with open models this year, because like the open models are good finally at these tests.
00:20:24.400 --> 00:20:27.119
I think it's it looks like they've been RL'd on kernel data.
00:20:27.200 --> 00:20:27.440
Yeah.
00:20:27.680 --> 00:20:30.559
And so I think it's gonna be a lot more exciting than it was like a year ago.
00:20:30.799 --> 00:20:42.160
I want to pitch you on another idea I had, which is you know, you talked about the idea of having like a package, like here's how to deploy a you know, like like an inference service to do this like at your company.
00:20:42.400 --> 00:20:48.640
I've been thinking about this idea of, well, you know, what we want to do is we we need two we need GPUs for two things.
00:20:48.720 --> 00:20:54.799
We need GPUs one to run the tests and the benchmarks, and two, we need GPUs to run the inference.
00:20:55.039 --> 00:21:01.920
And you know, there's like web search is something that like runs like inference, like, you know, on the inference provider side.
00:21:02.079 --> 00:21:11.359
Like, why couldn't we have like a you know inference provider side like tool call that just like like like that's like modal that just like runs the serverless thing there?
00:21:11.440 --> 00:21:21.119
And then like if you're a company and you have like you know an 88x B200 machine, it can both run your inference and run your serverless, you know, evals.
00:21:21.279 --> 00:21:24.240
Because a lot of people seem gated on the GPU access too.
00:21:24.559 --> 00:21:27.279
Yeah, I I think of these as like very related problems.
00:21:27.519 --> 00:21:31.200
It's just that like I would imagine more people would know what to do with tokens versus a GPU.
00:21:31.279 --> 00:21:33.599
So it's like I would imagine like broader market appeal.
00:21:33.920 --> 00:21:44.480
But yeah, I mean like ultimately, I think the way modal set up how you acquire GPU, I think this is the correct way to do it because it's very incentive aligned with your customers to charge for use.
00:21:44.640 --> 00:21:47.759
Like it's an honest business, basically the way I would frame it.
00:21:47.920 --> 00:21:55.920
Uh and I think and I think paying for access is dishonest because it it's uh it I I just yeah, like because whether you use it or not, it's like it's not great.
00:21:56.000 --> 00:21:56.480
I don't like it.
00:21:56.640 --> 00:21:59.759
Yeah, yeah, it's like if I could like my biggest thing with this.
00:22:00.160 --> 00:22:04.559
Nvidia right now is like I need like I have this stream of like free tokens from Brev.
00:22:04.799 --> 00:22:15.599
I just they if they could just make a serverless product, even if it's like a worse version of modal, then like I would not need to ask for as many tokens and like it could be so much more token efficient.
00:22:15.839 --> 00:22:21.599
By the way, I refreshed the uh I refreshed the tokens for the the profiling service again too.
00:22:21.839 --> 00:22:22.400
Thank you, sir.
00:22:22.480 --> 00:22:22.559
Yeah.
00:22:23.519 --> 00:22:29.039
Um so I think there's enough in there to run through to run through the end of July.
00:22:29.119 --> 00:22:32.160
But so when do how long does the optimizer contest run?
00:22:32.559 --> 00:22:36.319
The optimizer con we we don't know yet, but we're thinking of making it run for a month.
00:22:36.559 --> 00:22:36.880
Okay.
00:22:37.039 --> 00:22:41.279
I'm just gonna keep refreshing the tokens for the profiling service until somebody tells me to stop.
00:22:42.079 --> 00:22:43.119
Okay, that's that's wonderful.
00:22:43.359 --> 00:22:43.680
Yeah.
00:22:43.839 --> 00:22:44.160
Yeah.
00:22:45.039 --> 00:22:50.240
I think I think we didn't get the chance to talk much about the optimizer competition, but I think it's it's pretty it's pretty sick, actually.
00:22:51.039 --> 00:22:52.079
I'll tell you more about it next time.
00:22:52.319 --> 00:22:53.440
Uh yeah, I'm very excited.
00:22:53.519 --> 00:22:54.799
I I saw I started looking into it.
00:22:54.880 --> 00:22:56.000
Yeah, it looks like it's gonna be very cool.
00:22:56.799 --> 00:22:58.079
Do you mind if I just tell you about it for a bit?
00:22:58.240 --> 00:22:58.559
Yeah, sure.
00:22:58.960 --> 00:23:00.079
I think just okay.
00:23:00.240 --> 00:23:09.839
So so the optimizer competition, like like the sort of main idea was that we we've been very interested in picking tasks where transformers obviously fall flat on their face.
00:23:10.000 --> 00:23:12.480
And so the task we picked is one that involves depth.
00:23:12.559 --> 00:23:14.799
It's basically like repeated squaring and modulo.
00:23:15.119 --> 00:23:22.799
You try to make any like the largest transformers run well on this task, it's gonna run flat on, it's gonna fall flat on his face, and it's gonna try and cheat the task.
00:23:22.960 --> 00:23:40.480
And intrinsically, the only models that have like the right inductive biases to solve these problems are those that include like basically some form of looping or adaptive depth, or like basically and they basically involve like scaling training time compute to sort of like solve the task.
00:23:40.640 --> 00:23:53.279
And these are very exciting because if you sort of if you maybe have a brief physics background, like a lot of these, like anything that involves looping becomes like basically it turns into mush, like basically it just goes either to zeros or it goes to nan.
00:23:53.359 --> 00:23:55.039
Like the system has become very unstable.
00:23:55.200 --> 00:24:04.960
And so what we're very excited about here is we want people to come up with interesting like model architecture ideas to solve this very like simply simply formulated but very nasty problem.
00:24:05.119 --> 00:24:13.039
But because these architectures are quite unstable, they will also need to come up with sophisticated optimizers to solve this problem.
00:24:13.200 --> 00:24:16.960
And the sort of user experience we borrowed is the exact same as kernelbot.
00:24:17.039 --> 00:24:21.839
It's like you submit a single Python file and we run all of these evaluations for you.
00:24:22.160 --> 00:24:27.279
Right now we're basically going through like a trial period where people are reward hacking us to death, which is okay.
00:24:27.440 --> 00:24:29.359
It helps us like have the rules be more robust.
00:24:29.519 --> 00:24:31.279
We're gonna launch for real this Thursday.
00:24:31.359 --> 00:24:37.200
And the competition will be open for about a month, and we're kind of excited to sort of see who comes out of the woodworks here.
00:24:37.519 --> 00:24:54.000
What's been interesting is that like there's this like you know, there's like it's sort of this interesting intersection between people that do complexity theory and model architecture, where they will write very specific proofs as to things like, hey, like a diffusion model can't actually learn long sequence like video generation.
00:24:54.160 --> 00:24:58.480
And I I find this like to be really, really compelling, and I just can't wait to see what people do.
00:24:58.720 --> 00:25:07.680
Generally, for me, I think GPU mode really becomes healthier if people are aware of like frontier level problems, and this is like a frontier level problem that I think many, many people are interested in.
00:25:07.920 --> 00:25:10.400
So I think we made it very agent-friendly.
00:25:10.480 --> 00:25:11.759
The GPUs are all free.
00:25:11.920 --> 00:25:14.720
You get an hour of training time basically a day.
00:25:14.880 --> 00:25:16.400
So, you know, please have fun.
00:25:16.480 --> 00:25:23.119
The the website is like one layer deeper.ai, and I think it's gonna be like one of our most interesting challenges yet, and I'm very excited about it.
00:25:23.359 --> 00:25:28.720
Yeah, no, I think it's I mean, I not be not being not being an ML person.
00:25:28.960 --> 00:25:39.200
I uh I I like being more a kernel person than an ML person, it's a little out of my depth, but from what I've read up on it, it it looks like it's a very, a very cool problem.
00:25:39.279 --> 00:25:44.079
And um, I'm very excited to see what people will uh will will be able to do with it.
00:25:44.160 --> 00:25:53.119
And it also seems like it's something that is probably not in the training set and is like a frontier problem that we'll really be able to see what people can do with auto research for this.
00:25:53.279 --> 00:25:53.599
Yeah.
00:25:53.759 --> 00:25:54.240
Yeah.
00:25:54.559 --> 00:26:01.839
Alright, one one last one last question, and it hopefully will be a short one, because I know Bryce, you need to go to sleep because you're over in the Europe's.
00:26:02.400 --> 00:26:13.359
One of my it's not really a hobby, but something I've been doing over the last several months is I I Well, the details don't matter, but I listened to a lot of No no no, tell us the details.
00:26:13.599 --> 00:26:14.079
The details?
00:26:14.160 --> 00:26:15.119
Alright, very quickly.
00:26:15.279 --> 00:26:23.119
I mean, I have a a vibe-coded podcast player called Pod God, which uh it was gonna get released in the next couple weeks.
00:26:23.279 --> 00:26:26.319
But I've been using it since the beginning of February, and it's phenomenal.
00:26:26.480 --> 00:26:30.400
One of the things it allows you to do is it allows you to follow personalities.
00:26:30.480 --> 00:26:38.000
So there's something called the personality meta podcast, which is just a curated feed of every single time that person goes on a different podcast.
00:26:38.079 --> 00:26:43.920
Because there are some people that I care about and I want to listen to any time they're being interviewed, but I don't want to have to go and listen to example.
00:26:44.240 --> 00:26:46.160
The person that I'm about to mention is Boris Cherney.
00:26:46.240 --> 00:26:48.880
Like, but the thing is is one No idea who that is.
00:26:49.200 --> 00:26:49.359
What?
00:26:49.440 --> 00:26:50.960
The creator of Claude Code?
00:26:51.200 --> 00:26:52.079
No, no idea.
00:26:52.319 --> 00:26:52.960
That's very sad.
00:26:53.119 --> 00:27:04.079
He's like, he's honestly would be my number one pick to have as like a future guest because every every time he is interviewed, I think he has just like amazing takes.
00:27:04.240 --> 00:27:16.000
But anyway, so like there's a bunch of podcasts that one I don't want to subscribe to, like there's like Sequoia has this like kind of there's a bunch of these like VC money podcasts, but every once in a while they have someone interesting on.
00:27:16.079 --> 00:27:21.440
So like I don't want to have to subscribe to this and then like basically just like download the thing and then delete it.
00:27:21.599 --> 00:27:22.480
I just want the interviews.
00:27:22.640 --> 00:27:25.359
Anyways, that's I that's not long story short, that's long story long.
00:27:25.599 --> 00:27:34.799
Whenever you listen to Boris Cherney, like one out of every two podcasts, he gets like book recommendations or he gives book recommendations, and they've been phenomenal.
00:27:35.039 --> 00:27:45.759
The first one that he recommended, he actually mentioned in the Peterman podcast, which is like a former, I think, Instagram or Facebook guy, and he's just been it's it's great interviews.
00:27:46.000 --> 00:27:54.079
He interviewed uh Boris Cherney, and then he mentions that during the interview for Anthropic, he ended up talking about some book by Greg Egan.
00:27:54.319 --> 00:27:57.839
You know, this is back in like Claude, I don't know, 3.7, 4.0 days.
00:27:57.920 --> 00:28:02.319
I immediately go to like whatever ChatGPT ask, like, what book is he referring to?
00:28:02.559 --> 00:28:08.640
And it ended up being Permutation City, and that is like part of a trilogy that has Diaspora and Shield's Child.
00:28:08.720 --> 00:28:15.119
He on a more recent podcast mentioned Werner Vinge's A Fire Upon the Deep, which is in the Zones of Thought trilogy.
00:28:15.279 --> 00:28:17.039
I'm like one down, two to go.
00:28:17.279 --> 00:28:20.640
I bring all of this up, where's the question, to ask you?
00:28:20.799 --> 00:28:31.599
So you wrote the Robot Overlord manual, and you're the first person actually that we've interviewed on this podcast, which is a little bit surprising, that is as high on AI as Bryce and I.
00:28:31.920 --> 00:28:38.480
We we've interviewed some folks over the last little bit, but not like not people that are sipping the Kool-Aid as much as we are.
00:28:38.720 --> 00:28:44.240
And so uh the the answer to this question may be I don't have any recommendations.
00:28:44.480 --> 00:28:48.400
My question is, do you have any book recommendations in general?
00:28:48.640 --> 00:29:00.640
And on top of that, specifically, I would love to know if you have like any recommendations in the space of you know future AI, like good these kinds of good reads that like people like Boris Cherney recommend recommending.
00:29:00.799 --> 00:29:01.759
So over to you.
00:29:02.160 --> 00:29:04.720
Yeah, like I mean, I I love reading.
00:29:04.799 --> 00:29:05.920
Like this, these are all my books.
00:29:06.160 --> 00:29:11.599
The listener can't see, but there's also like a bookshelf in the background with what looks like at least a couple hundred books there.
00:29:11.680 --> 00:29:22.880
So the the there's like a couple of other hundred that I've given away over the years, but yeah, I mean, I I like I actually like I I do I I had a tweet about this, which is like my ten favorite books.
00:29:23.119 --> 00:29:23.599
Let me see.
00:29:23.839 --> 00:29:29.119
So I would say on the AI, like, okay, so these are these are the books that were most influential to me.
00:29:29.359 --> 00:29:30.160
There's ten of them.
00:29:30.319 --> 00:29:30.960
So let me see.
00:29:31.039 --> 00:29:32.480
Maybe I can pick two for you.
00:29:33.039 --> 00:29:36.720
Well, we'll include we'll we'll include uh the link to the tweet so you can get all ten.
00:29:36.880 --> 00:29:39.119
You can and feel free to just read all ten off if you want to.
00:29:39.440 --> 00:29:41.440
Okay, like it's a more interesting question.
00:29:41.519 --> 00:29:42.319
Okay, the all ten.
00:29:42.559 --> 00:29:48.160
So my like the book that I think was like the most fun to me was a book called The Journey to the Ants.
00:29:48.400 --> 00:29:51.599
So it's it's it's it's a book about like ant-based intelligence.
00:29:51.839 --> 00:29:57.440
And the reason why I found it very interesting is because like the the kinds of shit ants can do is pretty wild.
00:29:57.519 --> 00:30:05.200
Like, for instance, like you can have like the mother ant, the queen, like control the gender ratio of a population, control like people's tasks.
00:30:05.279 --> 00:30:13.440
Like you can be a worker versus a warrior elastically based on the demands of the colony, and they all sort of spread this message between each other to sort of dictate.
00:30:13.599 --> 00:30:20.960
This is very contrary to how like human societies operate, where we're like, oh, I'm a colonel guy, and I will never not do kernels, like we're very inelastic.
00:30:21.119 --> 00:30:26.160
So it's like pretty remarkable for me to see an example of um of societies that were not this way.
00:30:26.240 --> 00:30:27.599
And I think this book is wonderful.
00:30:27.839 --> 00:30:30.079
Another book I really love is visual group theory.
00:30:30.160 --> 00:30:33.519
So visual group theory, and uh, maybe I would add Burns Euclid here.
00:30:33.680 --> 00:30:42.960
What I found really lovely about these books is that like any anything that's algebraic tends to be like very dry and like kind of unintuitive to read because you just see look at these long formulas.
00:30:43.119 --> 00:30:49.920
Both of these books are pretty like beautiful, they almost feel like pieces of art where people are sort of like going through like these arguments like very rigorously.
00:30:50.079 --> 00:31:01.200
And so for me, they really made me appreciate like the idea of a beautiful explanation, which is like typically when I think of like my job, I don't necessarily think of it as like producing like beautiful code or fast code.
00:31:01.359 --> 00:31:04.640
I think of it as being able to produce a good explanation for why the code is fast.
00:31:04.720 --> 00:31:06.960
Like that's kind of almost like my love language.
00:31:07.200 --> 00:31:09.599
Another book I've adored is The Theory of Fun.
00:31:09.680 --> 00:31:13.920
So the Theory of Fun is a book on why people like playing games.
00:31:14.079 --> 00:31:18.880
It's sort of like the sort of the main thesis of the book is that playing is basically the way we learn.
00:31:19.039 --> 00:31:25.039
So I've carried that a lot into my projects where I've tried to make a lot of the things we've done with GPU mode fun.
00:31:25.200 --> 00:31:28.319
And as a result, people end up learning more and might find it more valuable.
00:31:28.400 --> 00:31:33.920
So a lot of what I learned with GPU mode was like basically early days of playing World of Warcraft and Warcraft 3.
00:31:34.000 --> 00:31:36.319
So like a lot of the lessons, funnel enough, do apply.
00:31:36.559 --> 00:31:46.880
I would say two technical books that I've I think I'm gonna skip over a couple of books, but two technical books that I think I highly recommend to everyone are uh game programming patterns and working with legacy code.
00:31:47.039 --> 00:31:50.960
Working with legacy code often surprises people as one of my favorite books of all time.
00:31:51.039 --> 00:31:56.079
But I found it really wonderful when I'm just like plopped into like a monstrous code base.
00:31:56.240 --> 00:31:58.559
How do you sort of like quickly make sense of it?
00:31:58.640 --> 00:32:03.680
And like what are good design patterns to make codes like simpler and easier to reason about?
00:32:03.839 --> 00:32:13.519
I think a lot of like my contributions in the in past projects has been like going into a complicated code base and like understanding it and then figuring out how to develop features more quickly on top.
00:32:13.680 --> 00:32:17.839
I think a lot of these ideas will make a comeback if we're like basically in mountains of slop.
00:32:18.000 --> 00:32:22.240
Maybe yet another book that I've really enjoyed, maybe okay, maybe I'll give you two more.
00:32:22.480 --> 00:32:25.200
Like Game Theoretic Foundations for Probability and Finance.
00:32:25.279 --> 00:32:27.680
It's not a finance book, even though it's in the title.
00:32:27.839 --> 00:32:33.680
It's a book about rewriting all the axiom of probability theory with game theory in a way that makes them a lot more interpretable.
00:32:33.839 --> 00:32:38.559
And so I've also really enjoyed this kind of book because I've always enjoyed like more wacky math.
00:32:38.640 --> 00:32:40.079
It just can help you build intuition.
00:32:40.160 --> 00:32:43.119
And if you didn't like one notation, maybe you'll appreciate another.
00:32:43.359 --> 00:32:46.880
The last one is the structure and interpretation of classical mechanics.
00:32:47.039 --> 00:32:50.799
I'm sure many of you might have read the structure interpretation of like computer programs.
00:32:51.359 --> 00:32:52.640
That's what I thought I was gonna say.
00:32:52.960 --> 00:33:06.720
I I think this book is better because it basically goes over like it's writes short programs for physics written in Lisp, and you basically end up with nature as like a short series of like self-modifying Lisp programs.
00:33:06.960 --> 00:33:17.440
It's one of those like books that like blow your mind the first time you read it, but like again, like as we get into the age of auto research and architecture research, I suspect a lot of these ideas are are making a big comeback.
00:33:17.599 --> 00:33:28.559
So these are things that I think are are you know there's a lot more, but like maybe you know, berserk and necronomicon, maybe with some pieces of fiction that I think are just like absolutely gorgeous.
00:33:28.799 --> 00:33:36.400
I I've always been a sucker for like there's some piece of knowledge that if you acquire it, I don't know, you'll blow up because you'll be rendered mad.
00:33:36.640 --> 00:33:42.960
Like I don't know why, but it's like one of those things that I've always like I've been like a sucker for this kind of like fantasy, fantasy work.
00:33:43.119 --> 00:33:44.720
But yeah, all these books are phenomenal.
00:33:44.799 --> 00:33:47.839
I'll share the full list if you like with you, like in a few minutes.
00:33:48.079 --> 00:33:51.680
But yeah, I I think these are sufficiently out of distribution to give you better ideas.
00:33:52.000 --> 00:33:57.039
I mean, that was I feel bad we haven't asked our other past guests because that was that was uh amazing.
00:33:57.200 --> 00:34:02.480
Even better than not that I you know had expectations for the quality of your answer, but uh even better than I expected.
00:34:02.640 --> 00:34:03.279
Go ahead, Bryce.
00:34:03.599 --> 00:34:11.760
I I have a I have a related question, which is it sound like it sounds like you have been a person who's played a lot of video games in the past.
00:34:12.000 --> 00:34:14.480
And I'm wondering if you have recommendations for video games.
00:34:14.880 --> 00:34:16.239
Uh let me see.
00:34:16.559 --> 00:34:19.760
Um so what are the greatest games I've played?
00:34:19.920 --> 00:34:27.760
So so the one of the issues with like like I think the greatest game I've ever played is probably Dota 2, but I would not recommend you play it because it just will take over your life.
00:34:27.920 --> 00:34:40.320
So I think it's like an optimizer's dream in some sense, and it teaches you how to never get mad at anything because like it's very formative to be insulted at in like five different languages and still manage how to win a game.
00:34:40.559 --> 00:34:41.679
So it's very formative.
00:34:41.840 --> 00:35:00.480
But but I would say, like, I mean, if I'm thinking like games that like left like a deep impression on me, I would say recently, I I would probably say it was like Expedition 33 as like sort of a story-based game, like really left like a deep impression on me, and I was like a big sucker for like role-playing games like growing up, you know, all the way ranging from like Pokemon to like Chrono Trigger.
00:35:00.639 --> 00:35:03.599
As a result, I've also really enjoyed Final Fantasy XVI recently.
00:35:03.679 --> 00:35:06.559
I've been playing Final Fantasy 7 as well, which I really enjoy.
00:35:06.719 --> 00:35:14.320
But yeah, otherwise, like these days, like I mostly spent a lot of time playing Slay the Spire, but I also feel like I spent too much time doing it.
00:35:14.400 --> 00:35:18.800
So if you give me an optimization puzzle, usually I can get obsessed with it.
00:35:18.960 --> 00:35:25.039
If you give me something that's fast and reflex-based, like Fury or Katana Zero, I'm also like a big sucker for those.
00:35:25.119 --> 00:35:27.119
Like I enjoy sort of this like flow feeling.
00:35:27.280 --> 00:35:30.960
And I think these are all games that will give you like a major, major uh sense of flow.
00:35:31.039 --> 00:35:33.039
So they're all treats if you haven't played them.
00:35:33.360 --> 00:35:41.840
So my my actually my my oldest friend, the person that I've had the longest contact with, is somebody I've never actually met in person.
00:35:42.000 --> 00:35:47.679
And he and I, I used to play a lot of MUDs, text-based games when I was a kid, and he and I played MUDs together.
00:35:47.920 --> 00:35:51.840
And in particular, my favorite type of MUD was a roleplay-intensive mud.
00:35:52.079 --> 00:35:55.840
So that's where everybody has to be in character at all times.
00:35:56.000 --> 00:36:08.639
I primarily played either Star Wars or Lord of the Rings themed MUDs, and these were all like community-run multiplayer games, and nobody was really concerned about the fact that like Star Wars and Lord of the Rings were IP that somebody else owned.
00:36:09.039 --> 00:36:14.159
But, anyways, him and I were having this conversation the other day where it's a it's a text-based game.
00:36:14.320 --> 00:36:25.119
Like, it is you could just like hook up, I'm sure, like a frontier model to it, and you know, the problem that these games have today is like, you know, the NPCs are not particularly good and there's just not a lot of players.
00:36:25.280 --> 00:36:29.280
And him and I were talking about like, well, we could just make as many players as we want.
00:36:29.360 --> 00:36:32.800
You know, you could just like you wouldn't even need to code them into the game.
00:36:32.960 --> 00:36:40.400
You could just connect an existing agent to an existing mud and just have it interact the game as it would any other player character.
00:36:40.559 --> 00:36:43.920
And I just like I I can't haven't been able to get this idea out of my head.
00:36:44.000 --> 00:36:52.079
And if I had if I had more infinite inference, I would go build my like AI mud game, and I'm sure I'd have a blast with it.
00:36:52.320 --> 00:37:01.199
I I I think we should try it out because I I think one hybrid idea I'd be very optimistic about would be something like uh like an in-person Dungeons and Dragons game.
00:37:01.360 --> 00:37:01.760
Oh yes.
00:37:02.000 --> 00:37:12.079
Where like the like where the DM can basically choose to like basically where there's a DM who's constantly prompting to try and anticipate where like things might go.
00:37:12.239 --> 00:37:13.760
That I'm I'm really optimistic about.
00:37:13.920 --> 00:37:21.840
But but I think a lot of the I think issues I foresee with current alums is like post-compaction, maybe they'll lose everything, like they'll just have no idea what the fuck's going on with the game.
00:37:22.239 --> 00:37:26.320
We should do a whole we we should we should talk some other time about my thoughts on compaction.
00:37:26.480 --> 00:37:29.119
I think sometimes compaction helps and sometimes it hurts.
00:37:29.360 --> 00:37:30.320
In the short.
00:37:30.480 --> 00:37:30.880
Yeah.
00:37:31.119 --> 00:37:39.039
The thing that the thing that's most valuable that I get that happens at compaction is most harnesses on compaction will reread skills.
00:37:39.199 --> 00:37:47.199
And that can sometimes help because then it's like a reminder of like, were you following, were you following like the prompts and skills that you were originally given?
00:37:47.360 --> 00:37:51.760
Anyways, I should go, it's very it's getting very late here, and we should let Mark get going.
00:37:52.320 --> 00:37:53.119
Good night, Bryce.
00:37:53.199 --> 00:37:54.639
And then thank you for inviting me, Bryce.
00:37:55.760 --> 00:37:57.039
It was a pleasure to chat with you.
00:37:57.360 --> 00:38:06.880
Be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions.
00:38:07.280 --> 00:38:07.840
Thanks for listening.
00:38:07.920 --> 00:38:09.519
We hope you enjoyed and have a great day.
00:38:10.400 --> 00:38:15.519
Low quality, high quality, that is the tagline of our podcast.
00:38:15.840 --> 00:38:16.719
That's not the tagline.
00:38:16.960 --> 00:38:19.760
Our tagline is chaos with sprinkles of information.
00:00:00.160 --> 00:00:12.000
Well, like, I think my take on events these days is just like the agentic era has made open-ended judging a bit cooked because it's like, how do you tell whether something is amazing versus not?
00:00:12.160 --> 00:00:14.480
If it's just like if Claude one-shot it.
00:00:14.800 --> 00:00:26.160
But if someone did a hill climb on a task that we can all collectively agree on is hard and made like a breakthrough, that's better than what I would do myself as like an advanced user.
00:00:26.399 --> 00:00:29.359
Like, if they can just do that during the hackathon, like I'm very impressed.
00:00:29.440 --> 00:00:32.960
And so, like, it just doesn't matter how they did it, it would matter is the output.
00:00:33.119 --> 00:00:40.960
So, I think that I'm very open to, and I think it's gonna become a lot more interesting doing it with open models this year, because like the open models are good finally at these tests.
00:00:41.119 --> 00:00:43.840
I think it's it looks like they've been RL'd on kernel data.
00:00:43.920 --> 00:00:47.840
Yeah, and so I think it's gonna be a lot more exciting than it was like a year ago.
00:00:57.280 --> 00:01:02.799
Welcome to ADSP the podcast, episode 304, recorded on July 20th, 2026.
00:01:02.880 --> 00:01:08.719
My name is Connor, and today with my co-host Bryce, we finish part three of our three-part conversation with Mark Seraphim.
00:01:08.799 --> 00:01:31.519
In this episode, we continue our conversation about GPU mode, auto research, Mark's favorite books, and more.7, which feels like a long time ago, was like auto research on Cubs a histogram.
00:01:31.680 --> 00:01:40.799
And that auto research, I didn't give it any steering, and it basically that auto research popped out six different relatively novel algorithms.
00:01:41.040 --> 00:01:47.840
Two of them were like kind of existing, a twist on existing techniques, but the rest were like really kind of I think novel.
00:01:48.000 --> 00:01:51.439
They did build on like existing things from literature.
00:01:51.519 --> 00:01:55.439
This is the SMIM cache that Connor that we talked about in the last episode.
00:01:55.599 --> 00:01:59.920
Like, certainly like caches, hash tables are like a thing that are in literature.
00:02:00.079 --> 00:02:06.400
And I think they're you know, histograms are very similar to to to to to a hash table in some way.
00:02:06.560 --> 00:02:23.280
So, you know, maybe there is actually some literature out there on this particular technique, but it really did feel like it was it was a novel, it wasn't a completely novel idea, but it was a novel combination of ideas, and there was no chance I would have ever come up with that idea on my own.
00:02:23.520 --> 00:02:23.840
Yeah.
00:02:24.159 --> 00:02:31.759
I I I guess Bryce, I'm curious to ask you on the specific example, like what was what was it like trying to merge it back into Cub?
00:02:31.840 --> 00:02:33.039
I don't know because I don't know if you did or something.
00:02:33.439 --> 00:02:34.800
I'm in the process of explaining that right now.
00:02:35.120 --> 00:02:46.960
So so the the interesting thing with that, with the cub histogram, was the key like auto research run was done in like I think the last auto research run I did for that was like end of May.
00:02:47.120 --> 00:02:52.879
And so I I came out of that with like the the the candidate, you know, code in kernels.
00:02:53.039 --> 00:03:01.599
And what I've had to do since then is I was running auto research with Cub's existing benchmark, which handles which has basically like two different distributions.
00:03:02.000 --> 00:03:08.080
And you know, I knew I needed to do a much wider input characterization.
00:03:08.319 --> 00:03:14.159
So I constructed a whole lot more inputs, I constructed adversarial inputs.
00:03:14.400 --> 00:03:22.240
Me and the agent learned that in fact the way that Cobb generated its histogram inputs for the benchmark have always been broken since the beginning of time.
00:03:22.639 --> 00:03:31.120
And the the the full like input characterization r takes like 30 hours on a B200 to like really get good data.
00:03:31.280 --> 00:03:39.840
And with Cobb, there's the constraint that we really can't ship six different algorithms, one for this particular dynamic shape, one for that particular dynamic shape.
00:03:40.000 --> 00:03:43.120
And and so I've done more like manual work.
00:03:43.439 --> 00:03:44.240
How come, by the way?
00:03:44.639 --> 00:03:47.120
Like, was it because it's sort of uh binary size?
00:03:49.759 --> 00:03:50.000
Yeah.
00:03:50.159 --> 00:03:57.599
So so I needed to really to pick like I was gonna have to do go with two algorithms, one for the small bin case and one for the large bin case.
00:03:57.680 --> 00:04:04.639
And there are a couple of like additional post-auto research optimizations that me and the agent did.
00:04:04.879 --> 00:04:18.959
And the the those things were basically they weren't new algorithms, it was just like the right combination of things, like combining the SMEM cache with the existing privatized, you know, histograms and global memory.
00:04:19.120 --> 00:04:32.720
And then like at one point we we turned off warp coalescing before you hit the SMEM cache was like a big loss, but doing coalescing on spills from the cache was a big win.
00:04:32.879 --> 00:04:38.399
And like I just got it sort of nailed down, and I I've looked at the code a few times throughout this process.
00:04:38.639 --> 00:04:40.160
I think there's two things I can try.
00:04:40.240 --> 00:04:50.399
The first thing is I can try to run my simplify scale, and the simplify scale is basically minimize the bytes of code difference, of non-common code difference from upstream.
00:04:50.959 --> 00:04:59.279
The alternative I think would be to identify like a series of commits of like, okay, this is like the here's like four PRs.
00:04:59.360 --> 00:05:06.720
We could break this down into four PRs, and let's just let's just do the four PRs as like from fresh.
00:05:06.800 --> 00:05:16.160
Like, don't try to take the existing code and like shape reshape it into a PR, but just like using the existing code as a reference point, generate the PRs.
00:05:16.319 --> 00:05:21.360
And I think those PRs are gonna go up uh sometime this month.
00:05:21.600 --> 00:05:26.319
And my guess is gonna be that it'll take two to two months or so to merge.
00:05:26.480 --> 00:05:29.519
I think it's in a shape now where it will be acceptable.
00:05:29.839 --> 00:05:33.759
The yeah, I mean the current the current code is it's a little bit of a mess.
00:05:33.920 --> 00:05:38.560
Like the biggest problem, the bit the single biggest problem was not following conventions in the code base.
00:05:38.639 --> 00:05:42.399
So like there was a convention for how tuning got done and it wasn't followed.
00:05:42.560 --> 00:05:47.759
And the a good part of it was things that the model never could have known.
00:05:48.079 --> 00:06:02.240
For example, like, oh, you know, this is the way that we used to do the tuning in this part of code, but you know, there's a new there's a new policy that's not documented anywhere for how we would do this, but the PR probably won't be accepted if it doesn't follow this new convention.
00:06:02.480 --> 00:06:18.079
Or like there's you know, there's this convention that we if you want to have an algorithm that dynamically dispatches to two, if you want to have a cub routine that can dynamically dispatches to two algorithms, like you need we need to get a special sign-off from someone.
00:06:18.160 --> 00:06:19.839
And that's not documented in the code.
00:06:19.920 --> 00:06:22.000
So how could the model have ever known that?
00:06:22.319 --> 00:06:23.279
This is this is interesting.
00:06:23.360 --> 00:06:42.639
Like basically, the this sort of makes sense if you sort of assume, like this is my guess, by the way, that like the cub team has been working on it for a while, that the original team hasn't disbanded much, and so a lot of these sort of like unwritten rules are actually in people's heads, and they all have internalized them, and there's no reason to write docs because maybe the team doesn't grow very quickly.
00:06:42.800 --> 00:06:47.680
But then, like in an age where you're forcing them to verbalize everything, they're like, Oh, goddammit, like this is so much work.
00:06:47.759 --> 00:06:49.600
I have to write docs now and I have to write tests.
00:06:49.759 --> 00:06:51.199
Yeah, yeah, I see interesting.
00:06:51.360 --> 00:07:10.800
Yeah, but I think it will be very interesting to see whether we can get the get this landed because Cobb in particular is a really high barrier to land because you need to be good across a really wide range of inputs, and you know, you you you you can't like ignore edge cases.
00:07:10.959 --> 00:07:25.360
And so it'll be I I I I'm hopeful because I do think it's a it's like a pretty big win uh for the for the cases that it like now I've gotten it to the point where essentially there's there's no adversarial inputs that are really bad for it.
00:07:25.439 --> 00:07:26.399
There may be one left.
00:07:26.480 --> 00:07:27.439
I gotta try that.
00:07:27.600 --> 00:07:28.000
Yeah.
00:07:28.160 --> 00:07:37.920
Have you have you have you done any auto research work that has landed upstream PRs in like big projects that have a high bar for quality?
00:07:38.319 --> 00:07:39.199
Personally, no.
00:07:39.360 --> 00:07:48.560
Like I think like around the time I left Meta, there was like I think an R like a fused RMS norm, which was like the first AI generated kernel that was merged, but that was like not my work.
00:07:48.639 --> 00:07:52.240
That was like Laura Laura's work on the kernel agent team.
00:07:52.480 --> 00:07:59.839
I do remember though, the main experience we had at the time was that like the kernel was quite long, and so I think it was important to deslopify it.
00:08:00.000 --> 00:08:10.800
I think there was sort of sometimes a bit of an ideological debate between like what's the point of deslopifying it, like just don't read it, to which like the PyTorch core maintainers were like, What are you talking about?
00:08:10.879 --> 00:08:14.079
Because this is maintenance debt, I need to understand every single line.
00:08:14.240 --> 00:08:24.240
I'm not sure which of those two stances is correct because like some of those PyTorch people felt very strongly about these things, are just like no longer working on the team, and so like I actually don't know.
00:08:24.319 --> 00:08:29.040
Like, like, but kind of like one thing that stands out to me at least was like this one.
00:08:29.120 --> 00:08:39.679
Um I think like this is sort of a paraphrase from a talk by Terence Stowl, so it's not exactly what he said, but it's like what I came away with, which is that like he was sort of talking about like I mean, let me just not just talk about kernels.
00:08:39.759 --> 00:08:54.879
Like, basically, like my my hypothesis is that typically by the time you're done writing a complicated kernel, you already have simplified it multiple times because as you're writing something, you're like evolving it, you're fixing it, and you've also like understood it.
00:08:55.039 --> 00:09:04.639
Like by the end of it, if you sort of like artisanally crafted something, basically uh generating, understanding, simplifying is like in some sense the same act.
00:09:04.799 --> 00:09:15.600
But like if we're in an age where people are only doing generation, it becomes like I think a bit of a problem because like let's say your kernel introduces a race condition, but it's like non-deterministic.
00:09:15.840 --> 00:09:18.320
Well, you know, that really sucks for a model training run.
00:09:18.399 --> 00:09:23.759
But maybe it's forgivable for like a tiny little inference service that like runs some sort of speculative optimization.
00:09:23.919 --> 00:09:29.360
So there's like a risk-reward thing where like maybe there's some areas where it need to be really paranoid and this is not acceptable.
00:09:29.519 --> 00:09:42.320
But generally, like when I broadly look at this, I just don't buy the people don't care because you know, as we talked at the beginning of this episode on like if people are freaked out about non-determinism, like boy are they gonna be freaked out about like straight up incorrectness.
00:09:42.399 --> 00:09:44.799
Um bugs that are like non-deterministic as well.
00:09:44.960 --> 00:09:49.120
Like, I think this is bound to cause conspiracy theories, uh, especially at scale, right?
00:09:49.200 --> 00:09:51.519
Like, if let's say 1% of your users only see something.
00:09:51.600 --> 00:09:51.919
Yeah, yeah.
00:09:52.399 --> 00:09:56.000
So, my broad take is like I think we should solve the hard problem.
00:09:56.080 --> 00:09:59.360
I don't think the pro the solution is people are reading the code.
00:09:59.519 --> 00:10:04.879
Like that that to me feels like uh like no, like, because it's let's say the QR problem.
00:10:05.039 --> 00:10:13.840
You know, at first I was like, hey, I'm gonna sort of synthesize all the kernels by all 170 users, and then I'm gonna try to come up with like a beautiful QR kernel.
00:10:14.000 --> 00:10:19.840
Uh, you know, like I've I've yet to do this, it feels onerous every time I think about like sitting down and doing this exercise.
00:10:20.080 --> 00:10:26.720
But at the same time, if I just ask it like Godex and I'm like, hey, Godex, go cook, like, look at all these kernels and tell me what comes up.
00:10:26.879 --> 00:10:28.240
In some sense, I don't even know.
00:10:28.320 --> 00:10:29.279
Did it read everything?
00:10:29.440 --> 00:10:31.519
Like, did it like really understand like everything?
00:10:31.679 --> 00:10:33.200
Did it capture the key ideas?
00:10:33.360 --> 00:10:40.320
Like, maybe there's a slow kernel that had more elegant ideas, will it really capture them or will it really just prioritize the first like dozen kernels that are fast?
00:10:40.480 --> 00:10:41.279
Like, I'm not sure.
00:10:41.440 --> 00:10:45.039
Like, in some sense, like I don't know what the model is doing, like I can't inspect it.
00:10:45.200 --> 00:10:47.600
Like, so it does feel like it's an alien artifact.
00:10:47.679 --> 00:10:48.879
I tell it go cook.
00:10:49.120 --> 00:10:59.360
But my hope is that like we can sort of figure out like how to come up with beautiful kernels from a large sum of fast but slop kernels.
00:10:59.600 --> 00:11:07.440
I just think like it's maybe in the beginning there's a bit more manual than we'd like, but I'm I'm fairly confident we can figure out like good tooling for for problems like this.
00:11:07.759 --> 00:11:24.720
So, first of all, I I gotta say, I I f I find I find it very interesting the proposition that a like a uh a race condition, slightly buggy kernel that's like good 99% of the time but and really fast, maybe it's good enough in some cases.
00:11:24.799 --> 00:11:26.000
I hadn't considered that before.
00:11:26.159 --> 00:11:27.120
Maybe maybe it is.
00:11:27.360 --> 00:11:28.399
Um for inferences.
00:11:29.279 --> 00:11:35.440
So my goal in auto research here is to think about what's the process that I as a human would do.
00:11:35.600 --> 00:11:40.799
And I think that both for humans and agents, it's very hard to optimize for multiple things at once.
00:11:40.960 --> 00:11:52.480
And every time that I've had to myself do like deep like kernel performance work, it's almost always where I have a phase where I'm trying to reach speed of light.
00:11:52.559 --> 00:12:05.759
And during that phase, you know, I'm adding restrict places, I'm adding in assumes, I'm adding in this and that, I'm rebenchmarking, I'm trying things, I'm if-defing stuff out, you know, I'm I'm I've got some dead code branches, etc.
00:12:06.159 --> 00:12:09.039
And then like, and then I reach performance.
00:12:09.120 --> 00:12:10.639
And I'm like, ah, great.
00:12:10.799 --> 00:12:11.840
This is the right thing.
00:12:12.000 --> 00:12:22.799
And then I've got this mess, and then what I do is I go back and I'm like, okay, okay, I'm gonna delete all the if-defs, I'm gonna, I'm gonna, you know, take out those like if I take out this assume, is it fine?
00:12:23.039 --> 00:12:26.879
Okay, and you know, and then I've got I've always got like four versions of this code.
00:12:27.039 --> 00:12:31.759
And it's like, okay, well, like I can get I can delete this variant that I don't need anymore, and I can delete that variant.
00:12:31.919 --> 00:12:43.120
And it's like I always do this process where I've first spent a bunch of time trying to find speed of light, and then once you've found speed of light, then like you have an Oracle answer.
00:12:43.279 --> 00:12:46.960
And then I can try to write better, cleaner versions.
00:12:47.200 --> 00:13:03.840
And so I think it's it's to me, it is like a natural process to do this V-shaped thing, or maybe not V or upside down V, where like you're you're climbing up to find the fast thing, and then like you're climbing down to make it not so ugly.
00:13:04.000 --> 00:13:11.679
And I think one of the, you know, maybe maybe I'm wrong about this idea that you or I said that like you can't always optimize for two things at once.
00:13:11.919 --> 00:13:16.000
Like it's harder to optimize for multiple things, harder to to optimize for multiple things.
00:13:16.159 --> 00:13:23.519
And I think that's true, but I think part of that is just that we don't have the right reward metrics and safeguards in place.
00:13:23.600 --> 00:13:35.840
That like the to me, if if with my auto research process always generating these like 10,000 line, you know, slop things that's got all this dead code over, that to me means I I haven't put the right guardrails in place.
00:13:35.919 --> 00:13:39.919
I haven't put the right linters and the right things to like for code quality.
00:13:40.080 --> 00:13:46.240
And if I did more of that, then like I could probably generate a better kernels.
00:13:46.320 --> 00:13:48.720
And also I think that would help with the stalls.
00:13:48.879 --> 00:13:53.440
And yeah, not not everything can be like deterministically checked or machine checked.
00:13:53.519 --> 00:14:05.919
The other thing is like I so right now I have in my framework I I uh you know the model can go off, I I let it it can do a macro optimization, a micro optimization, and then a combine.
00:14:06.000 --> 00:14:08.720
And I do have a fourth type called simplify.
00:14:08.879 --> 00:14:13.279
And usually I don't let it run simplify during the main auto research loop.
00:14:13.440 --> 00:14:14.240
But maybe I should.
00:14:14.320 --> 00:14:16.399
Maybe it should be simplifying as it goes.
00:14:16.559 --> 00:14:25.200
I just the reason I set it up this way is that I've I guess I've always done some of the simplification as I go, but usually I make a mess of things until I find speed of light.
00:14:25.519 --> 00:14:29.120
Yeah, it's an like your harness is like reflecting like your own process.
00:14:29.519 --> 00:14:29.759
Yeah.
00:14:30.000 --> 00:14:30.320
Yeah.
00:14:30.480 --> 00:14:31.039
I'm not sure.
00:14:31.360 --> 00:14:33.360
Like I think it's hard for me to say this is right or wrong.
00:14:33.519 --> 00:14:36.320
Like it's this seems like much more of a like personal choice.
00:14:36.399 --> 00:14:39.039
So I can see argument good arguments being made either way.
00:14:39.279 --> 00:14:39.679
Yeah.
00:14:39.840 --> 00:14:40.240
Yeah.
00:14:40.480 --> 00:15:02.080
So yeah, it's I I I find I find the the the the mission that that you guys have at core automation is very compelling to me because like it seems it seems very evident that like if you can make the model really good at automating the development of the model, then like you can like then it can all it can do all the rest, right?
00:15:02.240 --> 00:15:04.399
You can just sit back and sip tea.
00:15:05.440 --> 00:15:16.080
I mean what's cool about it too is that like I uh again, because like for me this is all new, but like by virtue of focusing on like on a few problems, I think it simplifies like a lot of things.
00:15:16.240 --> 00:15:24.559
Like uh, you know, we don't have to have like a model that's like say necessarily generally capable of all sorts of things, but it it primarily needs to be good at things that are tasks in the lab.
00:15:24.799 --> 00:15:28.879
So like you can think of every individual person in the lab as like an eval in some sense.
00:15:29.039 --> 00:15:29.279
Yeah.
00:15:29.679 --> 00:15:31.279
And I think this is like very helpful for me.
00:15:31.440 --> 00:15:37.360
And you know, when you look at like the kinds of tasks people do in the lab, like it's just it becomes like very self-evident, like how you should do the rest.
00:15:37.519 --> 00:15:46.159
That like, let's say for kernels at least, like none of the evals I know of open source, and I, you know, I know this because I've also created a couple of them, are like sufficient for what we want.
00:15:46.320 --> 00:15:56.799
And then it's like very cool thinking through like basically like how do you like basically what would you know the way I explain this to my colleagues is like what evals would help us like aura farm really hard?
00:15:57.039 --> 00:16:01.200
You know, and then these tend to be like capabilities that you yourself think are really impressive.
00:16:01.360 --> 00:16:01.519
Yeah.
00:16:01.679 --> 00:16:04.320
Like like what do I think is at the limits of my capabilities?
00:16:04.559 --> 00:16:10.159
And if I can get like our next release to do those things, you know, that then I think I'd be really, really proud of the work we did.
00:16:10.399 --> 00:16:10.799
Yeah.
00:16:11.039 --> 00:16:16.559
You know, and the the another the last point I'll touch on before I let everybody go and I'll go to sleep.
00:16:16.720 --> 00:16:23.840
You know, Connor, you mentioned earlier that like like the stuff you were doing, you know, like I stopped it because it it wasn't valuable.
00:16:24.080 --> 00:16:39.120
And I I just keep coming across places in which the usage data, the like the experimentation, like the the history of what has been tried is just so valuable.
00:16:39.360 --> 00:16:53.440
It's valuable for like one learning like an you can analyze it, you can improve things, you can analyze how particular tools or frameworks were used, and like once you have data at scale, like you can learn a lot from that.
00:16:53.600 --> 00:17:07.759
It it's it's out it's valuable for like simulating how you know how these long horizon tasks work, which you know can tell us a lot about how various like frameworks and hardware might perform.
00:17:07.920 --> 00:17:13.920
And then it can be valuable for training models for for doing post-training and RL.
00:17:14.000 --> 00:17:21.519
And it can be valuable for like extracting, like, oh, like maybe this is something that would be a good, you know, a good benchmark.
00:17:21.599 --> 00:17:24.640
And I I just I think there's so much value in all this data.
00:17:24.720 --> 00:17:39.440
And like that's one of the things I think is great about GPU mode is it's just it's this, you know, it's this farm that's producing all of this super high value data, like open data that we can all benefit from.
00:17:39.759 --> 00:17:51.200
I mean, Bryce, honestly, to me, it seems like the the more the more you're talking, like the more I get excited about this idea of like, why don't we just make it easier for people to compete in our competitions using open model tokens?
00:17:51.440 --> 00:17:57.359
We can subsidize the tokens by picking the problems to be the kernels that show up in the open models.
00:17:57.519 --> 00:18:05.920
And we get this like nice recursive self-improvement loop with people doing search over the best prompts collectively, because I think that's like highly non-obvious.
00:18:06.400 --> 00:18:11.359
For this, for these sprints over the weekend, like I and the Connor's definitely gonna have to cut this part out.
00:18:16.799 --> 00:18:26.480
Because it was like advertised as like, come, like we're gonna do agentic engineering on this hard problem, like they didn't have to learn anything about CUDA first.
00:18:26.640 --> 00:18:30.240
What they did was they had to learn how to set up auto research.
00:18:30.400 --> 00:18:43.359
And then, like on the second day, once they'd produced some auto research results, then like we like sat down and I talked to them about like the problem and taught them a bit about GPU architecture, and like they were able to learn more because they'd already already accomplished something.
00:18:43.440 --> 00:18:46.559
They they didn't know what they'd accomplished yet, because the agent had accomplished something.
00:18:47.759 --> 00:18:50.880
Yeah, but but but but they got like that sort of first like dopamine rush, right?
00:18:50.960 --> 00:18:51.920
Like I made something faster.
00:18:52.400 --> 00:19:21.839
Definitely like one thing I I I have this vision for like in the future of like doing, I guess the way I I know you said you weren't such a fan of like in-person events, but I'm imagining like GTC Auto Research Academy, where like you just like you have two hours every day where you come you give everybody unlimited inference for the week, you come in for two hours every day, and it's like a study group because you know you're like in between the study group meetings, people are going and running stuff and producing results, and then you come back and you you check in.
00:19:21.920 --> 00:19:38.720
And yeah, I think it'd be I I think it'd be very valuable to have open tokens, and I think people position open tokens as a public good as like this idea of like, oh, it's like it's goodwill, but for me it seems like it's insanely valuable to NVIDIA to do this.
00:19:38.880 --> 00:19:39.440
Yeah, yeah.
00:19:40.160 --> 00:19:57.519
Like, but basically, I just to sort of be specific on the like I think my take on events these days is just like the agentic era has made open-ended judging a bit cooked because it's like how do you tell whether something is amazing versus not if it's just like if Claude one-shot it?
00:19:58.079 --> 00:20:09.440
But if someone did a hill climb on a task that we can all collectively agree on is hard and made like a breakthrough, that's better than what I would do myself as like an advanced user.
00:20:09.599 --> 00:20:12.640
Like, if they can just do that during the hackathon, like I'm very impressed.
00:20:12.720 --> 00:20:16.160
And so, like, it just doesn't matter how they did it, it matters is the the output.
00:20:16.400 --> 00:20:24.319
So I think that I'm very open to, and I think it's gonna become a lot more interesting doing it with open models this year, because like the open models are good finally at these tests.
00:20:24.400 --> 00:20:27.119
I think it's it looks like they've been RL'd on kernel data.
00:20:27.200 --> 00:20:27.440
Yeah.
00:20:27.680 --> 00:20:30.559
And so I think it's gonna be a lot more exciting than it was like a year ago.
00:20:30.799 --> 00:20:42.160
I want to pitch you on another idea I had, which is you know, you talked about the idea of having like a package, like here's how to deploy a you know, like like an inference service to do this like at your company.
00:20:42.400 --> 00:20:48.640
I've been thinking about this idea of, well, you know, what we want to do is we we need two we need GPUs for two things.
00:20:48.720 --> 00:20:54.799
We need GPUs one to run the tests and the benchmarks, and two, we need GPUs to run the inference.
00:20:55.039 --> 00:21:01.920
And you know, there's like web search is something that like runs like inference, like, you know, on the inference provider side.
00:21:02.079 --> 00:21:11.359
Like, why couldn't we have like a you know inference provider side like tool call that just like like like that's like modal that just like runs the serverless thing there?
00:21:11.440 --> 00:21:21.119
And then like if you're a company and you have like you know an 88x B200 machine, it can both run your inference and run your serverless, you know, evals.
00:21:21.279 --> 00:21:24.240
Because a lot of people seem gated on the GPU access too.
00:21:24.559 --> 00:21:27.279
Yeah, I I think of these as like very related problems.
00:21:27.519 --> 00:21:31.200
It's just that like I would imagine more people would know what to do with tokens versus a GPU.
00:21:31.279 --> 00:21:33.599
So it's like I would imagine like broader market appeal.
00:21:33.920 --> 00:21:44.480
But yeah, I mean like ultimately, I think the way modal set up how you acquire GPU, I think this is the correct way to do it because it's very incentive aligned with your customers to charge for use.
00:21:44.640 --> 00:21:47.759
Like it's an honest business, basically the way I would frame it.
00:21:47.920 --> 00:21:55.920
Uh and I think and I think paying for access is dishonest because it it's uh it I I just yeah, like because whether you use it or not, it's like it's not great.
00:21:56.000 --> 00:21:56.480
I don't like it.
00:21:56.640 --> 00:21:59.759
Yeah, yeah, it's like if I could like my biggest thing with this.
00:22:00.160 --> 00:22:04.559
Nvidia right now is like I need like I have this stream of like free tokens from Brev.
00:22:04.799 --> 00:22:15.599
I just they if they could just make a serverless product, even if it's like a worse version of modal, then like I would not need to ask for as many tokens and like it could be so much more token efficient.
00:22:15.839 --> 00:22:21.599
By the way, I refreshed the uh I refreshed the tokens for the the profiling service again too.
00:22:21.839 --> 00:22:22.400
Thank you, sir.
00:22:22.480 --> 00:22:22.559
Yeah.
00:22:23.519 --> 00:22:29.039
Um so I think there's enough in there to run through to run through the end of July.
00:22:29.119 --> 00:22:32.160
But so when do how long does the optimizer contest run?
00:22:32.559 --> 00:22:36.319
The optimizer con we we don't know yet, but we're thinking of making it run for a month.
00:22:36.559 --> 00:22:36.880
Okay.
00:22:37.039 --> 00:22:41.279
I'm just gonna keep refreshing the tokens for the profiling service until somebody tells me to stop.
00:22:42.079 --> 00:22:43.119
Okay, that's that's wonderful.
00:22:43.359 --> 00:22:43.680
Yeah.
00:22:43.839 --> 00:22:44.160
Yeah.
00:22:45.039 --> 00:22:50.240
I think I think we didn't get the chance to talk much about the optimizer competition, but I think it's it's pretty it's pretty sick, actually.
00:22:51.039 --> 00:22:52.079
I'll tell you more about it next time.
00:22:52.319 --> 00:22:53.440
Uh yeah, I'm very excited.
00:22:53.519 --> 00:22:54.799
I I saw I started looking into it.
00:22:54.880 --> 00:22:56.000
Yeah, it looks like it's gonna be very cool.
00:22:56.799 --> 00:22:58.079
Do you mind if I just tell you about it for a bit?
00:22:58.240 --> 00:22:58.559
Yeah, sure.
00:22:58.960 --> 00:23:00.079
I think just okay.
00:23:00.240 --> 00:23:09.839
So so the optimizer competition, like like the sort of main idea was that we we've been very interested in picking tasks where transformers obviously fall flat on their face.
00:23:10.000 --> 00:23:12.480
And so the task we picked is one that involves depth.
00:23:12.559 --> 00:23:14.799
It's basically like repeated squaring and modulo.
00:23:15.119 --> 00:23:22.799
You try to make any like the largest transformers run well on this task, it's gonna run flat on, it's gonna fall flat on his face, and it's gonna try and cheat the task.
00:23:22.960 --> 00:23:40.480
And intrinsically, the only models that have like the right inductive biases to solve these problems are those that include like basically some form of looping or adaptive depth, or like basically and they basically involve like scaling training time compute to sort of like solve the task.
00:23:40.640 --> 00:23:53.279
And these are very exciting because if you sort of if you maybe have a brief physics background, like a lot of these, like anything that involves looping becomes like basically it turns into mush, like basically it just goes either to zeros or it goes to nan.
00:23:53.359 --> 00:23:55.039
Like the system has become very unstable.
00:23:55.200 --> 00:24:04.960
And so what we're very excited about here is we want people to come up with interesting like model architecture ideas to solve this very like simply simply formulated but very nasty problem.
00:24:05.119 --> 00:24:13.039
But because these architectures are quite unstable, they will also need to come up with sophisticated optimizers to solve this problem.
00:24:13.200 --> 00:24:16.960
And the sort of user experience we borrowed is the exact same as kernelbot.
00:24:17.039 --> 00:24:21.839
It's like you submit a single Python file and we run all of these evaluations for you.
00:24:22.160 --> 00:24:27.279
Right now we're basically going through like a trial period where people are reward hacking us to death, which is okay.
00:24:27.440 --> 00:24:29.359
It helps us like have the rules be more robust.
00:24:29.519 --> 00:24:31.279
We're gonna launch for real this Thursday.
00:24:31.359 --> 00:24:37.200
And the competition will be open for about a month, and we're kind of excited to sort of see who comes out of the woodworks here.
00:24:37.519 --> 00:24:54.000
What's been interesting is that like there's this like you know, there's like it's sort of this interesting intersection between people that do complexity theory and model architecture, where they will write very specific proofs as to things like, hey, like a diffusion model can't actually learn long sequence like video generation.
00:24:54.160 --> 00:24:58.480
And I I find this like to be really, really compelling, and I just can't wait to see what people do.
00:24:58.720 --> 00:25:07.680
Generally, for me, I think GPU mode really becomes healthier if people are aware of like frontier level problems, and this is like a frontier level problem that I think many, many people are interested in.
00:25:07.920 --> 00:25:10.400
So I think we made it very agent-friendly.
00:25:10.480 --> 00:25:11.759
The GPUs are all free.
00:25:11.920 --> 00:25:14.720
You get an hour of training time basically a day.
00:25:14.880 --> 00:25:16.400
So, you know, please have fun.
00:25:16.480 --> 00:25:23.119
The the website is like one layer deeper.ai, and I think it's gonna be like one of our most interesting challenges yet, and I'm very excited about it.
00:25:23.359 --> 00:25:28.720
Yeah, no, I think it's I mean, I not be not being not being an ML person.
00:25:28.960 --> 00:25:39.200
I uh I I like being more a kernel person than an ML person, it's a little out of my depth, but from what I've read up on it, it it looks like it's a very, a very cool problem.
00:25:39.279 --> 00:25:44.079
And um, I'm very excited to see what people will uh will will be able to do with it.
00:25:44.160 --> 00:25:53.119
And it also seems like it's something that is probably not in the training set and is like a frontier problem that we'll really be able to see what people can do with auto research for this.
00:25:53.279 --> 00:25:53.599
Yeah.
00:25:53.759 --> 00:25:54.240
Yeah.
00:25:54.559 --> 00:26:01.839
Alright, one one last one last question, and it hopefully will be a short one, because I know Bryce, you need to go to sleep because you're over in the Europe's.
00:26:02.400 --> 00:26:13.359
One of my it's not really a hobby, but something I've been doing over the last several months is I I Well, the details don't matter, but I listened to a lot of No no no, tell us the details.
00:26:13.599 --> 00:26:14.079
The details?
00:26:14.160 --> 00:26:15.119
Alright, very quickly.
00:26:15.279 --> 00:26:23.119
I mean, I have a a vibe-coded podcast player called Pod God, which uh it was gonna get released in the next couple weeks.
00:26:23.279 --> 00:26:26.319
But I've been using it since the beginning of February, and it's phenomenal.
00:26:26.480 --> 00:26:30.400
One of the things it allows you to do is it allows you to follow personalities.
00:26:30.480 --> 00:26:38.000
So there's something called the personality meta podcast, which is just a curated feed of every single time that person goes on a different podcast.
00:26:38.079 --> 00:26:43.920
Because there are some people that I care about and I want to listen to any time they're being interviewed, but I don't want to have to go and listen to example.
00:26:44.240 --> 00:26:46.160
The person that I'm about to mention is Boris Cherney.
00:26:46.240 --> 00:26:48.880
Like, but the thing is is one No idea who that is.
00:26:49.200 --> 00:26:49.359
What?
00:26:49.440 --> 00:26:50.960
The creator of Claude Code?
00:26:51.200 --> 00:26:52.079
No, no idea.
00:26:52.319 --> 00:26:52.960
That's very sad.
00:26:53.119 --> 00:27:04.079
He's like, he's honestly would be my number one pick to have as like a future guest because every every time he is interviewed, I think he has just like amazing takes.
00:27:04.240 --> 00:27:16.000
But anyway, so like there's a bunch of podcasts that one I don't want to subscribe to, like there's like Sequoia has this like kind of there's a bunch of these like VC money podcasts, but every once in a while they have someone interesting on.
00:27:16.079 --> 00:27:21.440
So like I don't want to have to subscribe to this and then like basically just like download the thing and then delete it.
00:27:21.599 --> 00:27:22.480
I just want the interviews.
00:27:22.640 --> 00:27:25.359
Anyways, that's I that's not long story short, that's long story long.
00:27:25.599 --> 00:27:34.799
Whenever you listen to Boris Cherney, like one out of every two podcasts, he gets like book recommendations or he gives book recommendations, and they've been phenomenal.
00:27:35.039 --> 00:27:45.759
The first one that he recommended, he actually mentioned in the Peterman podcast, which is like a former, I think, Instagram or Facebook guy, and he's just been it's it's great interviews.
00:27:46.000 --> 00:27:54.079
He interviewed uh Boris Cherney, and then he mentions that during the interview for Anthropic, he ended up talking about some book by Greg Egan.
00:27:54.319 --> 00:27:57.839
You know, this is back in like Claude, I don't know, 3.7, 4.0 days.
00:27:57.920 --> 00:28:02.319
I immediately go to like whatever ChatGPT ask, like, what book is he referring to?
00:28:02.559 --> 00:28:08.640
And it ended up being Permutation City, and that is like part of a trilogy that has Diaspora and Shield's Child.
00:28:08.720 --> 00:28:15.119
He on a more recent podcast mentioned Werner Vinge's A Fire Upon the Deep, which is in the Zones of Thought trilogy.
00:28:15.279 --> 00:28:17.039
I'm like one down, two to go.
00:28:17.279 --> 00:28:20.640
I bring all of this up, where's the question, to ask you?
00:28:20.799 --> 00:28:31.599
So you wrote the Robot Overlord manual, and you're the first person actually that we've interviewed on this podcast, which is a little bit surprising, that is as high on AI as Bryce and I.
00:28:31.920 --> 00:28:38.480
We we've interviewed some folks over the last little bit, but not like not people that are sipping the Kool-Aid as much as we are.
00:28:38.720 --> 00:28:44.240
And so uh the the answer to this question may be I don't have any recommendations.
00:28:44.480 --> 00:28:48.400
My question is, do you have any book recommendations in general?
00:28:48.640 --> 00:29:00.640
And on top of that, specifically, I would love to know if you have like any recommendations in the space of you know future AI, like good these kinds of good reads that like people like Boris Cherney recommend recommending.
00:29:00.799 --> 00:29:01.759
So over to you.
00:29:02.160 --> 00:29:04.720
Yeah, like I mean, I I love reading.
00:29:04.799 --> 00:29:05.920
Like this, these are all my books.
00:29:06.160 --> 00:29:11.599
The listener can't see, but there's also like a bookshelf in the background with what looks like at least a couple hundred books there.
00:29:11.680 --> 00:29:22.880
So the the there's like a couple of other hundred that I've given away over the years, but yeah, I mean, I I like I actually like I I do I I had a tweet about this, which is like my ten favorite books.
00:29:23.119 --> 00:29:23.599
Let me see.
00:29:23.839 --> 00:29:29.119
So I would say on the AI, like, okay, so these are these are the books that were most influential to me.
00:29:29.359 --> 00:29:30.160
There's ten of them.
00:29:30.319 --> 00:29:30.960
So let me see.
00:29:31.039 --> 00:29:32.480
Maybe I can pick two for you.
00:29:33.039 --> 00:29:36.720
Well, we'll include we'll we'll include uh the link to the tweet so you can get all ten.
00:29:36.880 --> 00:29:39.119
You can and feel free to just read all ten off if you want to.
00:29:39.440 --> 00:29:41.440
Okay, like it's a more interesting question.
00:29:41.519 --> 00:29:42.319
Okay, the all ten.
00:29:42.559 --> 00:29:48.160
So my like the book that I think was like the most fun to me was a book called The Journey to the Ants.
00:29:48.400 --> 00:29:51.599
So it's it's it's it's a book about like ant-based intelligence.
00:29:51.839 --> 00:29:57.440
And the reason why I found it very interesting is because like the the kinds of shit ants can do is pretty wild.
00:29:57.519 --> 00:30:05.200
Like, for instance, like you can have like the mother ant, the queen, like control the gender ratio of a population, control like people's tasks.
00:30:05.279 --> 00:30:13.440
Like you can be a worker versus a warrior elastically based on the demands of the colony, and they all sort of spread this message between each other to sort of dictate.
00:30:13.599 --> 00:30:20.960
This is very contrary to how like human societies operate, where we're like, oh, I'm a colonel guy, and I will never not do kernels, like we're very inelastic.
00:30:21.119 --> 00:30:26.160
So it's like pretty remarkable for me to see an example of um of societies that were not this way.
00:30:26.240 --> 00:30:27.599
And I think this book is wonderful.
00:30:27.839 --> 00:30:30.079
Another book I really love is visual group theory.
00:30:30.160 --> 00:30:33.519
So visual group theory, and uh, maybe I would add Burns Euclid here.
00:30:33.680 --> 00:30:42.960
What I found really lovely about these books is that like any anything that's algebraic tends to be like very dry and like kind of unintuitive to read because you just see look at these long formulas.
00:30:43.119 --> 00:30:49.920
Both of these books are pretty like beautiful, they almost feel like pieces of art where people are sort of like going through like these arguments like very rigorously.
00:30:50.079 --> 00:31:01.200
And so for me, they really made me appreciate like the idea of a beautiful explanation, which is like typically when I think of like my job, I don't necessarily think of it as like producing like beautiful code or fast code.
00:31:01.359 --> 00:31:04.640
I think of it as being able to produce a good explanation for why the code is fast.
00:31:04.720 --> 00:31:06.960
Like that's kind of almost like my love language.
00:31:07.200 --> 00:31:09.599
Another book I've adored is The Theory of Fun.
00:31:09.680 --> 00:31:13.920
So the Theory of Fun is a book on why people like playing games.
00:31:14.079 --> 00:31:18.880
It's sort of like the sort of the main thesis of the book is that playing is basically the way we learn.
00:31:19.039 --> 00:31:25.039
So I've carried that a lot into my projects where I've tried to make a lot of the things we've done with GPU mode fun.
00:31:25.200 --> 00:31:28.319
And as a result, people end up learning more and might find it more valuable.
00:31:28.400 --> 00:31:33.920
So a lot of what I learned with GPU mode was like basically early days of playing World of Warcraft and Warcraft 3.
00:31:34.000 --> 00:31:36.319
So like a lot of the lessons, funnel enough, do apply.
00:31:36.559 --> 00:31:46.880
I would say two technical books that I've I think I'm gonna skip over a couple of books, but two technical books that I think I highly recommend to everyone are uh game programming patterns and working with legacy code.
00:31:47.039 --> 00:31:50.960
Working with legacy code often surprises people as one of my favorite books of all time.
00:31:51.039 --> 00:31:56.079
But I found it really wonderful when I'm just like plopped into like a monstrous code base.
00:31:56.240 --> 00:31:58.559
How do you sort of like quickly make sense of it?
00:31:58.640 --> 00:32:03.680
And like what are good design patterns to make codes like simpler and easier to reason about?
00:32:03.839 --> 00:32:13.519
I think a lot of like my contributions in the in past projects has been like going into a complicated code base and like understanding it and then figuring out how to develop features more quickly on top.
00:32:13.680 --> 00:32:17.839
I think a lot of these ideas will make a comeback if we're like basically in mountains of slop.
00:32:18.000 --> 00:32:22.240
Maybe yet another book that I've really enjoyed, maybe okay, maybe I'll give you two more.
00:32:22.480 --> 00:32:25.200
Like Game Theoretic Foundations for Probability and Finance.
00:32:25.279 --> 00:32:27.680
It's not a finance book, even though it's in the title.
00:32:27.839 --> 00:32:33.680
It's a book about rewriting all the axiom of probability theory with game theory in a way that makes them a lot more interpretable.
00:32:33.839 --> 00:32:38.559
And so I've also really enjoyed this kind of book because I've always enjoyed like more wacky math.
00:32:38.640 --> 00:32:40.079
It just can help you build intuition.
00:32:40.160 --> 00:32:43.119
And if you didn't like one notation, maybe you'll appreciate another.
00:32:43.359 --> 00:32:46.880
The last one is the structure and interpretation of classical mechanics.
00:32:47.039 --> 00:32:50.799
I'm sure many of you might have read the structure interpretation of like computer programs.
00:32:51.359 --> 00:32:52.640
That's what I thought I was gonna say.
00:32:52.960 --> 00:33:06.720
I I think this book is better because it basically goes over like it's writes short programs for physics written in Lisp, and you basically end up with nature as like a short series of like self-modifying Lisp programs.
00:33:06.960 --> 00:33:17.440
It's one of those like books that like blow your mind the first time you read it, but like again, like as we get into the age of auto research and architecture research, I suspect a lot of these ideas are are making a big comeback.
00:33:17.599 --> 00:33:28.559
So these are things that I think are are you know there's a lot more, but like maybe you know, berserk and necronomicon, maybe with some pieces of fiction that I think are just like absolutely gorgeous.
00:33:28.799 --> 00:33:36.400
I I've always been a sucker for like there's some piece of knowledge that if you acquire it, I don't know, you'll blow up because you'll be rendered mad.
00:33:36.640 --> 00:33:42.960
Like I don't know why, but it's like one of those things that I've always like I've been like a sucker for this kind of like fantasy, fantasy work.
00:33:43.119 --> 00:33:44.720
But yeah, all these books are phenomenal.
00:33:44.799 --> 00:33:47.839
I'll share the full list if you like with you, like in a few minutes.
00:33:48.079 --> 00:33:51.680
But yeah, I I think these are sufficiently out of distribution to give you better ideas.
00:33:52.000 --> 00:33:57.039
I mean, that was I feel bad we haven't asked our other past guests because that was that was uh amazing.
00:33:57.200 --> 00:34:02.480
Even better than not that I you know had expectations for the quality of your answer, but uh even better than I expected.
00:34:02.640 --> 00:34:03.279
Go ahead, Bryce.
00:34:03.599 --> 00:34:11.760
I I have a I have a related question, which is it sound like it sounds like you have been a person who's played a lot of video games in the past.
00:34:12.000 --> 00:34:14.480
And I'm wondering if you have recommendations for video games.
00:34:14.880 --> 00:34:16.239
Uh let me see.
00:34:16.559 --> 00:34:19.760
Um so what are the greatest games I've played?
00:34:19.920 --> 00:34:27.760
So so the one of the issues with like like I think the greatest game I've ever played is probably Dota 2, but I would not recommend you play it because it just will take over your life.
00:34:27.920 --> 00:34:40.320
So I think it's like an optimizer's dream in some sense, and it teaches you how to never get mad at anything because like it's very formative to be insulted at in like five different languages and still manage how to win a game.
00:34:40.559 --> 00:34:41.679
So it's very formative.
00:34:41.840 --> 00:35:00.480
But but I would say, like, I mean, if I'm thinking like games that like left like a deep impression on me, I would say recently, I I would probably say it was like Expedition 33 as like sort of a story-based game, like really left like a deep impression on me, and I was like a big sucker for like role-playing games like growing up, you know, all the way ranging from like Pokemon to like Chrono Trigger.
00:35:00.639 --> 00:35:03.599
As a result, I've also really enjoyed Final Fantasy XVI recently.
00:35:03.679 --> 00:35:06.559
I've been playing Final Fantasy 7 as well, which I really enjoy.
00:35:06.719 --> 00:35:14.320
But yeah, otherwise, like these days, like I mostly spent a lot of time playing Slay the Spire, but I also feel like I spent too much time doing it.
00:35:14.400 --> 00:35:18.800
So if you give me an optimization puzzle, usually I can get obsessed with it.
00:35:18.960 --> 00:35:25.039
If you give me something that's fast and reflex-based, like Fury or Katana Zero, I'm also like a big sucker for those.
00:35:25.119 --> 00:35:27.119
Like I enjoy sort of this like flow feeling.
00:35:27.280 --> 00:35:30.960
And I think these are all games that will give you like a major, major uh sense of flow.
00:35:31.039 --> 00:35:33.039
So they're all treats if you haven't played them.
00:35:33.360 --> 00:35:41.840
So my my actually my my oldest friend, the person that I've had the longest contact with, is somebody I've never actually met in person.
00:35:42.000 --> 00:35:47.679
And he and I, I used to play a lot of MUDs, text-based games when I was a kid, and he and I played MUDs together.
00:35:47.920 --> 00:35:51.840
And in particular, my favorite type of MUD was a roleplay-intensive mud.
00:35:52.079 --> 00:35:55.840
So that's where everybody has to be in character at all times.
00:35:56.000 --> 00:36:08.639
I primarily played either Star Wars or Lord of the Rings themed MUDs, and these were all like community-run multiplayer games, and nobody was really concerned about the fact that like Star Wars and Lord of the Rings were IP that somebody else owned.
00:36:09.039 --> 00:36:14.159
But, anyways, him and I were having this conversation the other day where it's a it's a text-based game.
00:36:14.320 --> 00:36:25.119
Like, it is you could just like hook up, I'm sure, like a frontier model to it, and you know, the problem that these games have today is like, you know, the NPCs are not particularly good and there's just not a lot of players.
00:36:25.280 --> 00:36:29.280
And him and I were talking about like, well, we could just make as many players as we want.
00:36:29.360 --> 00:36:32.800
You know, you could just like you wouldn't even need to code them into the game.
00:36:32.960 --> 00:36:40.400
You could just connect an existing agent to an existing mud and just have it interact the game as it would any other player character.
00:36:40.559 --> 00:36:43.920
And I just like I I can't haven't been able to get this idea out of my head.
00:36:44.000 --> 00:36:52.079
And if I had if I had more infinite inference, I would go build my like AI mud game, and I'm sure I'd have a blast with it.
00:36:52.320 --> 00:37:01.199
I I I think we should try it out because I I think one hybrid idea I'd be very optimistic about would be something like uh like an in-person Dungeons and Dragons game.
00:37:01.360 --> 00:37:01.760
Oh yes.
00:37:02.000 --> 00:37:12.079
Where like the like where the DM can basically choose to like basically where there's a DM who's constantly prompting to try and anticipate where like things might go.
00:37:12.239 --> 00:37:13.760
That I'm I'm really optimistic about.
00:37:13.920 --> 00:37:21.840
But but I think a lot of the I think issues I foresee with current alums is like post-compaction, maybe they'll lose everything, like they'll just have no idea what the fuck's going on with the game.
00:37:22.239 --> 00:37:26.320
We should do a whole we we should we should talk some other time about my thoughts on compaction.
00:37:26.480 --> 00:37:29.119
I think sometimes compaction helps and sometimes it hurts.
00:37:29.360 --> 00:37:30.320
In the short.
00:37:30.480 --> 00:37:30.880
Yeah.
00:37:31.119 --> 00:37:39.039
The thing that the thing that's most valuable that I get that happens at compaction is most harnesses on compaction will reread skills.
00:37:39.199 --> 00:37:47.199
And that can sometimes help because then it's like a reminder of like, were you following, were you following like the prompts and skills that you were originally given?
00:37:47.360 --> 00:37:51.760
Anyways, I should go, it's very it's getting very late here, and we should let Mark get going.
00:37:52.320 --> 00:37:53.119
Good night, Bryce.
00:37:53.199 --> 00:37:54.639
And then thank you for inviting me, Bryce.
00:37:55.760 --> 00:37:57.039
It was a pleasure to chat with you.
00:37:57.360 --> 00:38:06.880
Be sure to check these show notes either in your podcast app or at adspthepodcast.com for links to anything we mentioned in today's episode, as well as a link to a GitHub discussion where you can leave thoughts, comments, and questions.
00:38:07.280 --> 00:38:07.840
Thanks for listening.
00:38:07.920 --> 00:38:09.519
We hope you enjoyed and have a great day.
00:38:10.400 --> 00:38:15.519
Low quality, high quality, that is the tagline of our podcast.
00:38:15.840 --> 00:38:16.719
That's not the tagline.
00:38:16.960 --> 00:38:19.760
Our tagline is chaos with sprinkles of information.
