Claude Code Leak! Rate Limits keep changing, and Building Agentic Systems | Ep 13

WEBVTT
00:00:01.199 --> 00:00:11.839
You are now tuned in to the Rate Limited Podcast with your host, Ray Fernandu, Gosu Coder, aka Adam Larson, and Eric Brownsche, the founder of Repo Prompt.
00:00:12.160 --> 00:00:14.800
We are here because there's a lot of rate limits going on.
00:00:14.880 --> 00:00:18.960
If you've been using Claude Code and all these different things, there's so much crazy news and gossip.
00:00:19.039 --> 00:00:30.640
And if you stay towards the very end, we have some really cool stuff about how to do agentic coding, like agents to agent frameworks and all these really great deep dive discussions from Adam Larson here.
00:00:30.719 --> 00:00:33.920
And so without further ado, Eric, go ahead and kick off the show.
00:00:34.320 --> 00:00:35.600
Yeah, thanks for that, Ray.
00:00:35.679 --> 00:00:45.600
So I mean the first news item this week that I think really deserves to be touched on is that it seems like the Claude Code team accidentally uploaded their source code uh online.
00:00:45.840 --> 00:00:53.039
And it didn't take long before the entire entirety of Twitter just filled it up, grabbed it, and distributed it all over GitHub.
00:00:53.119 --> 00:00:54.079
So it's all over the place.
00:00:54.159 --> 00:00:58.320
And now I think the lawyers are coming for all those uh all those forks and whatnot.
00:00:58.399 --> 00:01:02.719
Um so it's really interesting to see that like a mistake like that could happen.
00:01:02.880 --> 00:01:13.200
Um from what it seems, uh there was just like a process error where a developer accidentally included a source map, and the other team has been very gracious and not like throwing that dev under the bus who did it.
00:01:13.280 --> 00:01:16.480
Um and I think you know, we can all relate to mistakes happening in prod.
00:01:16.799 --> 00:01:23.280
Like, you know, there's processes involved for a reason, and we want to make sure that these things happen, you know, correctly in practice.
00:01:23.439 --> 00:01:31.840
Um, you know, Ray, you know, you worked at Apple for a long time, and you know, you were telling me earlier you, you know, these things these things can happen even at you know large companies like Apple, even without AI involved.
00:01:31.920 --> 00:01:35.680
So why don't you tell us a little bit about what that's like, you know, making big releases?
00:01:36.079 --> 00:01:43.519
Yeah, I think the most important lesson I learned from all these years of things and uh stuff, it's like, you know, I don't speak directly for the company.
00:01:43.599 --> 00:01:54.480
This is just kind of my own experience as a software engineer that like you know, if I'm running a company now and I have employees that are hired, uh, you know, the release process is extremely important and there's certain things that you want to check.
00:01:54.799 --> 00:02:00.319
Uh I actually one of my first bugs that I found out uh actually prevented, you know, a whole bunch of crazy stuff from happening.
00:02:01.280 --> 00:02:11.280
You know, I assumed it cost the company lots of money, and the the best part and the thing that I appreciated from all my managers were that it says like don't make the same mistake again.
00:02:11.520 --> 00:02:19.120
So uh I think that really got me to think as a really junior engineer to be like, okay, I really have to pay attention to all these details.
00:02:19.520 --> 00:02:25.120
Even if I do delegate to an AI system, you know, what the rules are in place, uh, you know, is there a human involved?
00:02:25.360 --> 00:02:28.000
Are there like really specific critical steps?
00:02:28.479 --> 00:02:38.080
And um, I if they're like that important to the business where it can be costly, uh, you know, more more eyes and more attention are gonna be on it until we have a system that can take over.
00:02:38.159 --> 00:02:48.639
And I think kind of that's the uh the big uh uh thing that I would say is a takeaway that maybe if you have some business critical thing that is gonna impact your business a lot, make sure you have some process around it.
00:02:48.879 --> 00:02:56.080
Um and either other folks to check or whatever that's involved, even if it takes time away, uh, you know, all your hard work can just go away overnight in in some way.
00:02:56.400 --> 00:02:57.759
So it's it's always pretty cool.
00:02:58.479 --> 00:03:01.439
Yeah, I mean it's uh it was a huge deal to see this source leak.
00:03:01.520 --> 00:03:05.919
You know, Anthropic's been very um secretive with a lot of their stuff.
00:03:06.000 --> 00:03:11.759
They're you know not so open, you know, like some of the other competitors, most of the other harnesses are you know open in this way.
00:03:11.919 --> 00:03:16.159
Um but so it's interesting to see it happen and and you know what's gonna come from it.
00:03:16.319 --> 00:03:18.479
Like I I I do wonder what's gonna happen now.
00:03:18.560 --> 00:03:28.159
Um, but I you know, I think you know, the takeaway, you know, seeing from people who are like you know scrutinizing the source, um, you know, there I don't think there's any magic in that harness.
00:03:28.319 --> 00:03:32.639
I think you know, like people are like trying to find you know what what is the special sauce.
00:03:33.599 --> 00:03:43.120
I think at the end of the day, it's just the model the model is is is uh used to a certain shape of of work with these with these uh you know response structures, message formats and such.
00:03:43.199 --> 00:03:53.759
And there's a nice little like set of you know tools around the UX of it, and I think there's some stuff to be learned around that, though I would be careful overly scrutinizing the source, you know, it's it's not public and to be careful what you read.
00:03:54.000 --> 00:04:04.000
But uh yeah, I don't think there's magic in there, and I think uh, you know, we're all just exploring and figuring it out as we go, and I think the labs are doing the same, and so it's interesting to see.
00:04:04.159 --> 00:04:07.599
Um, so yeah, any any thoughts on it, Adam, uh, before we move on?
00:04:07.919 --> 00:04:12.000
Yeah, I mean uh honestly it it was bound to happen at some point in my mind.
00:04:12.159 --> 00:04:31.759
Uh I worked in the video game industry for six, seven years, something like that, and uh pretty much every video game we released within days was already cracked, and source code was out there, and they're like it's like it it's just one of those things that like humans are gonna make a mistake, people are gonna figure out ways to decompile things.
00:04:31.920 --> 00:04:37.680
And to your point, Eric, I don't think there's anything that special about any of the harnesses really.
00:04:37.920 --> 00:04:43.920
I think a lot of it comes down to like how the model interacts with the harness and how they're trained and how they behave.
00:04:44.079 --> 00:04:57.040
Um, I would say, you know, as a company gets mature, to Ray's point, there are better checks and and like gates that something that products need to go through before releasing.
00:04:57.199 --> 00:05:03.920
And this just shows me, you know, anthropics moving fast, they're breaking things, they're probably still operating quasi in startup mode.
00:05:04.079 --> 00:05:21.600
And you know, as if they're a multi-billion dollar company now, they're probably gonna have to start putting in some actual like formal release processes and make sure that whatever boxes need to be checked, which I know a lot of people aren't gonna like, but it is just the nature of when you want if you need to protect your IP, you have to do that.
00:05:21.680 --> 00:05:27.759
And I'm sure there's a lot of people upset right now at Anthropic that their code is out there being looked at.
00:05:27.839 --> 00:05:29.759
But like I said, bound to happen.
00:05:29.920 --> 00:05:32.560
It's just been the nature of the internet in my entire life.
00:05:32.639 --> 00:05:35.839
You know, the entire time the internet's been around, stuff like that's been happening.
00:05:36.160 --> 00:05:41.920
You know, I I did read uh something interesting as well about this, is that um, you know, I mean it's JavaScript, right?
00:05:42.000 --> 00:05:42.720
It's TypeScript.
00:05:42.879 --> 00:05:49.279
So like even in the binary that's released, it's all like reverse engineerable, with it's like minified JS.
00:05:49.360 --> 00:05:50.639
Like you can recover a lot of that.
00:05:50.720 --> 00:05:56.079
Like, so so it's not like there were the stuff wasn't kind of available if you want if you knew where to look.
00:05:56.240 --> 00:06:07.279
I think the difference though, from what I did see, is that um what's interesting about this release is that there's a lot of like internal only parts of the code that were previously stripped from the build.
00:06:07.360 --> 00:06:12.720
Um, but because the source was like fully released now, uh all that stuff is available to kind of look at.
00:06:12.800 --> 00:06:24.800
Um so that's why some people were able to find like you know, secret model code names of stuff that's upcoming and whatnot, and um finding interesting prompts and stuff that are interesting too, like um just interesting things to kind of you know dig around.
00:06:24.959 --> 00:06:29.439
And there's lots of thoughts and people digging around on next if you want to read for it, and I'm sure on YouTube as well.
00:06:29.759 --> 00:06:33.519
Okay, one theory, the Tamagotchi game thing that they released in there.
00:06:33.600 --> 00:06:35.759
Yeah, it had to be an April Fool thing, right?
00:06:35.920 --> 00:06:37.040
Like that hat.
00:06:37.600 --> 00:06:38.079
I don't know.
00:06:38.240 --> 00:06:40.079
It sounds like uh it was in there and planned for a while.
00:06:40.160 --> 00:06:43.360
They put a lot of work into it, so it has to be an April Fool thing.
00:06:43.600 --> 00:06:44.480
Yeah, it's gotta be, man.
00:06:44.720 --> 00:06:51.680
I I think they're they're they're just cognizant of the a lot of people are just staring at the little thinking spinner, and they're like, Well, what are we gonna have people do while they're waiting?
00:06:51.759 --> 00:06:57.120
And start scrolling, and we want to keep engagement, so let's bring them into the app and start feeding a creature.
00:06:57.519 --> 00:06:58.160
Maybe.
00:06:58.399 --> 00:07:01.279
Yeah, uh just subway surfers, bro.
00:07:01.360 --> 00:07:02.639
We just gotta get those subway surfers.
00:07:02.879 --> 00:07:03.279
Yeah, yeah, yeah.
00:07:03.519 --> 00:07:05.600
Just Italian brain rot all over the place.
00:07:05.759 --> 00:07:07.519
Uh right into your code.
00:07:08.720 --> 00:07:09.279
All right.
00:07:09.439 --> 00:07:18.639
Uh, I mean, the next uh topic is I think a little bit more relevant to the folks at the pod here is that Anthropic actually really tightened up some rate limits uh in the last couple of weeks.
00:07:18.879 --> 00:07:22.240
A lot of people were seeing that their usage limits are blowing up.
00:07:22.399 --> 00:07:30.399
Um the official word was that actually they did a lot of efficiency improvements, and it's only 7% of users that would notice like increased rate limits.
00:07:30.639 --> 00:07:36.079
To be honest, 7%, in my opinion, like is actually quite high because all the power users are in that 7%.
00:07:36.399 --> 00:07:38.879
Pretty much everyone on X who's like talking about it are in that 7%.
00:07:39.279 --> 00:07:42.800
So it's bound to feel like a larger number than it kind of seems on paper.
00:07:43.040 --> 00:07:49.920
Like if you think about it, like a lot of the people outside of that sphere of like power users, they're probably using it quite casually.
00:07:50.000 --> 00:07:52.240
So of course they're not going to be hitting those limits.
00:07:52.399 --> 00:08:04.000
Um, but it seems like there's even something else going on on top of this, where you know limits are being hit faster, you know, potentially prompt caching is failing in cases, and uh that's like a quite interesting thing to observe.
00:08:04.240 --> 00:08:08.319
So I'm curious, uh, have either of you been hitting increased rate limits?
00:08:08.480 --> 00:08:14.639
I mean, it seems like they're being squeezed because of all the new users coming in, and there's only so much GPU capacity to go around.
00:08:14.879 --> 00:08:17.519
Um Ray, have you have you been hitting them a lot lately?
00:08:18.639 --> 00:08:23.600
I haven't really been using Opus like Claude Code in the actual Claude Code harness.
00:08:24.160 --> 00:08:32.159
Um a lot of my usage has really been from um you know like external agentic, you know, like the Claude SDK testing type of stuff.
00:08:32.720 --> 00:08:34.480
So I I haven't really seen it there.
00:08:34.559 --> 00:08:38.240
Um you know I have like this agentic framework that's kind of running in my own little thing.
00:08:38.720 --> 00:08:41.039
Um that's basically what I use a lot of my sub for.
00:08:41.120 --> 00:08:42.799
You know, it's like just testing around that.
00:08:43.519 --> 00:08:46.000
Um so I my limits look okay.
00:08:46.080 --> 00:08:51.759
I I use um you know the cloud cowork and claude dispatch, and they seem to be very generous there too.
00:08:51.840 --> 00:08:55.759
I have I have the$200 max plan too, so that's pretty good.
00:08:55.919 --> 00:09:02.720
Um and and you know, I I just did taxes, I did a whole bunch of stuff, all kinds of requests, didn't matter what time of the day I did.
00:09:03.039 --> 00:09:04.559
I still seem to have a lot of limits.
00:09:04.799 --> 00:09:08.480
Well, it seems like the the tighter limits are for you, I think 5 a.m.
00:09:08.639 --> 00:09:09.519
to 11 a.m.
00:09:09.759 --> 00:09:10.399
Pacific time.
00:09:10.720 --> 00:09:13.600
So if you're doing your work outside of those hours, I think they're pretty okay.
00:09:13.679 --> 00:09:18.720
And I know that they have like a temporary promo where outside of the peak hours you're getting double limits.
00:09:18.879 --> 00:09:26.879
So it's like kind of a whiplash where like if you're using it in peak hours, you're getting basically half limits, and then afterwards you're getting like a double or normal.
00:09:26.960 --> 00:09:30.159
So it's kind of hard to gauge like your usage and see like what's fair.
00:09:30.399 --> 00:09:34.480
Um, I I noticed that I I moved up to the hundred dollar plan.
00:09:34.559 --> 00:09:42.240
I was I'm I'm mainly a codex user, but uh I started turning to Claude for certain tasks um in the last couple of weeks, just so I wanted to kind of wrap up a little bit.
00:09:42.399 --> 00:09:52.960
I was on the pro$20 a plan, and that one I noticed when the first day when that those limbs kicked in, I was I just was using Opus a little bit, and within 25 minutes the limits were gone with two chats running.
00:09:53.039 --> 00:09:56.320
Like it you could just see the bar fill up, so that was really terrible.
00:09:57.200 --> 00:09:58.480
But but then especially the$100.
00:10:00.559 --> 00:10:03.200
Uh I mean, I mean I had it on just medium at some point.
00:10:03.279 --> 00:10:08.159
I've been pushing it up to high, but like because it's a lot smarter on high.
00:10:09.279 --> 00:10:12.159
Yeah, it's just uh it's just tough, you know, like to use it a lot.
00:10:12.320 --> 00:10:16.399
And I think the open AI limits are actually coming back down as well now.
00:10:16.559 --> 00:10:20.639
Uh with I think April 1st, April 2nd, where they're the 2x promos ending.
00:10:20.720 --> 00:10:22.720
So we're gonna see what what they're gonna do about that.
00:10:23.200 --> 00:10:31.360
They just reset rate limits again yesterday, so it's very hard to tell like what you get with codex as they just keep resetting them.
00:10:31.519 --> 00:10:34.080
Um so it's very hard to hit the weekly limits.
00:10:34.240 --> 00:10:44.639
What I have noticed with codex though is you're a lot more likely to hit the weekly limit than the five-hour limit, whereas with Claude, I almost never hit the month the weekly limit, but I almost always hit the five-hour limit.
00:10:44.799 --> 00:10:46.799
So, you know, that's a little unfortunate.
00:10:46.960 --> 00:10:48.240
Um, and what about you, Adam?
00:10:48.320 --> 00:10:50.559
Have you been spending any time in Claude Code lately?
00:10:50.799 --> 00:10:53.679
Yeah, it's my main that and cursor are my main two.
00:10:54.159 --> 00:10:57.519
I will say I'm actually uh I hate to even say this.
00:10:57.679 --> 00:11:05.360
I'm kind of thankful they put these limits in place because there was a time that there's so much outages, it was basically unusable.
00:11:05.519 --> 00:11:08.320
Yeah, and I no one wants these, I don't want these limits.
00:11:08.399 --> 00:11:09.200
I still hit them.
00:11:09.440 --> 00:11:16.399
Yeah, but I also want stability so that when I open up claw code, I'm like, oh crap, go check the cloud status page.
00:11:16.480 --> 00:11:17.440
Yep, it's down again.
00:11:17.519 --> 00:11:18.080
I gotta go.
00:11:18.320 --> 00:11:19.039
Yeah, yeah, yeah.
00:11:19.279 --> 00:11:21.440
So that that was my life for a while.
00:11:21.519 --> 00:11:24.240
And then it was like, okay, back to cursor for a bit.
00:11:24.559 --> 00:11:32.639
I am hopeful that they can stabilize, and if they can stabilize, then maybe they can loosen the reins, you know, if they figure things out.
00:11:32.879 --> 00:11:38.320
So I'm trying to keep that positive twist on things, but for me, I hit those limits all the time, honestly.
00:11:38.480 --> 00:11:42.320
Especially because I'm Eastern time zone and I feel like it's like prime coding times.
00:11:42.399 --> 00:11:43.120
And yeah.
00:11:43.360 --> 00:11:44.000
Yeah, yeah, yeah.
00:11:44.240 --> 00:11:47.200
In the morning, morning Eastern time is the worst time to be coding.
00:11:47.279 --> 00:11:48.399
It goes through to 2 p.m.
00:11:48.559 --> 00:11:53.120
Eastern, so that's like really tough if you're trying to get a work done in your workday, normal hours.
00:11:53.279 --> 00:11:56.799
Uh, they really want people to be night coders, I guess, like sleepy damned.
00:11:56.960 --> 00:11:58.399
That's just how it's gonna go now.
00:11:58.559 --> 00:12:06.720
Um so how would you like if you had like a couple, like, would you get two pro plans of one specific vendor?
00:12:06.879 --> 00:12:09.360
Would you get one in one, like you know, 400 bucks a little bit?
00:12:09.519 --> 00:12:10.080
I'd be one in one.
00:12:10.399 --> 00:12:16.960
Like to me, to me, Claude is down so often, at least in my personal opinion, especially during the times that I'm coding.
00:12:17.360 --> 00:12:19.200
Like I it was like literally daily.
00:12:19.279 --> 00:12:23.440
I'd have something happen, I'd go check the status page, and it's like, oh, they got something going on now.
00:12:23.600 --> 00:12:27.279
Or Opus is down, but Sonnet's okay, you know, whatever's happening.
00:12:27.519 --> 00:12:39.039
Uh I would be I would have to like bring a chow and have two different providers personally, just because of I want the reliability to actually have something work, which is why I like cursor and cloud code is my two main ones right now.
00:12:39.600 --> 00:12:44.159
Cursor and cloud code, because you gotta have just Opus, like you're not turning into the GPT models.
00:12:44.480 --> 00:12:46.799
I turn to GPT models when I have to.
00:12:46.960 --> 00:12:49.919
Uh I'm still definitely a bigger fan of the Opus models.
00:12:50.159 --> 00:12:58.480
Can you talk a little bit about like why what that split looks like for you and like why you feel like Opus or or you know just just like the cloud models in general are doing better for you?
00:12:58.799 --> 00:13:14.000
Do you know how like when you work with an engineer on your team and you're like, oh, okay, this person, you don't know them, but they just absolutely crush a project that they're working on, and now you've kind of built up that trust that with that they're actually gonna do a good job with it.
00:13:14.240 --> 00:13:23.679
What I've learned with Opus is if you do the planning up front, you could just let it rip, and 90% of the time it's gonna come back with something very solid.
00:13:24.000 --> 00:13:33.840
Now, I will say there are times that if you do not spend the proper up type of like planning, it will infer things it shouldn't do, and like all the things that would happen, just like a normal engineer.
00:13:34.080 --> 00:13:41.919
For me, Opus is just kind of built that trust that I kind of I when I want when I know I want something to work, I know how to talk to it, I know how to plan with it.
00:13:42.080 --> 00:13:45.919
If I need something to happen and be done, Opus is what I turn to.
00:13:46.159 --> 00:13:56.799
Now on the GPT five the GPT side, I actually like using GPT to review the code that Opus writes, just as that secondary like perspective.
00:13:56.879 --> 00:14:01.279
I was using Gemini for a while, but I've kind of merged into that a little bit more on the GPT side.
00:14:01.519 --> 00:14:10.159
And I don't know, maybe maybe I need to actually just like hard limit myself where I'm only using GPT models for a week just to see if I can build that trust with them.
00:14:10.399 --> 00:14:11.519
But right now it's Opus.
00:14:12.000 --> 00:14:16.960
Yeah, uh, I'm I'm curious though, because you're using it mainly for these long plans in cursor though, right?
00:14:17.279 --> 00:14:17.600
Yes.
00:14:17.840 --> 00:14:18.879
And cloud code.
00:14:19.039 --> 00:14:19.360
Yep.
00:14:19.600 --> 00:14:23.600
Okay, so so for my experience with cloud code, the compaction is just not that great.
00:14:23.759 --> 00:14:34.159
Um, and if you're using the 1 million model token uh with the context window on Opus, I find like over 200k tokens, it's really hard to trust the model.
00:14:34.480 --> 00:14:39.360
Um and and I I mean the same with GPT, like I don't trust like large context GPT either.
00:14:39.440 --> 00:14:57.919
But what I do find is that if I'm executing a large plan uh and I'm very clear up front and I provide like a key way for the model to kind of recheck where it needs to get the information it needs, um, then the GPT models are are better at kind of executing over multiple compactions to kind of solve the the full scope of your task.
00:14:58.080 --> 00:15:09.360
Uh whereas a Claude, like, you know, you can let it run for a while, but if it's just beyond the scope of where it's gonna cross, maybe like 300k tokens, and you you let it go without compacting, it's gonna start falling apart.
00:15:09.519 --> 00:15:12.240
But then if you let it compact too, it'll probably forget some things.
00:15:12.320 --> 00:15:17.120
Um, I know in cursor they have a better compaction system than what you get in Claude Code.
00:15:17.279 --> 00:15:21.039
Um, so I think you know that's something to consider as well, like the scope of your plan.
00:15:21.200 --> 00:15:25.120
I think in general, like you don't want to be doing huge, huge, huge one-shot plans.
00:15:25.200 --> 00:15:27.600
Like, I don't think that's you know the best practice at the moment.
00:15:27.759 --> 00:15:33.279
Um, but I'm curious like how that colors your experience and yeah, and I'm I'm curious as well, Ray.
00:15:33.440 --> 00:15:34.480
Afterwards, yeah, tell me.
00:15:34.720 --> 00:15:45.120
I I would agree with you, like there is a limit to how much you need to push, and I'm actually still of the mindset that 200k and under is pristine, which is why I don't use a lot of MCP.
00:15:45.440 --> 00:15:48.559
I do not like to pollute my context anything more than I have to.
00:15:48.720 --> 00:15:50.559
So I'm very much on that side.
00:15:50.720 --> 00:15:53.840
Uh, I do agree also that compaction is better in cursor.
00:15:53.919 --> 00:15:58.159
I find that it it feels very natural and it doesn't it doesn't compact so much.
00:15:58.240 --> 00:16:01.679
You'll actually watch it and it'll only maybe reduce it like 30 or 40 percent.
00:16:01.919 --> 00:16:05.279
So it's not like a really aggressive compaction, which I think is good.
00:16:05.519 --> 00:16:12.799
And I also agree that like if you were to try to do something where you're generating a giant PR to go like that is useless.
00:16:12.960 --> 00:16:19.279
So the plans that I I typically are working on try to get isolated to two or three files or one new feature that's going in.
00:16:19.519 --> 00:16:23.759
So I'm not trying to do like an entire like application from beginning to end.
00:16:24.000 --> 00:16:30.879
Um but I I still go back to I've done some amazing things with Opus, and it has just built that trust with me.
00:16:31.120 --> 00:16:39.919
And the way that I actually can build that trust with another model is I actually just need to like go and limit myself to just go use that and figure out how it works.
00:16:40.000 --> 00:16:42.240
Like pretend Opus doesn't exist for a week.
00:16:42.399 --> 00:16:45.120
I only can use GPT models and see what I can pull off.
00:16:45.200 --> 00:16:47.120
Maybe I'll do that this coming week, see what happens.
00:16:47.440 --> 00:16:48.000
It's worth trying.
00:16:48.159 --> 00:16:50.480
I mean, from my experience, GPT-5.4 is great.
00:16:50.639 --> 00:16:56.639
I think the only thing is like you have to just be a little bit more specific with how you talk to it because it'll take things more literally.
00:16:56.879 --> 00:17:01.120
But I find that like generally if you want code that works, GPT is better.
00:17:01.279 --> 00:17:12.079
The one downside though is if you kind of let it run too autonomously and figure things out on its own, it tends to produce a lot of complexity and it writes a lot of useless tests that you have to kind of pull back and clean up after.
00:17:12.240 --> 00:17:12.799
It loves stuff like that.
00:17:12.880 --> 00:17:13.920
That's like the one downside there.
00:17:14.160 --> 00:17:16.000
It loves like literally guard things.
00:17:16.079 --> 00:17:17.839
I'm like, bro, just chill, just chill for a second.
00:17:17.920 --> 00:17:18.079
Yeah.
00:17:18.400 --> 00:17:25.519
Well, both the models are allergic to breaking things, so they're both add all these complex backwards compatibility fallbacks, they're always doing that.
00:17:25.680 --> 00:17:27.440
Um, yeah, it's it's a lot.
00:17:27.519 --> 00:17:32.000
You gotta really monitor what these models are doing, or your code base becomes the model's code base very quick.
00:17:32.160 --> 00:17:32.640
So yeah.
00:17:32.960 --> 00:17:42.880
Yeah, I think I wanted to add on the cursor thing that I noticed is because you can reference past conversations in in that compaction things of what they do is that they basically just compact it.
00:17:42.960 --> 00:17:48.480
Uh they jettison your your messages, you know, the all of the agent messages into JSON L files.
00:17:49.119 --> 00:18:02.640
And so it looks like later on, you know, they from a compaction standpoint, uh they just you know keep some type of context going and then they're you know going back in at certain times and reading back in the relevant, you know, pieces that they need for uh their thing.
00:18:02.960 --> 00:18:06.480
But it's kind of crazy how much it's continued to evolve.
00:18:06.559 --> 00:18:10.319
And I I think the the speed is is really interesting.
00:18:10.480 --> 00:18:13.119
It's um it's like wearing different gloves, you know.
00:18:13.200 --> 00:18:27.119
Like you have this glove that will do this thing, and like you know, I used to play baseball, so like you know, pitcher's glove, catcher's glove, you know, there's those different uh things that you use them for uh depending on your use cases and teams, and I feel like that um i it goes all over the place.
00:18:27.279 --> 00:18:40.079
I've been actually um I'm signing back up again for the ChatGPT Pro specifically because I'm doing this massive architecture thing where uh I want to have all my agents talking to each other through like the Apple Watch, even though I don't wear it anymore.
00:18:40.160 --> 00:18:41.920
I'm I actually gotta get it back out.
00:18:42.160 --> 00:18:46.400
Um the Apple Watch has been a great forcing function to build UIs from, actually.
00:18:46.559 --> 00:18:54.079
It's like a perfect thing for agentic UI thinking because it really is just basic elements, you know, it's just a tap, a touch, and a microphone.
00:18:54.319 --> 00:18:56.640
And so you can do a lot just with those simple gestures.
00:18:56.799 --> 00:19:03.359
And um, you know, like one thing is like if you raise your wrist, you only have like two to three seconds to do something before your arm gets tired.
00:19:03.599 --> 00:19:09.200
So if you start to think about UI building in that space, you start to eliminate a lot of things in your UI.
00:19:09.440 --> 00:19:12.079
And then you start to focus on like what use case do I want?
00:19:12.240 --> 00:19:20.000
And so that type of thinking has been really cool because now uh I'm building a thing that's gonna track all my agents, like an open claw, all the things that it's spawned out.
00:19:20.079 --> 00:19:24.000
Uh it's gonna track like codecs, what things have I kicked off, clawed and so forth.
00:19:24.240 --> 00:19:33.119
And so, you know, I've kind of architected it in a way that I'm gonna have like this massive plugin system, so each of them kind of work independently and so forth, but I want to get a good reviewer on it.
00:19:33.440 --> 00:19:37.839
So I know I was gonna spit up repo prompt and then the chat GPT Pro thing.
00:19:37.920 --> 00:19:39.119
So I'm gonna hit you up, Eric.
00:19:39.359 --> 00:19:44.160
And uh we're we're gonna try to figure this out because I wanna like that's what I want to use the pro plan for.
00:19:44.240 --> 00:19:56.880
And and I think from what I understand from what you say, um right now it's just a bunch of like one you know, giant couple, like two to three markdown files that really describe the spec that I'm trying to build and each of these little details for the components.
00:19:57.119 --> 00:20:00.400
Um and like how would how would I approach this, Eric?
00:20:00.480 --> 00:20:04.880
Like, how should I, you know, it because it is it the the file is growing bigger, right?
00:20:04.960 --> 00:20:08.400
As I as I add more things, I start to separate my files, or like what's what do you think I should be?
00:20:08.720 --> 00:20:10.319
I mean, I can start building.
00:20:10.720 --> 00:20:14.240
I'll encode I have is like a basic HTML prototype that clicks through and shows what I want.
00:20:14.319 --> 00:20:14.960
But like, yeah.
00:20:15.279 --> 00:20:28.480
I mean my my advice is like don't over spec up front and like try and like start building, but laying down good foundations so that it's scalable and like constantly gut check, like read the code or like ask the agents like how can we make this more scalable?
00:20:28.640 --> 00:20:34.160
What can we do in terms of here's what I want to add eventually, like how can we lay down some groundwork so that that's possible?
00:20:34.400 --> 00:20:40.079
I think setting up projects is actually like a huge challenge with agents because they kind of just kind of slop it out.
00:20:40.160 --> 00:20:50.160
Um, and you have to really be disciplined because you know the the the bones of your project are what kind of determines the architecture going forward and how the project sprawls out and evolves.
00:20:50.319 --> 00:21:05.359
So, my my advice is like try and plan small early on, get the core functionality in, and then like once you have it working, like start adding in the pieces one by one that you need to add and try and clean up the architecture as you go so it not just like run it off and go.
00:21:05.519 --> 00:21:09.279
And once you have those clean bones, then you can kind of start running a little further and further.
00:21:09.440 --> 00:21:17.279
But like you want to make sure that like up front you're you're taking short sprints so that like you're able to kind of just make sure that your code is growing healthily.
00:21:17.599 --> 00:21:19.119
Yeah, that's uh great advice.
00:21:19.359 --> 00:21:25.839
I yeah, I'd add uh pretty similar, like get end-to-end as quickly as possible, and I think that's what you mean, Eric, by bones.
00:21:26.240 --> 00:21:30.960
There's too many people I see like not doing that, and then you end up in just a mess.
00:21:31.119 --> 00:21:35.200
Like end-to-end as quick as possible, get the framework in place and then go.
00:21:35.680 --> 00:21:46.559
Yeah, because then you might you might have like described some features and then you didn't really think through how they interact, or you might get like some some things that that kind of sound like what you wanted but aren't actually correct.
00:21:46.640 --> 00:21:47.440
And that's right.
00:21:47.680 --> 00:21:59.440
And I'd also invest a lot in finding ways to kind of API fy your code base uh so that like models can kind of call into it and test it headlessly without having to run through the whole UI.
00:21:59.519 --> 00:21:59.759
I think.
00:22:00.160 --> 00:22:08.160
That's like really powerful stuff now, like making a CLI for your app, like finding ways to kind of poke around the functionality and and create good test plans.
00:22:08.319 --> 00:22:17.599
Um, I mean, obviously, general unit testing frameworks are great, but like often having the whole machinery running is really helpful, and being able to call into the things that you need are is very helpful to test.
00:22:17.839 --> 00:22:23.519
So, um, yeah, closing the loop is very important, but also getting your hands dirty, playing with it, and seeing how it feels.
00:22:23.599 --> 00:22:31.279
Like the models can't do that well right now if there's a UI involved, and you need to be able to see how it feels and what looks like you know, what looks correct and what's missing.
00:22:31.440 --> 00:22:33.920
So the sooner you can get to that the better.
00:22:34.160 --> 00:22:34.480
Yeah.
00:22:34.720 --> 00:22:35.279
Sweet.
00:22:35.599 --> 00:22:39.519
Um, I I was just uh looking at this claw dispatch thing.
00:22:39.680 --> 00:22:44.640
Um, you know, we we that this just came out, and I know that the two of you have been playing with it a lot.
00:22:44.799 --> 00:22:49.680
So, you know, Adam, why don't you tell us like what it is and like how it's been affecting your day-to-day.
00:22:50.319 --> 00:22:57.599
Yeah, I actually I had a friend message me and say, like, hey, I've been using Claw Dispatch, you gotta check it out.
00:22:57.759 --> 00:23:01.920
Because I had heard about it, and you know, there's so much AI news, you can't really keep up on it all.
00:23:02.079 --> 00:23:04.240
And he was telling me what he was doing with it.
00:23:04.480 --> 00:23:06.720
And I was like, man, that actually sounds pretty cool.
00:23:06.960 --> 00:23:11.119
So what I've been using it for, well, first let me start with what it is.
00:23:11.200 --> 00:23:21.920
So within Claude Cowork, or in the Claude uh desktop app, you can actually go into cowork and you can there's a dispatch uh menu item on the left side.
00:23:22.079 --> 00:23:31.599
If you click on that, you basically can create an asynchronous task, and that asynchronous task will run to whatever completion, as long as you have the permission set up right.
00:23:31.839 --> 00:23:36.880
So for example, I've hooked up like Google Docs to it, and I have hooked up a coup a couple other connectors.
00:23:37.119 --> 00:23:57.759
So I can actually uh go through and I say, hey, I want you to do this research, I want you to link where you found it, I want you to build a spreadsheet, I want you to uh go ahead and put it in Google Drive for me, and then I'll come back in the morning and it'll have the research done for me, everything in a spreadsheet, all the formulas and things set up for me.
00:23:58.000 --> 00:24:03.119
It's actually like incredibly, incredibly nice because um it just runs.
00:24:03.279 --> 00:24:07.519
It runs in the background, and I go to bed, come back in the morning, and it's done.
00:24:07.680 --> 00:24:20.960
I've actually used it to create projects too, so I did it to create a few like task management oriented apps that I'm trying to do for this business that my my kids are starting up, and it's been super nice.
00:24:21.119 --> 00:24:24.640
I also Ray brought this up, but I also did use it for my taxes too.
00:24:24.799 --> 00:24:34.799
So I had some um a bunch of like individual receipts and uh things from credit cards.
00:24:34.880 --> 00:24:37.759
I had it like bring together, like, okay, this receipt matches this.
00:24:37.920 --> 00:24:40.319
Here's the categorization of things for me.
00:24:40.559 --> 00:24:42.160
It's it's incredible actually.
00:24:42.240 --> 00:24:43.039
I really like it.
00:24:43.200 --> 00:24:44.559
I Eric, you mentioned this.
00:24:44.640 --> 00:24:45.680
It is buggy at times.
00:24:45.759 --> 00:24:51.680
There have been times I've come back and for whatever reason it failed to create the Google Doc, and I had to like prompt it a little bit around it.
00:24:51.839 --> 00:24:54.480
But when it works, man, it is it is so nice.
00:24:54.799 --> 00:24:58.960
And how do you notice on the side that it like kicks off this little thing?
00:24:59.119 --> 00:25:02.880
It seems like its own separate context window to achieve the task that you're doing.
00:25:03.119 --> 00:25:03.279
Okay.
00:25:03.519 --> 00:25:10.319
It's got like a little like um almost like a project view where you can actually see all the context and the things that it's building up.
00:25:10.400 --> 00:25:13.200
You can even have have files like within its context of it.
00:25:13.279 --> 00:25:14.160
It's very, very nice.
00:25:14.480 --> 00:25:14.720
Yeah.
00:25:15.039 --> 00:25:25.039
So I'm curious, is it is it mainly the UI that you're finding you know appealing for this work, or is it like you know, just the experience of using it significantly different versus like cloud code directly?
00:25:25.599 --> 00:25:29.920
For me, for me, I gotta say, it's mostly the UI, like, and it's built.
00:25:30.640 --> 00:25:40.400
So in my day, in my day job, I always think about there's there's agents that you're interacting with to get a job done, and then there's autonomous agents.
00:25:40.559 --> 00:25:58.079
What this feels like to me is a purpose-built system for autonomous agents, where you give it a single job and it'll run it to completion without constantly asking for feedback along the way, which sometimes you want the autonomous nature of AI, and I think they've done a good job wrapping that up here.
00:25:58.799 --> 00:26:04.000
Yeah, to me, I would typically set up a cloud project and then keep a chat going.
00:26:04.400 --> 00:26:11.359
But the problem is like I would have to constantly create handoff to for all my different chats, even though I had a project folder, right?
00:26:11.759 --> 00:26:14.799
And then when I would go create a new chat, I would say, can you reference the other chat?
00:26:14.960 --> 00:26:23.440
And it would be, you know, recent in the last month they started to do that, you know, with the introduction of memory, so that within one project you can reference all of your chats.
00:26:24.000 --> 00:26:27.279
But with cowork, it just kind of like that's your project.
00:26:27.359 --> 00:26:30.319
Like that's the whole thing just in one big ass chat.
00:26:30.640 --> 00:26:37.039
And um then you see it launching off these different, you know, basically it's a sub agent in essence to achieve your task.
00:26:37.359 --> 00:26:38.720
And it has all the context for it.
00:26:39.119 --> 00:26:40.400
And so you just keep ripping.
00:26:40.880 --> 00:26:48.000
Um and so you you can still create a new a new thread or a new whatever, and then you you have a new like basically a different project that you're working on.
00:26:48.480 --> 00:26:52.880
And that that type of workflow is kind of what I actually appreciated about OpenClaw.
00:26:53.039 --> 00:26:59.440
It's like you could just have this thing that runs forever and you don't have to think about compactions or anything, and it does whatever it does.
00:26:59.519 --> 00:27:02.160
But to me, this is like less debugging.
00:27:02.720 --> 00:27:04.000
I know it's gonna pick up the things.
00:27:04.160 --> 00:27:08.720
I don't have to like I didn't spend any time saying, okay, now go pick up this chat or do this.
00:27:08.799 --> 00:27:10.079
It was just kind of like cool.
00:27:10.400 --> 00:27:11.680
Now let's build a spreadsheet.
00:27:12.000 --> 00:27:16.960
Okay, now if I my accountant was looking at this, what type of things will would she want to see?
00:27:17.039 --> 00:27:20.480
You know, and then it's just like and uh don't clip this anyone.
00:27:20.559 --> 00:27:22.720
But basically, finance bros are cooked.
00:27:23.759 --> 00:27:25.440
It's an incredible job.
00:27:25.599 --> 00:27:29.359
It's there's a skill for writing Excel files that I forgot that Claude had.
00:27:29.599 --> 00:27:30.480
It's so good, yeah.
00:27:30.559 --> 00:27:31.359
It's good.
00:27:31.759 --> 00:27:32.240
It's crazy.
00:27:32.559 --> 00:27:36.880
To add on to that, Ray, like one thing I was trying to do is like for me, the hardest thing is like meal planning.
00:27:37.039 --> 00:27:38.559
I know this sounds ridiculous, right?
00:27:38.799 --> 00:27:41.440
Meal planning, getting the grocery list together.
00:27:41.599 --> 00:27:43.599
So like I was like, okay, plan a meal.
00:27:43.759 --> 00:27:45.519
Here's the things that I've done in the past.
00:27:45.680 --> 00:27:47.359
So it did that, build a spreadsheet on it.
00:27:47.440 --> 00:27:50.319
And I said, go to Walmart.com, add everything to the cart.
00:27:50.480 --> 00:27:55.200
And it it added a bunch until Walmart realized it was a bot operating it, and it blocked it.
00:27:55.279 --> 00:27:56.480
So it's like, dang it.
00:27:56.559 --> 00:28:00.160
Like we need websites that are built for agents to interact with.
00:28:01.839 --> 00:28:02.319
We do.
00:28:02.400 --> 00:28:04.079
We need a Walmart CLI.
00:28:05.680 --> 00:28:06.880
I you know it's crazy.
00:28:06.960 --> 00:28:08.240
The grocery list thing is really nice.
00:28:08.319 --> 00:28:19.759
Like, there's a lot of little tedious things that uh you end up doing day-to-day that like if you find if you take the time to try and like let these agents automate it, like you can actually get like you know, a lot of time saved and quality of life gained back.
00:28:19.839 --> 00:28:26.319
Like it's very tedious to like pick a recipe, pull out the ingredients, make sure that you get the right amounts and you you get everything on that list.
00:28:26.400 --> 00:28:32.720
Like that's a lot of stuff that's wasted time, and it's nice to have that kind of automation for it, especially if you don't have to think about it and your cart's auto-filled.
00:28:32.799 --> 00:28:34.160
Like, I mean, that's a dream for sure.
00:28:34.480 --> 00:28:35.599
That's very cool.
00:28:35.839 --> 00:28:38.240
Um, so you know, like that's so that's great.
00:28:38.319 --> 00:28:39.440
You guys are are diving into it.
00:28:39.519 --> 00:28:45.279
Are you finding you using it a lot from your phone, like with the with the tools around that, or like you're just mostly using it on the computer?
00:28:46.240 --> 00:28:47.839
Probably 50-50 for me.
00:28:48.079 --> 00:28:49.920
Um yeah, really.
00:28:50.160 --> 00:28:51.599
I'm curious if if you're the same.
00:28:52.160 --> 00:28:54.319
Uh it's mostly been because it's on the laptop.
00:28:55.279 --> 00:29:01.119
Even though I have it on the Mac Mini, like all my financial stuff, my Mac mini is literally just for my little claw guy.
00:29:01.359 --> 00:29:03.920
I was like, no, no, no, no crazy stuff going on in there.
00:29:05.279 --> 00:29:22.240
Like I kind of have some isolation, but I want to, it's making me think whenever M5 stuff comes out, you know, with this Apple announcements, if they get an M5 Mac Mini, you know, I'm gonna that's gonna be my beast mode machine with like, you know, running all this like headless stuff and all this cool things that I want to really just bump it up a whole notch.
00:29:22.319 --> 00:29:24.079
Yeah, because I I like that system a lot.
00:29:25.519 --> 00:29:26.000
All right.
00:29:26.240 --> 00:29:27.440
Well, that's cool.
00:29:27.599 --> 00:29:31.440
I I do worry a little bit about overdoing on the uh agent side with your phone.
00:29:31.599 --> 00:29:44.640
I do find like it's a good it's like a little bit of a vampire, you know, if you're already over spending too much time on your computer, especially with the coding agents, that you start kind of delegating and managing agents constantly from your phone, like you know, it's just quite the suck of uh energy and attention.
00:29:44.720 --> 00:29:47.200
It's a bit of a bit of a concern that I I see going forward.
00:29:47.440 --> 00:29:48.960
Curious if you all feel the same about that.
00:29:49.119 --> 00:29:49.599
Yeah.
00:29:50.160 --> 00:29:51.119
I definitely do.
00:29:51.359 --> 00:29:52.319
Just to add on to that.
00:29:52.480 --> 00:29:55.359
Yeah, I mean, as a family man, you know, you got all that concern going on.
00:29:55.519 --> 00:29:56.319
Um definitely.
00:29:56.559 --> 00:29:56.960
Yeah, yeah.
00:29:57.039 --> 00:29:58.720
I've talked to a lot of people who feel that way too.
00:29:58.799 --> 00:30:01.680
Yeah, I think it's just uh do you think it's a phase?
00:30:02.000 --> 00:30:04.240
Like I'm just curious, like you know?
00:30:04.400 --> 00:30:10.400
Okay, no, I think like more and more the more automation this stuff comes in, the more people are gonna want to do on their phones.
00:30:10.559 --> 00:30:11.839
Like you see the draw for it.
00:30:11.920 --> 00:30:21.759
Like, I I could see the next generation, they're they they're just computerless, they're connecting to cloud agents, and they're just doing all their programming from their phones, like never looking up at a bigger screen, just drawing their eyes.
00:30:21.839 --> 00:30:23.039
It's it's terrible.
00:30:23.279 --> 00:30:27.680
I know it's not gonna be phone terminals, but it's it's not gonna be like I don't know.
00:30:27.839 --> 00:30:31.359
It's it's messaging apps, you know, the the form factor solved it on the phone.
00:30:31.920 --> 00:30:32.319
Let's go.
00:30:32.559 --> 00:30:33.599
Oh man, yeah, to the app.
00:30:33.680 --> 00:30:34.160
Oh jeez.
00:30:34.400 --> 00:30:37.440
We all are gonna need like a day where we just like don't use a phone at all.
00:30:37.599 --> 00:30:41.279
Like I think every every p person's gonna have to get to that point where it's like no phone day.
00:30:41.440 --> 00:30:44.240
Yeah, dangerous to yeah.
00:30:44.480 --> 00:30:46.720
Uh all right, I wanted to switch gears a little bit.
00:30:46.880 --> 00:30:53.759
Um, you know, we were talking a little bit before about you know what it's like to design agents, uh, you know, not for coding.
00:30:53.839 --> 00:30:58.079
And I want to kind of dive into that a little bit with you, Adam, because I know that's what you've been working on.
00:30:58.240 --> 00:31:06.559
Um so why don't you tell us a little bit about like the kind of work you're doing there with with these agents and like what kind of stuff you're thinking about when designing these agents for the for that work.
00:31:07.200 --> 00:31:07.839
Yeah.
00:31:09.359 --> 00:31:16.400
If you don't know, like my company that was acquired recently, we we were focused on building agents around the marketing space.
00:31:16.640 --> 00:31:31.039
So I've uh very fortunately had the opportunity to learn both the hard way uh from you know customer feedback that was not great, to like actually getting good customer feedback, like what works and what doesn't work from an agent perspective.
00:31:31.279 --> 00:31:41.359
But let's extrapolate that out because like if you think about the marketing hype around agents today, the promise is that it's going to basically automate everything that we do.
00:31:41.519 --> 00:31:51.359
Like it and let's start if we apply that to any vertical, banking is one that I'm familiar with, marketing is one I'm familiar with, accounting is one I'm familiar with.
00:31:51.599 --> 00:31:56.079
But then you start thinking about the the future of point solutions going away.
00:31:56.319 --> 00:32:04.640
So that the idea would be that instead of actually having to buy 10 different applications, you buy one that can basically run your entire business.
00:32:04.799 --> 00:32:07.519
So like each of us have businesses that we run.
00:32:07.680 --> 00:32:20.000
You can kind of imagine like a an agent that does everything you need to do from the marketing you need to do, the accounting you need to do, the payroll, the taxes, the operations, the management of tasks.
00:32:20.160 --> 00:32:23.680
Think about the context switching that has to happen in that world.
00:32:23.920 --> 00:32:27.680
So I don't want to get into too many specifics, but I'm just kind of like seeding the problem here.
00:32:28.240 --> 00:32:31.039
Coding agents, it's very like focused.
00:32:31.200 --> 00:32:39.680
You're working with a very small set of tools, and then what people do is they plug in MCPs to try to expose that, and people optionally bring those in.
00:32:39.920 --> 00:33:04.559
But if you're if Eric, you're running your business and you need to figure out your marketing plan right after you're talking about like your accounting and finances, right after you ran payroll, and all of this is being run agentically, whether a combination of autonomously or interactive, the problem really becomes all around how you manage the context of things and how you manage what tools and subagents are actually available at any given time.
00:33:04.880 --> 00:33:17.680
So the architecture that I've been building out around this is what we need to do in those types of systems is we're going to have to have a much thought more thoughtful process around building what I've been calling a context engine.
00:33:17.839 --> 00:33:26.480
Now, the and and this isn't a new thing, but it actually becomes more important as the scope of the things your agent needs to do scales.
00:33:26.640 --> 00:33:30.640
Because we all know you add a hundred tools to an agent, it's gonna be bad.
00:33:30.799 --> 00:33:33.839
So you can't have a hundred tools registered at any given time.
00:33:34.000 --> 00:33:36.640
So what you need to do is you need to have this intent.
00:33:36.880 --> 00:33:41.519
You need to have a way to like, based on what the user is starting a conversation with, what is their intent?
00:33:41.680 --> 00:33:50.079
How do you take that intent, map it to what skills and what tools and what context need to be actually built together to start that conversation?
00:33:50.400 --> 00:33:58.480
Then the the problem becomes, okay, you've got this really great, ideally, really great framework to now be able to do whatever job.
00:33:58.559 --> 00:34:00.640
So, like, Ray, maybe you're doing your taxes.
00:34:00.799 --> 00:34:08.719
So the context engines built that, but you get to the end, you're like, okay, now that I've done my taxes, I actually want to go do something completely unrelated.
00:34:08.880 --> 00:34:14.960
How do you actually switch over to that new um that that new batch of like context?
00:34:15.039 --> 00:34:16.000
How do you rebuild that?
00:34:16.159 --> 00:34:27.920
So I think there's like tooling that actually has to be built to manage these cross-intent activities while carrying along enough context to actually be able to uh to actually transition into that.
00:34:28.000 --> 00:34:39.280
Because some of the things you might have been doing in that initial uh conversation may carry into that second one, but the tools, the skills, everything that's needed to do that second job, completely different.
00:34:39.519 --> 00:34:42.960
And then what you end up with is like that is a very interactive, right?
00:34:43.039 --> 00:34:58.159
So that you're interacting with the chat, you're talking back and forth, but there is also the idea of proactive agents, and we were talking a little bit about it with dispatch, but let's and and we were talking a little bit about it before the show with like cron jobs that you can actually trigger to heck go have things run.
00:34:58.320 --> 00:35:02.320
There's the difference between interactive agents and autonomous agents.
00:35:02.559 --> 00:35:09.519
So you also need to manage the idea that you have autonomous agents that do something for you.
00:35:09.679 --> 00:35:17.760
That's so it basically does all the context engineering and it pulls you in, but you can jump in later and interact with what that autonomous agent did.
00:35:18.000 --> 00:35:20.079
So that's the model that I've been building out in my head.
00:35:20.159 --> 00:35:22.880
I'll I'll pause there because I I think I threw a lot at you guys.
00:35:23.199 --> 00:35:23.679
Yeah, yeah.
00:35:23.840 --> 00:35:24.800
So I'm curious.
00:35:24.960 --> 00:35:30.639
Uh so you're splitting off uh you know this work in terms of managing the context for spitting off these other agents.
00:35:30.800 --> 00:35:35.119
Um, and how do you basically decide like what models are going to do what?
00:35:35.280 --> 00:35:38.400
And basically, how are you evaluating the success rate?
00:35:38.480 --> 00:35:43.760
Because I know a lot of the benchmarks we're kind of relying on when we're looking at these models are very coding oriented.
00:35:43.840 --> 00:35:51.440
There's some that are basically computer use or tool calling oriented, but like generally that's like a little vague and it's hard to see like what are good success metrics.
00:35:51.519 --> 00:35:59.199
Like, you know, you're talking about intent inference and all that stuff is super important, especially for more non-technical users trying to manage their business using these things.
00:35:59.360 --> 00:36:03.760
So, like being able to decipher what the what the user actually wants, that's like a huge part of it too.
00:36:03.920 --> 00:36:08.960
So, how are you thinking about it in those terms and and how are you evaluating the success metrics there?
00:36:09.280 --> 00:36:11.280
Yeah, so you you're dead on.
00:36:11.440 --> 00:36:16.639
And some of the stuff that I've talked about is kind of subjective to like what is good in some of these situations.
00:36:16.880 --> 00:36:19.920
The the key thing is to measure, to start with measuring something.
00:36:20.079 --> 00:36:32.400
So I would always say like what we're doing, what we try to do is have some set of golden prompts that we're consistently testing against, ideally automated in a way that gets the outcome that you're anticipating at some threshold.
00:36:32.480 --> 00:36:32.719
Right?
00:36:32.800 --> 00:36:35.199
Maybe your threshold is 70%, maybe it's 80%.
00:36:35.679 --> 00:36:42.320
But the main thing is you need to understand when you make a change how much it's impacting the performance of some set of data.
00:36:42.559 --> 00:36:47.199
Now, that that golden set of uh prompts should change over time as you get user feedback.
00:36:47.440 --> 00:37:03.119
Because what you thought people were gonna do with your system and what you theorize around it may not be what actually happens, which goes back to that thing we were talking about earlier, Ray, which is getting end-to-end is so vital, especially getting it end-to-end where a user can actually use it, even if you're slightly embarrassed by it.
00:37:03.280 --> 00:37:08.880
Because you can actually start gaining that information where you're like, I thought people were gonna ask these three things.
00:37:09.039 --> 00:37:10.559
That is not at all what they're doing.
00:37:10.800 --> 00:37:14.800
We actually need to tune the system to be more geared towards these types of things.
00:37:14.960 --> 00:37:29.440
Uh, but it really is a series, it's a having that starting set of prompts, even if they change over time, but just so you can measure, so you can make changes without risking things, evaluating the outcome of that, and then iterating on what that golden set of prompts are.
00:37:29.760 --> 00:37:31.280
Yeah, it's it's interesting.
00:37:31.440 --> 00:37:38.559
Um, I I've actually been working on in my own workflow in Repo Prompt a system for exporting a prompt for another model.
00:37:38.639 --> 00:37:40.719
So, like when you're trying to pull GPT Pro.
00:37:40.960 --> 00:37:47.679
So there's like this layering of like a model calling another model to gather context and then exporting the result of that.
00:37:47.840 --> 00:37:55.599
And I was getting some user feedback from people who are like, hey, like the end result when I give it to GPT Pro, it's just giving me a prompt.
00:37:55.679 --> 00:38:02.079
And I could see that like when users were asking the the system, they were saying, Hey, make me a prompt to do this.
00:38:02.159 --> 00:38:03.360
That's what they were asking for.
00:38:03.519 --> 00:38:10.480
And then the model was like, Okay, great, I will prompt the task to be to make a prompt, but the system was already tasked with making a prompt.
00:38:10.639 --> 00:38:14.400
So it was just redundant and and people were just not asking the right thing.
00:38:14.480 --> 00:38:27.840
And so, like, I had to go in and prompt the system to actually extract the intent from the question to then funnel it in the right way to get to the actual outcome, which is to make a plan or execute a code review.
00:38:28.000 --> 00:38:36.800
And so that's like something that like I wouldn't have thought to think about because I'm just using it and it's working fine for me because I'm asking for a plan because I know what what to ask of it.
00:38:36.960 --> 00:38:42.719
But it's interesting how you know people adding these extra layers of of redundant prompting are causing issues.
00:38:42.880 --> 00:38:46.639
So I'm curious if that's something that like you're running into as well when you're you're thinking about these things.
00:38:47.760 --> 00:38:49.440
All the time, 100% of the time.
00:38:49.599 --> 00:38:56.159
Like it's uh what's amazing about this space, honestly, is it's so unsolved, and I love it.
00:38:56.400 --> 00:39:00.960
And and a lot of times people are getting paralyzed by like, well, what if it does a bad thing?
00:39:01.119 --> 00:39:02.079
Like, what if it does something wrong?
00:39:02.239 --> 00:39:03.920
What if it does something you didn't expect?
00:39:04.159 --> 00:39:05.440
Well, yeah, it's gonna do that.
00:39:05.519 --> 00:39:08.800
Like, that's AI, like it's probabilistic software.
00:39:08.960 --> 00:39:17.920
We have to change the way we are approaching these things into like, let's build a theory on what we think people are gonna do, and we shouldn't just make this up, like we should base it on real data if we have it.
00:39:18.480 --> 00:39:21.840
But then we need to iterate and be okay with iterating, exactly like you said.
00:39:21.920 --> 00:39:26.880
We've had we have that stuff happen constantly at Raylead where people are, I'm like, well, this works great.
00:39:26.960 --> 00:39:32.800
Like our but then it's like that's not at all how people were talking to it, and they were putting things in there that were very different.
00:39:32.960 --> 00:39:37.199
Yeah, and depending on the model, depending on the AI model, it will interpret it different ways.
00:39:37.360 --> 00:39:39.039
You also have that layer of it to deal with.
00:39:39.360 --> 00:39:49.840
Yeah, and if you have models talking to other models, then that layer of chaining and the prompting leads to areas where like you might actually have compounding misunderstandings that just really astray.
00:39:50.159 --> 00:39:52.800
So it's a really challenging thing when you're orchestrating models like this.
00:39:52.960 --> 00:39:59.360
Um, I'm curious on the architecture and like the problem solving piece of it.
00:39:59.440 --> 00:40:04.400
I feel like people who are running businesses right now are really curious about the agentic coding.
00:40:04.559 --> 00:40:12.800
People are thinking, let me just go to open claw because people are talking about that, and all the YouTube videos are showing, you know, how they've transformed their businesses overnight.
00:40:13.039 --> 00:40:17.119
And you know, there's like a lot of hype around that, and then there's like the new Hermes model and so forth.
00:40:17.199 --> 00:40:22.639
And you know, Eric, you're just basically um and and you know, Adam, you're like you're just going right into the code and so forth.
00:40:22.800 --> 00:40:32.239
Um, you know, if I go to 11 labs, they have like 11 labs agents that are pretty nice GUI interfaces that just kind of get me started in thinking about you know, a basic agent, right?
00:40:32.320 --> 00:40:35.039
Like they already have the stuff there and it's just plug and play.
00:40:35.440 --> 00:40:57.199
How how should uh what priorities of things that should people like think about first before maybe you know like not only playing with the agents because you just need to learn how they work, but like you it sounds like Adam, you've done a lot of thinking about the frameworks for things and like if you were to distill down maybe to your children and say, Hey, we're we want to run this like cool store, e-commerce store, uh what what part of it can I use as agents?
00:40:57.280 --> 00:40:59.599
You know, I'm gonna try a lot of things and like how would you guide them?
00:41:00.079 --> 00:41:05.280
Yeah, I would say let's say you ru you're in a business and your CEO is like we need to build an agent.
00:41:05.760 --> 00:41:08.960
There are a couple different things that you should start thinking about immediately.
00:41:09.119 --> 00:41:11.199
Literally everybody in the world can build an agent.
00:41:11.360 --> 00:41:13.440
Small agents are incredibly easy to build.
00:41:13.599 --> 00:41:17.920
I I no offense to anybody that believes otherwise, but they really are very easy to build.
00:41:18.079 --> 00:41:20.880
The AI is really good at doing the basic stuff.
00:41:21.119 --> 00:41:22.639
Well, can I just say one thing on that?
00:41:22.719 --> 00:41:34.400
So so just so you know, a basic agent, if you want to just get that working, you just download one of the SDKs from and you can use AI SK, which wraps all of them, but like the agents are basically built into the model now.
00:41:34.480 --> 00:41:39.679
All it is is you have a query and you give it a list of tools, and the model will just call the tools.
00:41:39.840 --> 00:41:42.480
And so you don't have to have a four, there's no loop, there's nothing there.
00:41:42.559 --> 00:41:45.920
It's just the model will call tools and then it will answer, and that's your agent.
00:41:46.000 --> 00:41:47.039
So that's the basics of it.
00:41:47.119 --> 00:41:48.800
It's it's nothing more than that.
00:41:49.039 --> 00:41:53.920
And you're spot on, like literally, there is nothing that's there is no moat there.
00:41:54.480 --> 00:41:56.960
Like it is it is very simple.
00:41:57.199 --> 00:41:59.840
So, what what if you're a business, what sets you apart?
00:41:59.920 --> 00:42:06.639
What is something that you can do that open AI, that Anthropic, that Gemini, that Google or whatever cannot do?
00:42:06.880 --> 00:42:11.119
And ideally, what that is going to be geared around is context and action.
00:42:11.360 --> 00:42:15.679
So, what what is the context that your business has that open AI won't have?
00:42:15.840 --> 00:42:29.280
And if you're if you're a business with a good data foundation, you need to start thinking and theorizing around what are the things that I can do in my agent that actually give better outcomes to my customer that they couldn't get just going to chatGPT.com.
00:42:29.440 --> 00:42:32.079
Otherwise, people are just going to go to chatGPT.com.
00:42:32.400 --> 00:42:40.400
And then the action side of it is like, what is it that your company does with whatever the job to be done that is hard to do elsewhere?
00:42:40.639 --> 00:42:43.280
And how do you make that agentically accessible?
00:42:43.519 --> 00:42:50.800
So, for example, if it's like, hey, I need to, if I want to go design an email, I can go to chatGPT.com today, design an email.
00:42:50.880 --> 00:42:52.960
But if what if I wanted to send that to a thousand people?
00:42:53.119 --> 00:42:55.360
That is an action ChatGPT can't do.
00:42:55.519 --> 00:42:58.960
So those are examples of things that you I would really anchor to.
00:42:59.039 --> 00:43:21.599
And but do not forget the context because a lot of people are jumping into building agents and they have no thought press around the data that needs to go in, how you should actually manage, inject that context into the prompts that actually are going in, but also give the data, give the agent the ability to ask questions about your data so that it can make good reasoning decisions based on the question to be done.
00:43:21.840 --> 00:43:31.519
Yeah, I think another point on that that's really important is how are you designing the tools for kind of retrieving that data in such a way that like it doesn't overwhelm the context window when it returns?
00:43:31.679 --> 00:43:37.519
Like a lot of people's first mistakes when they say, like, oh, let me just have the model query database and it returns like 10 million results.
00:43:37.760 --> 00:43:39.760
Like that's just an overwhelming amount.
00:43:39.840 --> 00:43:48.639
Like the model's gonna try and get broad results, but like, how do you kind of distill down what it needs from that call and try to org optimize the call so that it kind of gets to what it needs faster?
00:43:48.800 --> 00:43:56.239
And I think a lot of this is a lot of iteration, but it's also like formatting output and being clean about like you know, restricting what happens.
00:43:56.400 --> 00:43:59.679
I know cursor's approach is that they just dump that to a file and let the model.
00:44:00.159 --> 00:44:01.119
kind of read from it.
00:44:01.199 --> 00:44:05.440
But in my experience, a lot of the time the model will not necessarily go and open that file.
00:44:05.519 --> 00:44:09.840
Or it maybe it will in some cases, but you know, and then it adds more tool calls to retrieve the data.
00:44:10.000 --> 00:44:18.719
So finding good ways to kind of you know massage the data and return it in a way that is clean and efficient as like goes a long way and it's one of the things that I do with my MCP tools.
00:44:18.960 --> 00:44:20.079
I actually love that Eric.
00:44:20.480 --> 00:44:20.719
Yeah.
00:44:20.880 --> 00:44:22.320
Yeah I I do I love that too.
00:44:22.400 --> 00:44:26.079
The one thing I would also add on this is a theory I haven't actually implemented this part yet.
00:44:26.159 --> 00:44:29.039
Everything else I've talked about I've definitely worked through.
00:44:29.920 --> 00:44:51.440
But I honestly believe code gen, we need to start like we need to start giving agents the ability to start operating and so for example in data analysis if someone asks something like hey I want to understand the propensity of blah blah blah there's no reason we shouldn't give the agent the ability to to actually go do some of that work for us via code.
00:44:51.519 --> 00:45:02.400
That's my theory currently again I haven't executed on it but I think figuring out like the sandbox architecture and making sure we can run things safely is going to be the next like pivotal thing to kind of figure out an agent development.
00:45:02.719 --> 00:45:18.960
So actually on that note what's interesting um is I was having a kind of markdown like the other day I was working with with an agent to take a markdown file and convert into a PDF and I was noticing the agent would go off and and just make a throwaway script and a temp folder and it was just like I'm just going to make this temp script to kind of solve your problem.
00:45:19.199 --> 00:45:23.039
But like I think the temp script part of it is actually the wrong thing to do.
00:45:23.119 --> 00:45:28.719
It's like oh well maybe I'll need to iterate on this script or maybe I want to keep that script around for next time when I need to do it.
00:45:28.880 --> 00:45:31.679
And so pushing the model to kind of have reusable scripts is very important.
00:45:31.840 --> 00:45:40.079
I think that's what like skill files are good for is like you can kind of guide the model write the script store it for later and and just like pushing the model to write this reusable code.
00:45:40.239 --> 00:45:49.760
Like of course the models need to be able to write code to kind of analyze data I think is super important but like if everything's ephemeral they'll forget how they did it and they'll have to relearn it every time that's just inefficient.
00:45:49.920 --> 00:45:53.199
So yeah finding good ways to make things reusable yeah very important.
00:45:53.519 --> 00:45:54.159
Yeah.
00:45:54.559 --> 00:46:35.599
Anything else uh on this like work of dealing with multiple agents especially like I know a lot of people are very obsessed with orchestration these days you know they want to kind of token max and and I'm curious like what you're thinking on uh with that yeah I saw you unmute so I'm gonna give you a chance to jump in here before I I close out um well I I think we will save it for the next episode because I gotta head on out soon and I think there's a I'm gonna have an interview today um with some of the folks who were basically um token maxing and they're from Korea and these guys are just like crazy next level and they uh even the Jeff Huntley the inventor of the Ralph Loop saying that these guys are the token slot masters.
00:46:36.079 --> 00:47:20.079
So for him to give that to them like it makes his Ralph loops they use Ralph in their looping uh type of thing uh he's just kind of has about you know like give them the crown of like wow these guys are yeah thinking about things a little bit differently and we could talk more about the architecture for the next episode so stay tuned folks love it uh yeah just a quick thought on that personally like I think like once you have more than one layer of indirection I think you're kind of overdoing it in my opinion like when you have one agent managing some agents I think they can manage one layer of agents like that's doable and you can kind of still audit what's happening but like a second layer uh to me that's too much and then you're kind of getting away off the rails and I don't I don't think we're there yet in terms of what you're able to do.
00:47:20.159 --> 00:47:27.039
But it's also because like you're not able to have oversight over this and and I I think we're not quite there at the point where slot maxing is worthwhile.
00:47:27.119 --> 00:47:29.119
Like I think you're just burning tokens for the sake of it.
00:47:29.519 --> 00:47:39.840
Don't listen to Jensen who says that if your 500k a year engineer is not spending half a million sorry quarter million a year in tokens that they're they're falling behind like that's just bullshit.
00:47:39.920 --> 00:47:52.639
Like actually someone did the math on that and I don't think like I don't think it's reasonable for anyone to even come close to spending like the amount of tokens you'd need to kind of burn a a quarter million of of in tokens it's just it's just such an insane amount.
00:47:52.800 --> 00:47:59.280
I mean maybe once the next generation these guys do but not one billion tokens per day two billion.
00:47:59.519 --> 00:48:01.119
Yeah that's too much just for one guy.
00:48:01.360 --> 00:48:03.119
Yeah and what are they shipping with that?
00:48:03.440 --> 00:49:36.559
They got like oh they have like the oh my uh codecs and then oh my the oh my guys oh yeah yeah yeah yeah the omai guys yeah yeah yeah they're crazy I mean listen good for them I'm happy for that yeah I'll have more details I would like some of those billions of dollars if they don't mind I'll tell I'll take the 250k you can take you can keep the tokens oh jeez all right well I think that's uh kind of a good place to kind of wrap up for today then so I stuff to think about if you guys all have questions for Adam or myself or Ray uh on on what what it is that we're thinking about in terms of you know multi-agent orchestration or you know wanting to dive deeper on building agents outside of coding as well I think that's like an important thing to start thinking about uh please do drop a comment and let us know uh I want to hear from you yeah any uh closing thoughts there for today Adam on this nope I appreciate you all and uh if you guys have better ideas on how to do any of this stuff drop it in the comments below please yeah we're always learning back yeah yeah yeah yeah make sure to leave those five star reviews we are aiming to be the number one AI podcast so if you watched our first episode Claude Sonnet had claimed to be the number one AI coder uh just because they you know denounced that themselves and of course you can give your own self medals and so uh we're asking you for help as well so five stars all across the board uh whether it's Apple Podcasts Spotify uh make sure you go ahead and do that it actually helps the rankings so we're moving up on our little rankings here and I appreciate all the the reviews have been coming in amazing thank you so much guys and uh hope to hear from you hope you have a great rest of your week everyone cheers later