WEBVTT
00:00:00.000 --> 00:00:11.000
Hey everybody, it is Runtime Arguments, episode number 31, and as always, off by one, because it's our 30-second episode.
00:00:11.000 --> 00:00:23.000
Um… This week, yeah, it's AI again, but this time it's Jim who has been running AI models locally.
00:00:22.000 --> 00:00:24.000
Umm.
00:00:23.000 --> 00:00:25.000
On his own machines! For free!
00:00:24.000 --> 00:00:30.000
Uh, Wolf? Well, yes, yes, that's right, Wolf. Uh, you didn't introduce yourself.
00:00:30.000 --> 00:00:32.000
Oh, crap.
00:00:32.000 --> 00:00:40.000
I'm Wolf, and my best friend here as always is Jim. Say hi, Jim.
00:00:40.000 --> 00:00:42.000
Hey everybody, how you doing?
00:00:42.000 --> 00:00:55.000
This, apparently, off to a weird start. I guess we're gonna… Let's see how it goes. Why don't we start with, how was your week? How was your week, Jim? Or two weeks?
00:00:53.000 --> 00:01:02.000
Oh, I had a nice, uh, I, um, I go through periods of, of, uh, not being very productive.
00:01:02.000 --> 00:01:19.000
And then, fortunately, that's followed by weeks of real productivity. And I've had a couple of really good weeks, and I'm really enjoying it. And, um, you know, I had a thought, you know, we… this whole podcast came out of Wilf and I having lunch every Saturday.
00:01:19.000 --> 00:01:31.000
And this past Saturday, uh, we just had a blast. I think there was five of us there? Uh, the, the group, the group is growing, and, and, uh, it's, it's really a lot of fun. And it.
00:01:24.000 --> 00:01:25.000
There were five.
00:01:30.000 --> 00:01:32.000
Our new friend Patrick came.
00:01:32.000 --> 00:01:44.000
That's right, that's right. And I just got confirmation he's coming again this Saturday, so that's nice. But one of the things, as I was sitting there, I was sort of thinking about how.
00:01:44.000 --> 00:01:56.000
how much I enjoy my smart friends. I enjoy all my friends, but I enjoy having smart people to be around so much because I grow from that.
00:01:56.000 --> 00:02:05.000
And, uh, and Wolf, you're, you're at the top of that list. And, uh, the other guys that show up, um, it's just so much fun. And…
00:01:59.000 --> 00:02:02.000
Oh, you make me blush.
00:02:05.000 --> 00:02:12.000
And I learn and I thrive on that and it's great. So yeah, that was sort of my thought for the week.
00:02:12.000 --> 00:02:14.000
How you doing, Wolf?
00:02:14.000 --> 00:02:18.000
Well, I'm never satisfied.
00:02:18.000 --> 00:02:28.000
Sometimes I'm happy, but I'm never satisfied. And there's a tool I use, and for people who aren't using this, I don't know why you're not.
00:02:28.000 --> 00:02:41.000
Uh, and that tool is… DIRENV. D-I-R-E-N-V. And, uh, DIRENV is a pretty simple tool, super useful, written in Go, but there's some shell stuff that goes along with it.
00:02:41.000 --> 00:02:56.000
And the idea behind direnv is that when you cd into a directory that has the magic file, and the magic file is named .envrc, E-N-V-R-C.
00:02:57.000 --> 00:03:17.000
Um, then DurM does stuff that's in that file, and typically it's easy stuff, like… exporting some environment variables. So, for instance, maybe what you have is a couple of different projects that access different AWS accounts.
00:03:17.000 --> 00:03:29.000
Uh, you could make it be that when you CD into one of those directories, it exports the appropriate AWS credentials for that account.
00:03:29.000 --> 00:03:42.000
Um, and then when you CD out of it, you don't have to do anything, they just go away, they're gone, and you CD into the other directory, and you get the new ones. Um, I love that. Um… I.
00:03:42.000 --> 00:03:49.000
write Python, so I'm always using some kind of project.
00:03:49.000 --> 00:04:05.000
Form shape and I have been for a long time in the old days we didn't use shapes but then there were like gosh there's a zillion but like poetry is what the two things that I lean on most heavily.
00:04:05.000 --> 00:04:18.000
these days, is, um, a normal Python project I will manage with UV. UV is a tool that's super fast, written in Rust, replaces a bunch of stuff.
00:04:18.000 --> 00:04:32.000
Um… It has a way of making sure you've got your dependencies, and they're the right ones, and either executing a tool as though you have activated a virtual environment.
00:04:32.000 --> 00:04:43.000
or just activating a virtual environment. You can do either of those. Um, and then Pixie is another, uh, tool that, um, manages projects. The main difference is.
00:04:43.000 --> 00:05:00.000
if all you care about is normal, ordinary Python packages for your dependencies, your requirements, you would use UV. If some of your dependencies are from Conda, from Anaconda, from that world, then you use Pixie, because Pixie knows how to get both.
00:05:01.000 --> 00:05:07.000
The annoying thing about dir env… Is.
00:05:07.000 --> 00:05:20.000
for almost any kind… and Jiraenv applies to everything. Uh, like, if you're building a Ruby project, or a… whatever. It has layouts. It knows what to do.
00:05:20.000 --> 00:05:31.000
for whatever language or shape project you're working with. Um, and you normally start your .envrc file with, for instance, if it's Ruby, layout Ruby.
00:05:31.000 --> 00:05:45.000
Right? Um… We have an existing one for Python, Layout Python, that assumes you're gonna use the old style of Python version and VN… anyway.
00:05:46.000 --> 00:05:50.000
They don't have layout UV.
00:05:52.000 --> 00:06:08.000
And that means I have to implement my own and keep it around in my config files and then I can reference it, but then my projects aren't as portable because my .envrc sample says.
00:06:08.000 --> 00:06:13.000
Um, layout UV at the top, and there is no layout UV. Ugh.
00:06:13.000 --> 00:06:15.000
friction.
00:06:15.000 --> 00:06:16.000
I hate friction.
00:06:16.000 --> 00:06:26.000
So, it's an itch, and I had to scratch it. Um, there were open tickets, there are people who want this, there's people who explicitly don't want it.
00:06:26.000 --> 00:06:36.000
People provided implementations that had really good stuff in them, and basically… It all stalled in 2024.
00:06:36.000 --> 00:06:47.000
and that, that's not good enough for me. So, I took everything good that the people who came before had, um, I wrote a modern.
00:06:47.000 --> 00:07:06.000
PR implementation and ticket that cited the old tickets, referenced the contributors who had thoughts, addressed the three different groups. The three groups were… NVARC doesn't… Durin doesn't do this, so don't.
00:07:06.000 --> 00:07:23.000
And one group was, this is great, but just activate the venv. Don't try to bring it up to date. Don't install any packages. And the third group was, yeah, do it all. And I addressed each of those three groups.
00:07:23.000 --> 00:07:32.000
what you should get or have. And I submitted the PR. Nobody has noticed yet, so it's been a while, but it's there.
00:07:33.000 --> 00:07:41.000
So I'm pretty excited about that. More exciting to me is we've been talking about DappMux.
00:07:41.000 --> 00:07:48.000
my Debug Adapter Protocol Multiplexer that lets you hook up as many.
00:07:48.000 --> 00:07:55.000
pieces of machinery as you want to do debugging. So, for Python, for instance.
00:07:55.000 --> 00:08:09.000
uh, IPython, plus the debugger, plus this super smart guy, um, named Sean Perry, wrote, uh, uh, DAP Observer. It might be being renamed to DAP Tools.
00:08:09.000 --> 00:08:23.000
Um, and your editor, whatever editor you want, and you hook them all together, and they're all working at the same time. We've talked about this. Well, Microsoft, the place where DAP comes from, has a DAP home page.
00:08:23.000 --> 00:08:26.000
where they list implementations.
00:08:26.000 --> 00:08:34.000
But an implementation is like, I have a debugger, I have an editor, and it does that. They didn't have a page for this.
00:08:34.000 --> 00:08:42.000
So I submitted a PR that says, here's a whole new category of tools, and the first one is DatMux.
00:08:42.000 --> 00:09:05.000
And they accepted my PR. Um, I'm not sure it's… it is merged! They did it, but maybe it's not published, so I don't actually see the page live as of when I last checked, which was yesterday evening or something, or else maybe I didn't understand how to make the menu work or something. I thought it was all automatic.
00:09:05.000 --> 00:09:10.000
Uh, we'll see, I'll keep checking. But yeah, um, it's a public thing now.
00:09:11.000 --> 00:09:27.000
Now if only I could show Julia Evans that it addresses the problem she was facing and blogged about. That would make my day. So that was my week or two weeks. Rewarding, but a lot about me.
00:09:27.000 --> 00:09:32.000
solving problems maybe I shouldn't have bothered with. But I did.
00:09:32.000 --> 00:09:43.000
Yeah, you're on fire, and I love watching that. I see your little posts, I see your posts on Mastodon talking about this stuff, and it's exciting. I like that.
00:09:43.000 --> 00:09:48.000
Um… Let's talk about feedback.
00:09:48.000 --> 00:09:55.000
I hear you have some with respect to IPv6.
00:09:51.000 --> 00:09:53.000
Up.
00:09:54.000 --> 00:10:08.000
Uh, yeah, just a little bit. Back, uh, episode 11, back in September, we talked about IPv6, and it was a fun conversation, a lot of useful information, I think. And, uh, Google just announced, uh, within the last couple of weeks.
00:10:08.000 --> 00:10:14.000
that, uh, more than 50% of their traffic worldwide is now over IPv6.
00:10:14.000 --> 00:10:31.000
I think that's pretty good. I think it's being adopted pretty well. I know I'm using it here at home when I connect between my machines. I'm doing the dynamic DNS thing where I set up a machine. For instance, for this episode, I set up a machine called AI.
00:10:31.000 --> 00:10:47.000
So I can just SSH to ai.local and it finds it. I don't have to set up a DNS entry. I don't even need an entry in my SSH config file or anything. I just, I can just point at ai.local and it connects and the connection is IPv6.
00:10:47.000 --> 00:10:54.000
Now, there's nothing about dynamic DNS that says it has to be IPv6, that's just the one it chooses.
00:10:54.000 --> 00:11:12.000
before IPv4. Um, so that's… that's kind of neat. I… I like to see people adopting IPv6. Uh, I know there's a number of, um… Um, uh, internet providers that don't offer it. Uh, I think yours, right? Well, if you're doing, what's, what's the name? You've got a fiber.
00:11:11.000 --> 00:11:23.000
Because I have carrier-grade NAT, IPv6 doesn't go through my, uh… is it an ONT? I always forget what the name of the fiber box is.
00:11:20.000 --> 00:11:31.000
Well, carrier-grade NAT doesn't restrict you from doing IPv6. They wouldn't need NAT at all if they would just implement IPv6.
00:11:29.000 --> 00:11:35.000
Oh, maybe it's not that I have, uh, CGNAT, maybe it's just that it's my provider.
00:11:35.000 --> 00:11:48.000
Yeah, there's a number of providers that just don't supply it yet, and that's too bad. But hey, it's getting better. Over 50% of the traffic to Google is IPv6, so that's kind of neat.
00:11:47.000 --> 00:11:49.000
I have a little feedback.
00:11:49.000 --> 00:11:50.000
You do.
00:11:50.000 --> 00:11:59.000
I do. And my feedback is, um, that someone was talking to me about the show.
00:11:59.000 --> 00:12:11.000
Um, and they said, uh, that the show, uh, gets them fired up about, um, software engineering, about solving problems. Um, that…
00:12:09.000 --> 00:12:10.000
Nice.
00:12:12.000 --> 00:12:24.000
boy, did that hit me in the feels. I mean, that's what I want. That was exactly a great thing to hear. I was very excited about that.
00:12:17.000 --> 00:12:19.000
Yep.
00:12:24.000 --> 00:12:47.000
And, uh, I'm happy that we're starting to make an impact. I'm not, um… I never assume that it's gonna… anything is gonna go as fast as you think or plan for it to go. Um, but the fact that we have listeners, and you know, we're… We're well over 2,000 downloads, 2,500 or something like that.
00:12:45.000 --> 00:12:48.000
Uh, 30… no, 3,500.
00:12:47.000 --> 00:12:52.000
Oh, 3,500, shit. Oh, crap, you'll have to beat that out.
00:12:48.000 --> 00:13:04.000
Yeah, yeah, we're, I mean, you know, compared to some of the big podcasts out there, that's, that's, uh, that's, they laugh. Yeah, they laugh at that. But hey, we're happy. We're having fun. Our, our, our listeners are, uh, are, uh, happy, I think.
00:12:55.000 --> 00:12:58.000
Yeah, that's what they get in the first hour.
00:13:05.000 --> 00:13:07.000
I hope.
00:13:06.000 --> 00:13:07.000
Yes.
00:13:07.000 --> 00:13:09.000
Um, why don't…
00:13:08.000 --> 00:13:09.000
Some of them are.
00:13:09.000 --> 00:13:22.000
Why don't we do this? Why don't you tell us about saving money? Oh wait, I changed the topic. Why don't you tell us about running AI locally?
00:13:19.000 --> 00:13:21.000
Yeah.
00:13:22.000 --> 00:13:38.000
Yeah, yeah. Now, I'm gonna get into depth about that, and right off the top, for me, it's not about saving money, because I'm not hardly spending any money on AI. It's more about the ability to run things local, to keep my stuff out of the cloud.
00:13:38.000 --> 00:13:48.000
Um, and because I work in a… in the healthcare field, um, I can't use AI for any of the data that I work on. Um, so I'm looking for ways that I can.
00:13:48.000 --> 00:14:07.000
do that. Anyway, um… You know, AI is a controversial topic, to say the least, right? There are people that love it, and there are people that hate it, and you can't convince either one of them about the other's point of view.
00:14:07.000 --> 00:14:33.000
kind of somewhere in the middle. I like some of the power of it, and the things that I can do, yet I realize the… the evil corporations that are building data centers and, you know, doing all the things, harvesting data all over the internet. That doesn't bother me enough to stay away from it.
00:14:33.000 --> 00:14:53.000
Um, but that's out there. So, uh, I, I know Wolf, you use it for a few things yourself. Um, and, and I, yeah, I, I, I, I love hearing about that too. Uh, there's another episode, I forget the number, but Wolf talks about how he uses Claude for a lot of, uh, a lot of stuff.
00:14:41.000 --> 00:14:45.000
A few.
00:14:53.000 --> 00:14:54.000
And it's pretty neat.
00:14:53.000 --> 00:14:55.000
Unexpected stuff.
00:14:55.000 --> 00:15:12.000
Yeah, yeah. You know, I use it primarily as a coding assistant, and it's working pretty well for me to explore languages that I'm not very familiar with. I'm doing a lot of TypeScript now, and Claude is helping me a great deal with that.
00:15:12.000 --> 00:15:21.000
I appreciate it. Uh, you know, back, uh, episode 8, that was August of last year, uh, we did an episode on, on AI, and Wolf.
00:15:21.000 --> 00:15:32.000
talked about, um… you know, how to do programming with it, and it was pretty neat, and I think the title of that episode was, uh, Dragging Me Into It, and that's kind of what he was doing.
00:15:32.000 --> 00:15:50.000
Um… I wasn't using it at all back then, and now I am. And one of my real concerns was because I don't understand how it works. I mentioned during that episode that I understand how compilers work, and I understand how the shell works.
00:15:50.000 --> 00:15:57.000
And I understand all that stuff, so I'm comfortable using it. AI, I don't have a clue. So this episode is…
00:15:55.000 --> 00:15:58.000
See, I want to insert words.
00:15:57.000 --> 00:15:59.000
Yeah, yeah.
00:15:58.000 --> 00:16:03.000
Because I think there's two things here.
00:16:03.000 --> 00:16:16.000
I think you do understand how it works, or you could easily, but here's the part. I think you don't understand why it makes some decisions.
00:16:16.000 --> 00:16:22.000
Knowing how it works, and knowing why it makes a particular decision.
00:16:22.000 --> 00:16:25.000
is two different things.
00:16:24.000 --> 00:16:25.000
Sure.
00:16:25.000 --> 00:16:32.000
And I think that's like being married, whichever side you're on, and I'm not going to pick one.
00:16:32.000 --> 00:16:38.000
I think it's like that for your spouse. Like, you know.
00:16:38.000 --> 00:16:40.000
How they work?
00:16:40.000 --> 00:16:41.000
Yes.
00:16:40.000 --> 00:16:48.000
You even know what they know, but you don't know why they say those things. Why did they say that? And I think it's the same with AI.
00:16:46.000 --> 00:16:48.000
Or why did they do that?
00:16:48.000 --> 00:17:00.000
Yeah, probably. Although… although I… I don't know how it works. This episode is my adventure into trying to understand. It's just a small bit of how AI works.
00:16:49.000 --> 00:16:51.000
I know how it works.
00:17:00.000 --> 00:17:15.000
Uh, and it's, uh, it's getting really interesting and, and I'm, uh, I'm enjoying this little adventure and I'm gonna be on this adventure for a long time. So there will probably be future episodes where we talk more about how AI works. Um, yeah.
00:17:15.000 --> 00:17:34.000
a couple of weeks ago was the Apple Worldwide Developer Conference, and… that… the keynote and pretty much every talk was almost all about AI, to the point where I wasn't really that interested in listening to the talks, because they talked so much about AI.
00:17:34.000 --> 00:17:46.000
But one of the things that they did mention was Xcode, which I use for developing iOS apps. They're tightening up the integration with AI into Xcode, which is pretty cool. It works with Xcode now.
00:17:46.000 --> 00:17:56.000
I'm sorry, it works with Claude Code, and it works with, I think, Copilot and OpenAI's GPT.
00:17:56.000 --> 00:18:02.000
But, one of the things that they mentioned was, you'll be able to connect up local LLMs.
00:18:02.000 --> 00:18:04.000
to Xcode.
00:18:04.000 --> 00:18:20.000
Which I thought that was pretty neat. And they mentioned a few of them like Llama and Quen. And I think they mentioned maybe Kiwi. I didn't know what all those things were. I knew about Llama a little bit, but I didn't know what the rest were. So I started digging in and that's what I thought.
00:18:20.000 --> 00:18:30.000
this might be an interesting episode, because I'm sure other people are in the same boat, right? They hear these words, they don't really know what they mean. So today, we're going to get into some of that meaning.
00:18:30.000 --> 00:18:40.000
Now, uh, there are a number of commercial models. I've already talked about OpenAI, they've got their GPT, I think they're up to 5.5 right now?
00:18:40.000 --> 00:18:49.000
It's a very capable model. You use it with Codex, and I think ChatGPT probably uses it.
00:18:49.000 --> 00:19:00.000
Um, and Anthropic, uh, they've got a couple of models that, that I am a bit more familiar with. They've got, uh, I think the current version is Opus 4.8.
00:18:59.000 --> 00:19:01.000
That's correct.
00:19:00.000 --> 00:19:01.000
There's.
00:19:01.000 --> 00:19:08.000
Uh, and it seems like about every 2 months they come out with another one, so it probably won't be too long before they release yet another.
00:19:09.000 --> 00:19:18.000
And then there's Sonnet 4.6, which is the, how would you say it, the less capable, faster model.
00:19:18.000 --> 00:19:20.000
If you want to do deep reasoning.
00:19:18.000 --> 00:19:20.000
Uh, believe it or not.
00:19:20.000 --> 00:19:35.000
I use 4.6 predominantly. Um, I'm, uh, when you pick what model you use, I have selected, uh, the word OpusPlan. It's an 8-character word, and what that means is.
00:19:23.000 --> 00:19:25.000
Yum.
00:19:31.000 --> 00:19:33.000
Yah.
00:19:35.000 --> 00:19:52.000
do almost all the work in Sonnet 4.6, but when you do planning, switch to 4.8. But, of course, Haiku, the really fast, cheap one, is always available, and inside any.
00:19:44.000 --> 00:19:45.000
Okay.
00:19:52.000 --> 00:20:00.000
skill you write, you can say, oh, by the way, this skill requires models such and such. And then you get that one.
00:19:58.000 --> 00:19:59.000
Okay.
00:20:01.000 --> 00:20:13.000
Okay, um, you know, I just have mine set to Opus 4.8, because I don't use it that much, and to me, the reason why you would switch between one and the other, I believe Opus.
00:20:13.000 --> 00:20:16.000
Choose up your tokens faster?
00:20:16.000 --> 00:20:18.000
It does.
00:20:16.000 --> 00:20:34.000
Uh, and it's slower and, and I'm not worried so much about the speed and I believe I'm nowhere near hitting my limits on token usage. So I just leave it at 4.8 and don't worry about it. Uh, I suppose if I bump into the limits, uh, then the choice there is either pay more money and increase my limits.
00:20:19.000 --> 00:20:20.000
It is.
00:20:34.000 --> 00:20:36.000
or switch to Sonnet.
00:20:36.000 --> 00:20:41.000
which is uses tokens.
00:20:41.000 --> 00:20:46.000
uses fewer tokens. So, uh, there's a couple of choices there.
00:20:46.000 --> 00:20:56.000
Now, Anthropic also has a model called Mythos, and they have something called Fable 5, and this has been in the news the last couple of weeks.
00:20:56.000 --> 00:21:13.000
Mythos is interesting. I was confused about it. I didn't know what's the difference between that and Opus and Sonnet. Well, Mythos is highly advanced, specialized cybersecurity model that can be used.
00:21:13.000 --> 00:21:29.000
Uh, to both, uh, identify and create sophisticated software exploits. So this is something that it seems very, very dangerous. So the way they were licensing Mythos was a small, select group of people were allowed to use it.
00:21:29.000 --> 00:21:44.000
Um, I know like, um, uh, people who are in the security department of projects, uh, for instance, uh, Daniel Stenberg, the guy that runs, uh, uh, the curl project, he has access to Mythos.
00:21:45.000 --> 00:22:02.000
Um, and that lets him run it against his code, and I guess they tightly control what you can do with it, but he runs it against the curl code, looking for exploits, and he found a couple of exploits, uh, nothing major, because that code is so tight anyway, and people are looking at it so carefully.
00:22:02.000 --> 00:22:20.000
But he has access to that, and other people, like the people in the Linux kernel, have access to it, and corporations and stuff. Well, Fable 5 is a model that's basically Mythos, but they put some pretty strict guardrails on it, so you're really limited at what you can do.
00:22:20.000 --> 00:22:32.000
There's an ongoing feud between Anthropic and the U.S. government, and the U.S. government has basically put a stop to that.
00:22:32.000 --> 00:22:46.000
Nobody can use either Mythos or Fable 5 right now. Um, and there's a pretty interesting article on the Anthropic site, sort of explain… explaining their stand on this. Uh, and they basically, in order to comply.
00:22:44.000 --> 00:22:46.000
They will be painting.
00:22:46.000 --> 00:22:52.000
They will be painting both of those models American Flag Blue.
00:22:46.000 --> 00:22:48.000
I'm sorry?
00:22:51.000 --> 00:23:07.000
That's right. The article is pretty interesting explaining their stand on that. And they've basically made it so nobody can use either of those two models just in order to stay in compliance.
00:23:07.000 --> 00:23:22.000
with the federal government. And that's too bad. I think that'll probably change. There'll probably be court cases and all that kind of stuff. One of the downsides is the NSA, National Security Agency of the U.S. government.
00:23:22.000 --> 00:23:31.000
uh, was relying on Mythos for a lot of stuff they do, and they don't get to do that anymore. By…
00:23:29.000 --> 00:23:31.000
That's a downside.
00:23:31.000 --> 00:23:35.000
Uh, well, to them.
00:23:35.000 --> 00:23:47.000
Right? They no longer have access, and they're a bit upset about that, but OpenAI says, or I'm sorry, Anthropic says, hey, too bad, you said we can't license it?
00:23:47.000 --> 00:23:49.000
That includes you.
00:23:49.000 --> 00:23:55.000
And I think that's — I sort of admire Anthropic for being that way.
00:23:55.000 --> 00:24:02.000
But, you know, there are other commercial models as well. I haven't looked into them.
00:24:02.000 --> 00:24:10.000
But they're out there. You can find them easily enough by googling. But what I really want to talk about today is local models.
00:24:10.000 --> 00:24:19.000
You can actually install and run local models on your computer, and it turns out you don't need that much of a computer to do it.
00:24:19.000 --> 00:24:26.000
Now, I have a Mac Studio with an M1 processor, that's Apple Silicon.
00:24:26.000 --> 00:24:41.000
And, um, 64 gigs of RAM. It's not a huge machine. Um, it, it, it, right now you can get an M4. Uh, unfortunately, Apple is limiting the RAM to 64 gigs, I think, uh, because of the shortage of RAM out there.
00:24:41.000 --> 00:24:44.000
But my little M1…
00:24:42.000 --> 00:24:46.000
Uh, I, wait a second, you're M4 Studio, you mean?
00:24:46.000 --> 00:24:48.000
My M4 studio.
00:24:48.000 --> 00:24:54.000
Right, like, because that you can get an M5 laptop.
00:24:48.000 --> 00:24:49.000
Yes.
00:24:52.000 --> 00:25:00.000
You can now get an M5 MacBook Pro with 128GB of RAM. I looked at it.
00:25:00.000 --> 00:25:02.000
Boy, is that pricey.
00:25:02.000 --> 00:25:07.000
And it's gonna be more expensive, Apple says.
00:25:04.000 --> 00:25:19.000
I'm wondering if they already increased the price, because if you want a pretty well-decked-out MacBook Pro with an M5 and 128GB of RAM and 4TB of storage.
00:25:19.000 --> 00:25:24.000
I kinda do want that. I'm not gonna ever have it, but…
00:25:19.000 --> 00:25:23.000
Um, it, it, it, it's about.
00:25:23.000 --> 00:25:25.000
It, it, it's about eight grand.
00:25:25.000 --> 00:25:27.000
Yes.
00:25:27.000 --> 00:25:32.000
Can you imagine that? $8,000 for a, for a, a, a laptop.
00:25:29.000 --> 00:25:33.000
Um, yeah, I can imagine that pretty hard.
00:25:33.000 --> 00:25:34.000
Oh.
00:25:35.000 --> 00:25:53.000
I'll be in the market for a new machine this fall, hopefully. I was sort of hoping that the studio would have an M5, but… Uh, between the prices and the limited RAM, I'm not sure I'm gonna make any, any change at all, uh, for the foreseeable future, because, man, 8 grand, that's a lot of money.
00:25:53.000 --> 00:26:01.000
Anyway, let's get into the models. There's a number of models you can install and run locally.
00:26:01.000 --> 00:26:16.000
And it turns out, it's very simple to do. Extremely easy to do. I did it in 15 minutes. It was so easy. There's a program called Ollama. Now, I want to make a distinction here. Meta has models called Llama.
00:26:16.000 --> 00:26:19.000
That's L-L-A-M-A.
00:26:19.000 --> 00:26:41.000
There's a program called Ollama, O-L-L-A-M-A. That's a program for downloading and managing your local models. The Llama models are… some of those models you can run in Ollama. It's confusing. I'm not quite sure why they named things like that, but anyway, Ollama is the program you want to install, and it's a really simple install.
00:26:41.000 --> 00:26:55.000
Um, I, I won't get into the instructions on how to install it here, that's, it's all out on the internet. Uh, but you, you install Ollama, and, and all you gotta do is, is, uh, if you wanna run, like, uh, Meta's Llama 3.
00:26:55.000 --> 00:27:00.000
You type ollama run llama3.
00:27:00.000 --> 00:27:27.000
it'll go download the model for you, and start it running. And these models… there… there's some big ones, and there's some small ones. I think, um… Uh, I forget the size. It seemed like Ollama was 17 gigs in size or something. It downloaded pretty quickly. Um, there, there are, there's a, that's the, I think the 8 billion parameter model. Uh, there's a 70.
00:27:27.000 --> 00:27:45.000
billion parameter model that's, like, 47 gigabytes. Fortunately, I've got a nice, fast internet connection, so these download in a matter of minutes, so it's pretty nice. And they both run, those two Llama 3 models, they both run pretty well on my little Mac Studio.
00:27:45.000 --> 00:27:47.000
I was kind of happy with that.
00:27:47.000 --> 00:27:49.000
Umm.
00:27:49.000 --> 00:27:57.000
I know you want to talk about a bunch of these models, but you did say some numbers.
00:27:53.000 --> 00:27:55.000
Yeah.
00:27:56.000 --> 00:27:58.000
Okay.
00:27:57.000 --> 00:28:03.000
And on behalf of the listeners, I feel required to ask.
00:28:03.000 --> 00:28:04.000
Eight billion.
00:28:04.000 --> 00:28:09.000
Parameters? What? 70 billion parameters? What does that mean?
00:28:09.000 --> 00:28:23.000
Yeah, okay, um… Parameters are an adjustable internal value, primarily weights and biases, that the AI uses to process the language and generate responses.
00:28:23.000 --> 00:28:39.000
um, they're like dials, or a recipe, uh, measurement, um, that you encode into the model, or they, they, the people who built the model encode these things in. And the number of parameters that a model has is, is kinda like the.
00:28:39.000 --> 00:28:41.000
power of the model. It's not the only thing.
00:28:41.000 --> 00:28:57.000
Um, but it's, it's, it's how they tune, uh, these, these models. Think of, uh, think of an 8 billion model, uh, uh, uh, an 8 billion parameter model as having 8 billion knobs that can be adjusted.
00:28:57.000 --> 00:29:07.000
to tune that… that model, uh, to do what it's want. That's… that's what training is all about. Uh, when you hear about model training, that's where the real horsepower is needed.
00:29:04.000 --> 00:29:05.000
I…
00:29:08.000 --> 00:29:09.000
And you're gonna say something?
00:29:08.000 --> 00:29:19.000
I sort of get what you're saying, um, and it sounds like the way you're saying it is way better for, uh, uh, uh… A non-expert to hear it.
00:29:19.000 --> 00:29:20.000
Yah.
00:29:19.000 --> 00:29:36.000
But, um… you know, my familiarity with how these things work is that they are not unlike a thing I've worked with for decades, um, called Markov models. And a Markov model is a graph, a big old.
00:29:33.000 --> 00:29:34.000
Right.
00:29:36.000 --> 00:29:46.000
graph. So there's nodes and edges, and typically the edges are weighted, um, and you'll move.
00:29:46.000 --> 00:30:03.000
between nodes by crossing the appropriate edge, depending on whatever thing it is you're letting guide you through the model. So if you're using it for compression, then as you get a new input symbol, you'll cross the right edge into the next.
00:30:03.000 --> 00:30:19.000
node and the edge you cross tells you, for instance, you pick that edge based on what the input was and the node you land in gives you some bits to emit or something.
00:30:19.000 --> 00:30:23.000
When you are looking at one of these giant.
00:30:24.000 --> 00:30:32.000
And there's several different ways to implement AIs. But when you're looking at one of these.
00:30:32.000 --> 00:30:36.000
It is not unlike those Markov models.
00:30:36.000 --> 00:30:42.000
And the thing that an AI expert calls a parameter?
00:30:42.000 --> 00:30:47.000
is what I would have identified as an edge.
00:30:47.000 --> 00:30:54.000
But in a lot of my models, from back in the day, compression and whatnot, the.
00:30:54.000 --> 00:31:11.000
thing that's on an edge, yeah, it's a weight, the number of times you cross it or whatever, is very simple. It might be a character or a word. It turns out that the thing that lets you pick which edge to cross, which parameter to use.
00:31:11.000 --> 00:31:32.000
is… Not even just one token, but some set of tokens, of which maybe you need any particular one, or some combination of them. And then nodes are parts of the answer, what things to put out next, because remember, these are giant predictors.
00:31:32.000 --> 00:31:45.000
So, when you say, knobs that need tweaking, that's the thing that hits me weird, and yet, it really is the right way to say it. And how you build the model?
00:31:45.000 --> 00:31:53.000
Because you start off with no nodes, no knowledge at all, um, and what you do is you acquire knowledge.
00:31:48.000 --> 00:31:49.000
Mmhm.
00:31:53.000 --> 00:32:08.000
In whatever way. You know, reading books, or stealing them, or however you got it. And you add new nodes, and as you… figure out the tokens of the thing that you're grabbing and pulling in?
00:32:08.000 --> 00:32:12.000
You build these new edges.
00:32:12.000 --> 00:32:31.000
And that's the knobs that get tweaked. That's the expensive part that you're talking about, where you need all the horsepower to identify the tokens, and pick which ones go together to make an edge, and determine what's going to be emitted at the brand new node you just made.
00:32:31.000 --> 00:32:36.000
So that's how I think of it. They're not all.
00:32:34.000 --> 00:32:36.000
Sure.
00:32:39.000 --> 00:32:41.000
Go ahead.
00:32:39.000 --> 00:32:54.000
They're not all shaped like this. Um, there's… there's many different ways to do it, and a neural net is not like this, and uh… you know, it all depends on what you're making, but the thing I just described is, um…
00:32:41.000 --> 00:32:43.000
Yeah.
00:32:43.000 --> 00:32:44.000
Yeah, okay.
00:32:54.000 --> 00:33:09.000
a way that these two things Markov models and the general form of one of these multi-parameter AI graphs is built. This is how they kind of correspond.
00:33:09.000 --> 00:33:16.000
They're not the same, but they're so much in the same family that you can think about them that way.
00:33:11.000 --> 00:33:12.000
Yah.
00:33:12.000 --> 00:33:41.000
Well, this… this is… this is the part that I just still don't understand. And I'm… I'm… I'm gonna work my way towards it slowly. Uh, it… I… I do think it's pretty interesting. Um… anyway, think of, uh, you know, the smaller models are 8 billion parameters. Think of 8 billion knobs that… that can be tweaked. And, you know, there's… there's not a human, uh, looking at each one of those and saying, oh, I think this should be a 7.
00:33:41.000 --> 00:33:51.000
Right? It's all… it's all done automatically. Um… in… when they're doing it, they are… they first, they initialize.
00:33:51.000 --> 00:34:08.000
all 8 billion parameters to random numbers. So the output then is just gibberish. And then they, as part of the training, they go through something called forward propagation, where they make a guess at what each of those numbers should be.
00:34:08.000 --> 00:34:13.000
or where each knob should be set.
00:34:12.000 --> 00:34:22.000
And that should sound super familiar to anybody who listened to our Bayes episode, because when you are doing Bayesian thinking.
00:34:18.000 --> 00:34:20.000
Yes, yes.
00:34:22.000 --> 00:34:37.000
and you don't have any priors, um, you start with arbitrary knowledge, you know, sometimes 50-50, or, you know, whatever kind of decision you're making. This is setting up priors when you don't have any.
00:34:30.000 --> 00:34:33.000
Right. Right.
00:34:37.000 --> 00:34:55.000
Yeah. They'll feed it a sentence. For instance, the example that I saw on the internet was, they'll feed it a sentence like, the cat sat on the… and they leave the last word out, and then they use the model to try to predict what the next word is going to be. And before they've.
00:34:55.000 --> 00:35:10.000
tuned anything, uh, it's gonna just come up with some random word, like, imagine it came up with the word banana. Uh, the cat sat on the banana. Now, that doesn't make any sense. So they go through and they calculate how far off they are, and I think that's where they're using the Bayesian.
00:35:10.000 --> 00:35:27.000
thinking, and they keep working through this and working through this, they run these 8 billion parameters. They might run through this model billions of times, trying to refine the model over and over and over again.
00:35:27.000 --> 00:35:44.000
And this process can take months. I think this is why, like, like, uh, Opus comes out, you know, uh, uh… Anthropic comes out with a new version of Opus, like, every two months, because it takes a long time to… Retrain the model on, you know, they, they gather new data.
00:35:41.000 --> 00:35:48.000
Lots of time, lots of electricity, and lots of GPUs!
00:35:48.000 --> 00:36:04.000
Yes, yes, they run it over… Anthropic won't say just what the size of their compute farm is, but it's tens, if not hundreds of thousands of GPUs, maybe millions at this point, and it just runs over and over and over again.
00:36:04.000 --> 00:36:12.000
refining the model and refining it until they finally come up with something where it can predict what the next word is going to be.
00:36:13.000 --> 00:36:24.000
And I still don't know how they get from that to what I'm doing, you know, by generating code with Claude Code. It's still magic to me, right?
00:36:24.000 --> 00:36:34.000
Um… Anyway, so we sort of talked about, yeah, Llama, Meta has their Llama models, which are.
00:36:34.000 --> 00:36:49.000
Pretty capable. There's Llama 3 that I played with quite a bit and Llama 4, which is newer and larger. That the Llama 4 is like 17 billion parameters. In fact, it's really they did something.
00:36:50.000 --> 00:36:54.000
I've got it here in my notes. They did something called MOE.
00:36:54.000 --> 00:36:55.000
Umm.
00:36:56.000 --> 00:37:02.000
Um, I don't see that right. Anyway, um, this thing, um, um.
00:37:02.000 --> 00:37:13.000
I'm sorry. Anyway, they break the model up into smaller pieces, so that it's not really processing the whole model at once.
00:37:13.000 --> 00:37:26.000
it, uh, it, it, they break it down. So, um, for instance, uh, Llama 4 is 108 billion parameters, but there's only 17 billion active at any given time.
00:37:26.000 --> 00:37:37.000
So they can make much, much larger models that are much more capable, but they sort of partition it, and they look at small pieces of it at once, and those small pieces can fit into memory.
00:37:35.000 --> 00:37:43.000
So this sounds like… You are starting to answer the very next question I wanted to ask you.
00:37:42.000 --> 00:37:44.000
Yes.
00:37:43.000 --> 00:37:50.000
And that is, you know, $70 billion of anything is a lot.
00:37:49.000 --> 00:37:50.000
Sure.
00:37:50.000 --> 00:37:51.000
Yah.
00:37:50.000 --> 00:37:59.000
Um, and we could do a little math, but that would have big numbers in it, and we already know how many episodes I've talked about hating big numbers. Uh…
00:37:57.000 --> 00:37:59.000
Yep.
00:37:59.000 --> 00:38:04.000
No matter what you say, if it starts with $70 billion, there's gonna be a lot of RAM involved.
00:38:04.000 --> 00:38:16.000
So my question is, how on earth do you get these giant models to run on your computer, your personal computer?
00:38:05.000 --> 00:38:06.000
Right.
00:38:16.000 --> 00:38:31.000
Okay, you're right. You know, an 8 billion parameter model, you know, if a parameter was one byte, that's 8 gigabytes of space, right? Am I doing the math right? That's a fairly simple one. 70 billion parameters, that's going to be 6 billion.
00:38:31.000 --> 00:38:49.000
Uh, 70 gigs. And let me tell you, there's models that are much, much, much larger than that. Um, Anthropic won't release, uh, the, the size of their models. They won't tell us how many parameters there, but it's estimated, uh, uh, Opus is estimated to be between 800 billion.
00:38:49.000 --> 00:38:52.000
and 2 trillion parameters.
00:38:52.000 --> 00:39:07.000
Uh, now they're using this MOE thing so that they break it down into smaller pieces so that it can run and not, not have to load the whole 2 trillion in memory at once. And again, that 2 trillion is just a guess. Um, uh, uh, open AI with their GPT 5.5.
00:39:07.000 --> 00:39:13.000
That's estimated to be between 1.5 trillion and 9 trillion parameters.
00:39:13.000 --> 00:39:28.000
Um, so yeah, how do they, no matter what, what the number is, 8 billion, 17 billion, 70 billion, whatever, how do they fit it into memory? Uh, well, Wolf talked about these, these nodes, uh, in, in the, um, the, the 8 billion.
00:39:28.000 --> 00:39:32.000
parameters, or nodes in the… in what you call it, the Markov model?
00:39:32.000 --> 00:39:36.000
Right, the parameters are the edges.
00:39:32.000 --> 00:39:33.000
Up.
00:39:33.000 --> 00:39:35.000
Okay.
00:39:36.000 --> 00:39:51.000
Yeah, okay, so the edges, if you've got 8 billion of those, well, when they're building these models, those are usually a 32-bit floating point number. So imagine 32-bit, that's 42 bits.
00:39:51.000 --> 00:40:06.000
4 bytes times 8 billion. That's a lot. That's an awful lot of memory right there. So they do something called quantization. I think I'm saying it right. Where they will reduce that number.
00:40:06.000 --> 00:40:15.000
Uh, mathematically, down to something much smaller, and it's very common for the, for the models you would run on your, on your own computer to, to use a 4-bit.
00:40:15.000 --> 00:40:24.000
to have 4-bit quantization. That means each one of those nodes, or each one of those edges, is only 4 bits.
00:40:18.000 --> 00:40:21.000
They're putting it into buckets.
00:40:24.000 --> 00:40:26.000
So, 8 billion…
00:40:24.000 --> 00:40:27.000
Cause you're picking which bucket it is.
00:40:26.000 --> 00:40:28.000
Yeah, okay.
00:40:28.000 --> 00:40:44.000
You know a lot more about this than I do. So anyway, that's how they do it. It's a method of compression. They take that 32-bit number, and they shrink it down to 4 bits, because that's enough to give you what you need for.
00:40:44.000 --> 00:40:53.000
for these models to do what they do. Now, if you had more bits, I think it would be a more powerful model.
00:40:53.000 --> 00:41:38.000
Uh, it's just one of the things, right? Maybe a more accurate model? Um, but that's… that's how they do it. They… they, uh, instead of a 32-bit number, it's a 4-bit number. Um… And, you know, because we don't have that much RAM on our machines. You know, like I said, my machine here is 64 gigs of RAM. And it's funny because I tried a bunch of different models and I tried the… Llama 4 17 billion parameter model. It's, there's a name for it. I'm sorry. Scout. No, no, Behemoth isn't available. Scout.
00:41:33.000 --> 00:41:34.000
Behemoth?
00:41:35.000 --> 00:41:36.000
Scout.
00:41:38.000 --> 00:41:52.000
is a 17 billion, uh, active parameter model. Maverick is a 400 billion. Uh, Behemoth, which isn't available to us, is 2 trillion parameters. Um… So I ran the Scout model.
00:41:52.000 --> 00:41:57.000
And… I pushed my system so hard.
00:41:57.000 --> 00:42:12.000
It's a 67 gigabyte download. So that took about 30 minutes to download. I was pegging my… I've got a 1.2 gigabit internet connection, but unfortunately my router maxes out at about 450 megabits.
00:42:12.000 --> 00:42:13.000
Ouch.
00:42:13.000 --> 00:42:30.000
Um, yeah, I've got to replace it one of these days. It's just a, it's a Netgate. It has great software on it, but it can't handle the full bandwidth of my internet connection. So I pegged that thing for 30 minutes to the point where I had a VPN running to a customer.
00:42:30.000 --> 00:42:51.000
That died. I had other things running that just wouldn't work. Uh… that was just the download. Once I finished downloading it, and I started running it, um… I pushed this little M1 64 gigabit machine so hard. Um, I, I.
00:42:51.000 --> 00:43:20.000
I tried asking a question. In fact, I did ask this question of several models. I was driving in the car, uh… last week, you know? When I drive out to Ann Arbor for lunch, I have, like, 50 minutes in the car by myself. So I have all kinds of weird thoughts, and I… somehow, this random thought entered my mind, and that is… Um, if water is H2O, that is, uh, two hydrogen atoms and one oxygen atom, and they're both gaseous at room temperature.
00:43:20.000 --> 00:43:24.000
Why, when they combine, is it liquid?
00:43:24.000 --> 00:43:33.000
I didn't understand, okay? So I thought, hey, there's a question to ask AI. So I did. I downloaded these various models, and I asked that question.
00:43:33.000 --> 00:43:38.000
And boy, I asked that question of Llama 4 Scout.
00:43:39.000 --> 00:43:59.000
And it took, uh, maybe… 10 or 15 minutes. I didn't count exactly how long. I didn't want to run it again to count it. It took at least 10 minutes to run and it maxed out my RAM to the point where I was using up almost all my swap. And I've never pushed my studio this hard. You know, I run.
00:43:59.000 --> 00:44:14.000
I run Xcode, which is kind of a pig, and that goes through short bursts of heavy usage, but this thing, that fan was running like crazy on that CPU, and it used up, like.
00:44:14.000 --> 00:44:29.000
every bit of the memory that it could, and it finally came out with an answer, and I was surprised. All these models came out with great answers to that question, and it describes covalent bonding, and positive-negative charges, and why it's liquid, and it was pretty interesting.
00:44:29.000 --> 00:44:33.000
But all these models were capable of doing that, all of them that I tried.
00:44:33.000 --> 00:45:03.000
Uh, I… I think the best model, uh, at least what I thought was the best answer, the most complete answer, was when I used QWEN. That's Q-W-E-N. And that's… that's a model with, uh, 36 billion parameters. And… I was surprised. Asking that question only took 34 seconds for it to answer. And that's on my M1. Now I also, you know, Ollama is not limited to Mac OS. It runs fine on Linux and Windows.
00:45:03.000 --> 00:45:08.000
So, I fired up a Linux virtual machine.
00:45:08.000 --> 00:45:24.000
And I ran Qwen there with Ollama, and it took 121 seconds. So, yeah, it's slower. Now, that machine is just a VM. It doesn't have access to any GPU at all. It's just CPU.
00:45:25.000 --> 00:45:29.000
And Ollama will do what it can with the hardware you have.
00:45:29.000 --> 00:45:32.000
Uh, now, you do have to have some memory.
00:45:29.000 --> 00:45:35.000
So… If you, um…
00:45:32.000 --> 00:45:33.000
But yeah.
00:45:35.000 --> 00:45:39.000
I assume Quen is not the only.
00:45:39.000 --> 00:45:53.000
good, fast model. And yeah, it's big, you described big numbers, but if you can run a good, fast model that gives you a great answer on.
00:45:50.000 --> 00:45:51.000
Okay.
00:45:53.000 --> 00:45:54.000
Your own machine?
00:45:54.000 --> 00:46:00.000
Ummm… Why would you spend money on?
00:46:00.000 --> 00:46:05.000
ChatGPT or Anthropic or Copilot or why?
00:46:06.000 --> 00:46:08.000
Well, you know the the.
00:46:10.000 --> 00:46:20.000
First of all, those models, the Sonnet and Opus and GPT-5.5 and Copilot, they all run out in the cloud. They don't run locally on your machine.
00:46:20.000 --> 00:46:21.000
Umm.
00:46:22.000 --> 00:46:48.000
And there's some benefits to running locally, but there's some downsides as well, and that is the sheer horsepower you need to actually run those. Now, the… the bulk of the processing is done when they train the models. And by the time we're using the model, it's trained. You don't care about that anymore. But the other thing that you need the horsepower for is inference. That is, taking what I type and trying to figure out what it is that I typed, and turning that into something that means something.
00:46:48.000 --> 00:46:59.000
so that it can process it with a model, and then come up with an answer, and spit that out. All that stuff takes a lot of horsepower.
00:46:59.000 --> 00:47:16.000
Um… And yeah, these models… you know, I did mention that I used the… it was the Llama 4 model that I ran that took, like, 10 minutes to run. Llama 3 is much, much faster, and Llama 3, the 8 billion parameter model, came out with a great answer.
00:47:16.000 --> 00:47:44.000
in 31 seconds or something, um, which is comparable with the Qwen model. Um, but back to Wolf's question, why would you… uh, pay, uh, Anthropic to use Claude Code, uh, to do your work. And, and I think… like I said, to do the real horsepower, to do the real heavy lifting, you gotta run the model on really big hardware. So if I had… I don't have access to a Linux box with a GPU.
00:47:44.000 --> 00:48:01.000
If I had access to that Linux box with lots of RAM and a GPU, I think you could do really well running these. Now, one of the things that I learned in this process, I was always it was a little bit muddy to me what the difference was between.
00:48:01.000 --> 00:48:06.000
Claude, Sonnet, Opus, uh, Claude Code.
00:48:06.000 --> 00:48:22.000
And then over the OpenAI world, ChatGPT, Codex using the GPT models. I didn't really understand what all those things really meant. And it turns out that, like, Claude Code is just an application. It's not a model. It's an application.
00:48:22.000 --> 00:48:32.000
That lets you… use models. Codex is an application, and it lets you use a model.
00:48:32.000 --> 00:48:42.000
Interesting thing is, you can use Claude Code, and sure, you can pick between Opus and Sonnet, and what's the thing you used again? Sonnet…
00:48:40.000 --> 00:48:44.000
Uh, there is Haiku, as well.
00:48:43.000 --> 00:48:48.000
Yeah, right, but there's a thing that you use where it sort of switches back and forth automatically.
00:48:45.000 --> 00:48:56.000
Oh, so the name of the thing I'm using is Opus Plan, but that's not a new model, that just picks between models depending upon the situation.
00:48:50.000 --> 00:48:51.000
Yah.
00:48:51.000 --> 00:48:53.000
Okay.
00:48:53.000 --> 00:49:01.000
Oh, it's… That's probably a feature of Claude Code that's doing that switching for you.
00:49:00.000 --> 00:49:02.000
Yes, it is.
00:49:01.000 --> 00:49:09.000
I'm guessing. Okay, well, it turns out that if you download these local models, you can tell Claude Code to use the local model.
00:49:02.000 --> 00:49:03.000
It is.
00:49:09.000 --> 00:49:13.000
So you can tell Claude Code to use Quinn or Llama.
00:49:13.000 --> 00:49:30.000
Uh, which means you can do your programming with that. I didn't… I did run it, just to see that it works, and yes, it does. Um, but I didn't try it in an actual programming exercise to see the quality of what it came out with. Uh, I think that would be a really interesting thing to do. Maybe I'll give it a try.
00:49:30.000 --> 00:49:34.000
And give, uh, some feedback, uh, next episode.
00:49:34.000 --> 00:50:07.000
But, again, back to Wolf's question, why would you pay? I think, right now, we're paying for the quality of the model. In my head, anyway, I think the… the Opus model is probably gonna be far, far superior to anything I could run locally. And, you know, I know, Wolf, you've got 128 gigs of RAM and an M4. I think… uh, running a model, uh, in the cloud, uh, like at Anthropic is still gonna give you better results. Maybe a local model is good enough results. Uh, I, I…
00:50:06.000 --> 00:50:15.000
Well, to me, the deciding question is not, why is H2O a liquid? The deciding question is.
00:50:15.000 --> 00:50:26.000
can you write a plan and us iterate on it? Um, that's what Opus does for me, is I want to do some incredibly complicated task.
00:50:26.000 --> 00:50:42.000
that's going to have many steps, and I want to figure out what steps, some of which I'm going to do, some of which I'm going to let Claude do, some of which I'm going to decide shouldn't happen at all, some of which.
00:50:42.000 --> 00:50:53.000
Claude won't think of, and I'm gonna say, by the way, we also need to do this, and it's gonna take me 12 tries to get it, and the plan is, you know, 8 pages long.
00:50:49.000 --> 00:50:51.000
Sure.
00:50:53.000 --> 00:51:00.000
Um… I want to know, can a local model do that?
00:51:01.000 --> 00:51:03.000
Is it even in the running?
00:51:03.000 --> 00:51:13.000
Okay, my understanding is Claude Code is what provides you with that ability to do a plan. That's a feature, a function of.
00:51:03.000 --> 00:51:04.000
Umm.
00:51:13.000 --> 00:51:17.000
your app, whether it's Cloud Code or Codex.
00:51:16.000 --> 00:51:19.000
Well, the idea… well…
00:51:17.000 --> 00:51:18.000
Um.
00:51:19.000 --> 00:51:21.000
The interacting, the…
00:51:19.000 --> 00:51:24.000
It's a really big answer that came from the model.
00:51:24.000 --> 00:51:31.000
The fact that I can look at that answer, edit it, and iterate on it, that's a feature of the app.
00:51:31.000 --> 00:51:46.000
Right, that's not the model, that's the app. So I think if you're using like Claude Code and you're used to, you know, doing like slash plan to start your plan going, that's Claude Code doing that. Now, if you have Claude Code plugged into Llama or Quinn.
00:51:31.000 --> 00:51:33.000
Um…
00:51:46.000 --> 00:51:54.000
I don't see why it wouldn't work just as well. Now, would the results be as good? Would the results be good enough?
00:51:52.000 --> 00:51:58.000
That's what I want to know. Can it give me a sensible 8-page plan?
00:51:54.000 --> 00:51:56.000
That's worth trying.
00:51:58.000 --> 00:52:11.000
Yeah, that's worth trying, and we should do that. And like I said, it's like a 15-minute install to get Llama up and running, or Olama up and running. You ought to give it a try with your big monster MacBook Pro.
00:52:12.000 --> 00:52:19.000
Because only you can really say whether it's good enough for you or not. I can certainly give it a try.
00:52:19.000 --> 00:52:22.000
So yeah, I just gave you some homework.
00:52:24.000 --> 00:52:37.000
Alright, so… I'm amazed at what these… what these, uh, local models can do. It's pretty… pretty neat. I just sort of touched the, you know, the tip of the iceberg here.
00:52:32.000 --> 00:52:33.000
Well.
00:52:37.000 --> 00:52:44.000
This… this gives me… maybe this is my final question, because I don't want us to go so… so long, but…
00:52:37.000 --> 00:52:38.000
Umm.
00:52:44.000 --> 00:52:56.000
Um, what if these models aren't good enough? What if I want something that knows about the stuff I care about? Can I build my own models?
00:52:55.000 --> 00:53:04.000
Turns out you can. Um, there's a couple of ways. You can, uh, you can fine-tune an existing model.
00:53:04.000 --> 00:53:33.000
Or you can start from scratch and build your own model. That's a huge task, but there are tools out there to let you… to help you with that. But I'm more interested in the first one, because, uh… And so after looking at all this, I'm kind of wondering, what does all this mean? How can I actually use this? What can I do with it? And, you know, I run a company where we do software for doctors.
00:53:33.000 --> 00:53:49.000
And there's a query builder where people can just go in and point and click and add what they want. But the serious people, there's a couple of them, they write SQL to do this and it's complex. And, you know, my customer, my big customer is a.
00:53:49.000 --> 00:53:53.000
They're a cancer place, a place that detects breast cancer.
00:53:54.000 --> 00:54:10.000
And the queries are pretty complex, and I love writing queries. I think I'm pretty good at it, and I help whenever I can, because I get so much joy from it. But I'm kind of wondering, is there a way that I could provide an AI model.
00:54:10.000 --> 00:54:22.000
to help them create their own queries. Now, like I mentioned, I can't give access to my database to anything running in the cloud. I just can't do that.
00:54:22.000 --> 00:54:35.000
Uh, but what if I could run my own model locally that has access to the schema of my database, and it has access to the library of over 2,000 queries that we've written to, to, to, you know.
00:54:35.000 --> 00:54:49.000
do cancer research. That's a lot of what they do there. They have tons of data, they have data on millions of patients. Wouldn't it be cool if I could say, hey, here's this AI model that can do that for you?
00:54:48.000 --> 00:54:50.000
And can you?
00:54:49.000 --> 00:55:04.000
Uh, well, it turns out, yes, I can. I have not tried yet, but I'm very, very interested in doing this, because imagine going into the report creator, and the point where you go to create a query. What if you could just ask.
00:55:04.000 --> 00:55:19.000
Um, my LLM to write the query for you with its knowledge of, and now my, you know, the knowledge isn't great. 2000 queries is not a lot to learn on, but it's a start. Uh, the database schema is obviously crucial.
00:55:19.000 --> 00:55:24.000
Uh, for it to understand. But I could fine-tune one of the models.
00:55:24.000 --> 00:55:40.000
to do exactly that. And then the neat thing about Ollama is, when you run Ollama, it sets up a server listening on a port, and you can connect to that port. It's a RESTful API. You can connect to that port with curl, if you want, and you can feed it a query.
00:55:40.000 --> 00:55:56.000
You know, you can feed it your the text of your query, you know, something like, write me an SQL query that will find the number of cancers in a patient population of 40 to 50 year old women.
00:55:56.000 --> 00:56:07.000
Uh, who are African American. That's my goal, is to be able to, through an API, ask that question, and the output should be an SQL query that'll run in Postgres.
00:56:07.000 --> 00:56:20.000
Um, the tools are there. Uh, I just have to find the time to, to play with it. And I think that's, that's where the power really comes, with your own models, uh, running them locally.
00:56:20.000 --> 00:56:24.000
Training your own model or fine tuning an existing model.
00:56:24.000 --> 00:56:38.000
And, you know, I mentioned, uh, a whole bunch of models. Ollama, or, uh, Llama, um, Quinn, uh, there's DeepSeek, that's a Chinese model that got a lot of attention several months ago. That's out there. I downloaded it, I ran it, it works.
00:56:38.000 --> 00:56:55.000
But there's a, there's like a whole place to go on the internet called huggingface.com, which is just chock full of models. And they have models for all kinds of things that specialize in all kinds of neat things. Like maybe you want to generate graphic images.
00:56:55.000 --> 00:57:23.000
There's a model for that, or there's a whole bunch of models for that. Maybe you want to feed it graphic images and have it describe what's in that image. There's models to do that. You can download those things and run them locally. It's… It's a huge world, and it's only getting bigger, and I'm kind of anxious to see where it takes us, and I'm really interested in seeing how I can integrate this local thing into my application.
00:57:23.000 --> 00:57:26.000
Any more questions, Wolf?
00:57:27.000 --> 00:57:31.000
I am dry. You used me up.
00:57:32.000 --> 00:57:37.000
This was a good one. I was pretty excited by the stuff you had to talk about.
00:57:33.000 --> 00:57:50.000
You said… Yeah, it's pretty neat stuff, and I don't totally understand it, but I'm trying to learn, and I'll tell you what, as I learn more, I'll be happy to share it with you and with the world through this podcast. So yeah, it was a lot of fun. I really enjoyed it.
00:57:50.000 --> 00:57:53.000
So… Why don't you take us out?
00:57:53.000 --> 00:58:10.000
Let me thank everybody who made it this far. In fact, let me thank the people who didn't even make it this far. We're super happy you listened and we hope that we brought something to the table that you.
00:58:10.000 --> 00:58:21.000
uh, enjoyed. Uh, we love feedback. Please send us feedback. You can email us, feedback at runtimearguments.fm.
00:58:21.000 --> 00:58:29.000
There's a website. It is http colon slash slash runtime arguments dot fm.
00:58:29.000 --> 00:58:45.000
Um, you can reach us on Mastodon, at runtimearguments at hackyderm.io. Uh, Jim is on Mastodon, and let's see if I can get it right this time. It would be a first, after a year.
00:58:45.000 --> 00:58:55.000
Uh, let's try at jammcq at, uh, hackyderm.io? Is that right, Jim?
00:58:54.000 --> 00:58:56.000
That's exactly right. That's it.
00:58:55.000 --> 00:59:03.000
Oh my god, I got it right. And I am @yesjustwolf@hackyderm.io.
00:58:56.000 --> 00:58:58.000
Good job.
00:59:03.000 --> 00:59:36.000
Um, we'd love to hear from you. We want your opinions. What did we do right? What did we do wrong? What things would you like us to talk about in the future? Um… Anything you have to think about, anything we've done anytime in the past, uh… When you go to our website, you will see there's a transcript, there's show notes, there's gonna be links, um… Thanks for listening, and I think that's it from me. Goodbye, everybody. Jim?
00:59:35.000 --> 00:59:38.000
Yeah, thanks everybody. Appreciate it.
00:00:00.000 --> 00:00:11.000
Hey everybody, it is Runtime Arguments, episode number 31, and as always, off by one, because it's our 30-second episode.
00:00:11.000 --> 00:00:23.000
Um… This week, yeah, it's AI again, but this time it's Jim who has been running AI models locally.
00:00:22.000 --> 00:00:24.000
Umm.
00:00:23.000 --> 00:00:25.000
On his own machines! For free!
00:00:24.000 --> 00:00:30.000
Uh, Wolf? Well, yes, yes, that's right, Wolf. Uh, you didn't introduce yourself.
00:00:30.000 --> 00:00:32.000
Oh, crap.
00:00:32.000 --> 00:00:40.000
I'm Wolf, and my best friend here as always is Jim. Say hi, Jim.
00:00:40.000 --> 00:00:42.000
Hey everybody, how you doing?
00:00:42.000 --> 00:00:55.000
This, apparently, off to a weird start. I guess we're gonna… Let's see how it goes. Why don't we start with, how was your week? How was your week, Jim? Or two weeks?
00:00:53.000 --> 00:01:02.000
Oh, I had a nice, uh, I, um, I go through periods of, of, uh, not being very productive.
00:01:02.000 --> 00:01:19.000
And then, fortunately, that's followed by weeks of real productivity. And I've had a couple of really good weeks, and I'm really enjoying it. And, um, you know, I had a thought, you know, we… this whole podcast came out of Wilf and I having lunch every Saturday.
00:01:19.000 --> 00:01:31.000
And this past Saturday, uh, we just had a blast. I think there was five of us there? Uh, the, the group, the group is growing, and, and, uh, it's, it's really a lot of fun. And it.
00:01:24.000 --> 00:01:25.000
There were five.
00:01:30.000 --> 00:01:32.000
Our new friend Patrick came.
00:01:32.000 --> 00:01:44.000
That's right, that's right. And I just got confirmation he's coming again this Saturday, so that's nice. But one of the things, as I was sitting there, I was sort of thinking about how.
00:01:44.000 --> 00:01:56.000
how much I enjoy my smart friends. I enjoy all my friends, but I enjoy having smart people to be around so much because I grow from that.
00:01:56.000 --> 00:02:05.000
And, uh, and Wolf, you're, you're at the top of that list. And, uh, the other guys that show up, um, it's just so much fun. And…
00:01:59.000 --> 00:02:02.000
Oh, you make me blush.
00:02:05.000 --> 00:02:12.000
And I learn and I thrive on that and it's great. So yeah, that was sort of my thought for the week.
00:02:12.000 --> 00:02:14.000
How you doing, Wolf?
00:02:14.000 --> 00:02:18.000
Well, I'm never satisfied.
00:02:18.000 --> 00:02:28.000
Sometimes I'm happy, but I'm never satisfied. And there's a tool I use, and for people who aren't using this, I don't know why you're not.
00:02:28.000 --> 00:02:41.000
Uh, and that tool is… DIRENV. D-I-R-E-N-V. And, uh, DIRENV is a pretty simple tool, super useful, written in Go, but there's some shell stuff that goes along with it.
00:02:41.000 --> 00:02:56.000
And the idea behind direnv is that when you cd into a directory that has the magic file, and the magic file is named .envrc, E-N-V-R-C.
00:02:57.000 --> 00:03:17.000
Um, then DurM does stuff that's in that file, and typically it's easy stuff, like… exporting some environment variables. So, for instance, maybe what you have is a couple of different projects that access different AWS accounts.
00:03:17.000 --> 00:03:29.000
Uh, you could make it be that when you CD into one of those directories, it exports the appropriate AWS credentials for that account.
00:03:29.000 --> 00:03:42.000
Um, and then when you CD out of it, you don't have to do anything, they just go away, they're gone, and you CD into the other directory, and you get the new ones. Um, I love that. Um… I.
00:03:42.000 --> 00:03:49.000
write Python, so I'm always using some kind of project.
00:03:49.000 --> 00:04:05.000
Form shape and I have been for a long time in the old days we didn't use shapes but then there were like gosh there's a zillion but like poetry is what the two things that I lean on most heavily.
00:04:05.000 --> 00:04:18.000
these days, is, um, a normal Python project I will manage with UV. UV is a tool that's super fast, written in Rust, replaces a bunch of stuff.
00:04:18.000 --> 00:04:32.000
Um… It has a way of making sure you've got your dependencies, and they're the right ones, and either executing a tool as though you have activated a virtual environment.
00:04:32.000 --> 00:04:43.000
or just activating a virtual environment. You can do either of those. Um, and then Pixie is another, uh, tool that, um, manages projects. The main difference is.
00:04:43.000 --> 00:05:00.000
if all you care about is normal, ordinary Python packages for your dependencies, your requirements, you would use UV. If some of your dependencies are from Conda, from Anaconda, from that world, then you use Pixie, because Pixie knows how to get both.
00:05:01.000 --> 00:05:07.000
The annoying thing about dir env… Is.
00:05:07.000 --> 00:05:20.000
for almost any kind… and Jiraenv applies to everything. Uh, like, if you're building a Ruby project, or a… whatever. It has layouts. It knows what to do.
00:05:20.000 --> 00:05:31.000
for whatever language or shape project you're working with. Um, and you normally start your .envrc file with, for instance, if it's Ruby, layout Ruby.
00:05:31.000 --> 00:05:45.000
Right? Um… We have an existing one for Python, Layout Python, that assumes you're gonna use the old style of Python version and VN… anyway.
00:05:46.000 --> 00:05:50.000
They don't have layout UV.
00:05:52.000 --> 00:06:08.000
And that means I have to implement my own and keep it around in my config files and then I can reference it, but then my projects aren't as portable because my .envrc sample says.
00:06:08.000 --> 00:06:13.000
Um, layout UV at the top, and there is no layout UV. Ugh.
00:06:13.000 --> 00:06:15.000
friction.
00:06:15.000 --> 00:06:16.000
I hate friction.
00:06:16.000 --> 00:06:26.000
So, it's an itch, and I had to scratch it. Um, there were open tickets, there are people who want this, there's people who explicitly don't want it.
00:06:26.000 --> 00:06:36.000
People provided implementations that had really good stuff in them, and basically… It all stalled in 2024.
00:06:36.000 --> 00:06:47.000
and that, that's not good enough for me. So, I took everything good that the people who came before had, um, I wrote a modern.
00:06:47.000 --> 00:07:06.000
PR implementation and ticket that cited the old tickets, referenced the contributors who had thoughts, addressed the three different groups. The three groups were… NVARC doesn't… Durin doesn't do this, so don't.
00:07:06.000 --> 00:07:23.000
And one group was, this is great, but just activate the venv. Don't try to bring it up to date. Don't install any packages. And the third group was, yeah, do it all. And I addressed each of those three groups.
00:07:23.000 --> 00:07:32.000
what you should get or have. And I submitted the PR. Nobody has noticed yet, so it's been a while, but it's there.
00:07:33.000 --> 00:07:41.000
So I'm pretty excited about that. More exciting to me is we've been talking about DappMux.
00:07:41.000 --> 00:07:48.000
my Debug Adapter Protocol Multiplexer that lets you hook up as many.
00:07:48.000 --> 00:07:55.000
pieces of machinery as you want to do debugging. So, for Python, for instance.
00:07:55.000 --> 00:08:09.000
uh, IPython, plus the debugger, plus this super smart guy, um, named Sean Perry, wrote, uh, uh, DAP Observer. It might be being renamed to DAP Tools.
00:08:09.000 --> 00:08:23.000
Um, and your editor, whatever editor you want, and you hook them all together, and they're all working at the same time. We've talked about this. Well, Microsoft, the place where DAP comes from, has a DAP home page.
00:08:23.000 --> 00:08:26.000
where they list implementations.
00:08:26.000 --> 00:08:34.000
But an implementation is like, I have a debugger, I have an editor, and it does that. They didn't have a page for this.
00:08:34.000 --> 00:08:42.000
So I submitted a PR that says, here's a whole new category of tools, and the first one is DatMux.
00:08:42.000 --> 00:09:05.000
And they accepted my PR. Um, I'm not sure it's… it is merged! They did it, but maybe it's not published, so I don't actually see the page live as of when I last checked, which was yesterday evening or something, or else maybe I didn't understand how to make the menu work or something. I thought it was all automatic.
00:09:05.000 --> 00:09:10.000
Uh, we'll see, I'll keep checking. But yeah, um, it's a public thing now.
00:09:11.000 --> 00:09:27.000
Now if only I could show Julia Evans that it addresses the problem she was facing and blogged about. That would make my day. So that was my week or two weeks. Rewarding, but a lot about me.
00:09:27.000 --> 00:09:32.000
solving problems maybe I shouldn't have bothered with. But I did.
00:09:32.000 --> 00:09:43.000
Yeah, you're on fire, and I love watching that. I see your little posts, I see your posts on Mastodon talking about this stuff, and it's exciting. I like that.
00:09:43.000 --> 00:09:48.000
Um… Let's talk about feedback.
00:09:48.000 --> 00:09:55.000
I hear you have some with respect to IPv6.
00:09:51.000 --> 00:09:53.000
Up.
00:09:54.000 --> 00:10:08.000
Uh, yeah, just a little bit. Back, uh, episode 11, back in September, we talked about IPv6, and it was a fun conversation, a lot of useful information, I think. And, uh, Google just announced, uh, within the last couple of weeks.
00:10:08.000 --> 00:10:14.000
that, uh, more than 50% of their traffic worldwide is now over IPv6.
00:10:14.000 --> 00:10:31.000
I think that's pretty good. I think it's being adopted pretty well. I know I'm using it here at home when I connect between my machines. I'm doing the dynamic DNS thing where I set up a machine. For instance, for this episode, I set up a machine called AI.
00:10:31.000 --> 00:10:47.000
So I can just SSH to ai.local and it finds it. I don't have to set up a DNS entry. I don't even need an entry in my SSH config file or anything. I just, I can just point at ai.local and it connects and the connection is IPv6.
00:10:47.000 --> 00:10:54.000
Now, there's nothing about dynamic DNS that says it has to be IPv6, that's just the one it chooses.
00:10:54.000 --> 00:11:12.000
before IPv4. Um, so that's… that's kind of neat. I… I like to see people adopting IPv6. Uh, I know there's a number of, um… Um, uh, internet providers that don't offer it. Uh, I think yours, right? Well, if you're doing, what's, what's the name? You've got a fiber.
00:11:11.000 --> 00:11:23.000
Because I have carrier-grade NAT, IPv6 doesn't go through my, uh… is it an ONT? I always forget what the name of the fiber box is.
00:11:20.000 --> 00:11:31.000
Well, carrier-grade NAT doesn't restrict you from doing IPv6. They wouldn't need NAT at all if they would just implement IPv6.
00:11:29.000 --> 00:11:35.000
Oh, maybe it's not that I have, uh, CGNAT, maybe it's just that it's my provider.
00:11:35.000 --> 00:11:48.000
Yeah, there's a number of providers that just don't supply it yet, and that's too bad. But hey, it's getting better. Over 50% of the traffic to Google is IPv6, so that's kind of neat.
00:11:47.000 --> 00:11:49.000
I have a little feedback.
00:11:49.000 --> 00:11:50.000
You do.
00:11:50.000 --> 00:11:59.000
I do. And my feedback is, um, that someone was talking to me about the show.
00:11:59.000 --> 00:12:11.000
Um, and they said, uh, that the show, uh, gets them fired up about, um, software engineering, about solving problems. Um, that…
00:12:09.000 --> 00:12:10.000
Nice.
00:12:12.000 --> 00:12:24.000
boy, did that hit me in the feels. I mean, that's what I want. That was exactly a great thing to hear. I was very excited about that.
00:12:17.000 --> 00:12:19.000
Yep.
00:12:24.000 --> 00:12:47.000
And, uh, I'm happy that we're starting to make an impact. I'm not, um… I never assume that it's gonna… anything is gonna go as fast as you think or plan for it to go. Um, but the fact that we have listeners, and you know, we're… We're well over 2,000 downloads, 2,500 or something like that.
00:12:45.000 --> 00:12:48.000
Uh, 30… no, 3,500.
00:12:47.000 --> 00:12:52.000
Oh, 3,500, shit. Oh, crap, you'll have to beat that out.
00:12:48.000 --> 00:13:04.000
Yeah, yeah, we're, I mean, you know, compared to some of the big podcasts out there, that's, that's, uh, that's, they laugh. Yeah, they laugh at that. But hey, we're happy. We're having fun. Our, our, our listeners are, uh, are, uh, happy, I think.
00:12:55.000 --> 00:12:58.000
Yeah, that's what they get in the first hour.
00:13:05.000 --> 00:13:07.000
I hope.
00:13:06.000 --> 00:13:07.000
Yes.
00:13:07.000 --> 00:13:09.000
Um, why don't…
00:13:08.000 --> 00:13:09.000
Some of them are.
00:13:09.000 --> 00:13:22.000
Why don't we do this? Why don't you tell us about saving money? Oh wait, I changed the topic. Why don't you tell us about running AI locally?
00:13:19.000 --> 00:13:21.000
Yeah.
00:13:22.000 --> 00:13:38.000
Yeah, yeah. Now, I'm gonna get into depth about that, and right off the top, for me, it's not about saving money, because I'm not hardly spending any money on AI. It's more about the ability to run things local, to keep my stuff out of the cloud.
00:13:38.000 --> 00:13:48.000
Um, and because I work in a… in the healthcare field, um, I can't use AI for any of the data that I work on. Um, so I'm looking for ways that I can.
00:13:48.000 --> 00:14:07.000
do that. Anyway, um… You know, AI is a controversial topic, to say the least, right? There are people that love it, and there are people that hate it, and you can't convince either one of them about the other's point of view.
00:14:07.000 --> 00:14:33.000
kind of somewhere in the middle. I like some of the power of it, and the things that I can do, yet I realize the… the evil corporations that are building data centers and, you know, doing all the things, harvesting data all over the internet. That doesn't bother me enough to stay away from it.
00:14:33.000 --> 00:14:53.000
Um, but that's out there. So, uh, I, I know Wolf, you use it for a few things yourself. Um, and, and I, yeah, I, I, I, I love hearing about that too. Uh, there's another episode, I forget the number, but Wolf talks about how he uses Claude for a lot of, uh, a lot of stuff.
00:14:41.000 --> 00:14:45.000
A few.
00:14:53.000 --> 00:14:54.000
And it's pretty neat.
00:14:53.000 --> 00:14:55.000
Unexpected stuff.
00:14:55.000 --> 00:15:12.000
Yeah, yeah. You know, I use it primarily as a coding assistant, and it's working pretty well for me to explore languages that I'm not very familiar with. I'm doing a lot of TypeScript now, and Claude is helping me a great deal with that.
00:15:12.000 --> 00:15:21.000
I appreciate it. Uh, you know, back, uh, episode 8, that was August of last year, uh, we did an episode on, on AI, and Wolf.
00:15:21.000 --> 00:15:32.000
talked about, um… you know, how to do programming with it, and it was pretty neat, and I think the title of that episode was, uh, Dragging Me Into It, and that's kind of what he was doing.
00:15:32.000 --> 00:15:50.000
Um… I wasn't using it at all back then, and now I am. And one of my real concerns was because I don't understand how it works. I mentioned during that episode that I understand how compilers work, and I understand how the shell works.
00:15:50.000 --> 00:15:57.000
And I understand all that stuff, so I'm comfortable using it. AI, I don't have a clue. So this episode is…
00:15:55.000 --> 00:15:58.000
See, I want to insert words.
00:15:57.000 --> 00:15:59.000
Yeah, yeah.
00:15:58.000 --> 00:16:03.000
Because I think there's two things here.
00:16:03.000 --> 00:16:16.000
I think you do understand how it works, or you could easily, but here's the part. I think you don't understand why it makes some decisions.
00:16:16.000 --> 00:16:22.000
Knowing how it works, and knowing why it makes a particular decision.
00:16:22.000 --> 00:16:25.000
is two different things.
00:16:24.000 --> 00:16:25.000
Sure.
00:16:25.000 --> 00:16:32.000
And I think that's like being married, whichever side you're on, and I'm not going to pick one.
00:16:32.000 --> 00:16:38.000
I think it's like that for your spouse. Like, you know.
00:16:38.000 --> 00:16:40.000
How they work?
00:16:40.000 --> 00:16:41.000
Yes.
00:16:40.000 --> 00:16:48.000
You even know what they know, but you don't know why they say those things. Why did they say that? And I think it's the same with AI.
00:16:46.000 --> 00:16:48.000
Or why did they do that?
00:16:48.000 --> 00:17:00.000
Yeah, probably. Although… although I… I don't know how it works. This episode is my adventure into trying to understand. It's just a small bit of how AI works.
00:16:49.000 --> 00:16:51.000
I know how it works.
00:17:00.000 --> 00:17:15.000
Uh, and it's, uh, it's getting really interesting and, and I'm, uh, I'm enjoying this little adventure and I'm gonna be on this adventure for a long time. So there will probably be future episodes where we talk more about how AI works. Um, yeah.
00:17:15.000 --> 00:17:34.000
a couple of weeks ago was the Apple Worldwide Developer Conference, and… that… the keynote and pretty much every talk was almost all about AI, to the point where I wasn't really that interested in listening to the talks, because they talked so much about AI.
00:17:34.000 --> 00:17:46.000
But one of the things that they did mention was Xcode, which I use for developing iOS apps. They're tightening up the integration with AI into Xcode, which is pretty cool. It works with Xcode now.
00:17:46.000 --> 00:17:56.000
I'm sorry, it works with Claude Code, and it works with, I think, Copilot and OpenAI's GPT.
00:17:56.000 --> 00:18:02.000
But, one of the things that they mentioned was, you'll be able to connect up local LLMs.
00:18:02.000 --> 00:18:04.000
to Xcode.
00:18:04.000 --> 00:18:20.000
Which I thought that was pretty neat. And they mentioned a few of them like Llama and Quen. And I think they mentioned maybe Kiwi. I didn't know what all those things were. I knew about Llama a little bit, but I didn't know what the rest were. So I started digging in and that's what I thought.
00:18:20.000 --> 00:18:30.000
this might be an interesting episode, because I'm sure other people are in the same boat, right? They hear these words, they don't really know what they mean. So today, we're going to get into some of that meaning.
00:18:30.000 --> 00:18:40.000
Now, uh, there are a number of commercial models. I've already talked about OpenAI, they've got their GPT, I think they're up to 5.5 right now?
00:18:40.000 --> 00:18:49.000
It's a very capable model. You use it with Codex, and I think ChatGPT probably uses it.
00:18:49.000 --> 00:19:00.000
Um, and Anthropic, uh, they've got a couple of models that, that I am a bit more familiar with. They've got, uh, I think the current version is Opus 4.8.
00:18:59.000 --> 00:19:01.000
That's correct.
00:19:00.000 --> 00:19:01.000
There's.
00:19:01.000 --> 00:19:08.000
Uh, and it seems like about every 2 months they come out with another one, so it probably won't be too long before they release yet another.
00:19:09.000 --> 00:19:18.000
And then there's Sonnet 4.6, which is the, how would you say it, the less capable, faster model.
00:19:18.000 --> 00:19:20.000
If you want to do deep reasoning.
00:19:18.000 --> 00:19:20.000
Uh, believe it or not.
00:19:20.000 --> 00:19:35.000
I use 4.6 predominantly. Um, I'm, uh, when you pick what model you use, I have selected, uh, the word OpusPlan. It's an 8-character word, and what that means is.
00:19:23.000 --> 00:19:25.000
Yum.
00:19:31.000 --> 00:19:33.000
Yah.
00:19:35.000 --> 00:19:52.000
do almost all the work in Sonnet 4.6, but when you do planning, switch to 4.8. But, of course, Haiku, the really fast, cheap one, is always available, and inside any.
00:19:44.000 --> 00:19:45.000
Okay.
00:19:52.000 --> 00:20:00.000
skill you write, you can say, oh, by the way, this skill requires models such and such. And then you get that one.
00:19:58.000 --> 00:19:59.000
Okay.
00:20:01.000 --> 00:20:13.000
Okay, um, you know, I just have mine set to Opus 4.8, because I don't use it that much, and to me, the reason why you would switch between one and the other, I believe Opus.
00:20:13.000 --> 00:20:16.000
Choose up your tokens faster?
00:20:16.000 --> 00:20:18.000
It does.
00:20:16.000 --> 00:20:34.000
Uh, and it's slower and, and I'm not worried so much about the speed and I believe I'm nowhere near hitting my limits on token usage. So I just leave it at 4.8 and don't worry about it. Uh, I suppose if I bump into the limits, uh, then the choice there is either pay more money and increase my limits.
00:20:19.000 --> 00:20:20.000
It is.
00:20:34.000 --> 00:20:36.000
or switch to Sonnet.
00:20:36.000 --> 00:20:41.000
which is uses tokens.
00:20:41.000 --> 00:20:46.000
uses fewer tokens. So, uh, there's a couple of choices there.
00:20:46.000 --> 00:20:56.000
Now, Anthropic also has a model called Mythos, and they have something called Fable 5, and this has been in the news the last couple of weeks.
00:20:56.000 --> 00:21:13.000
Mythos is interesting. I was confused about it. I didn't know what's the difference between that and Opus and Sonnet. Well, Mythos is highly advanced, specialized cybersecurity model that can be used.
00:21:13.000 --> 00:21:29.000
Uh, to both, uh, identify and create sophisticated software exploits. So this is something that it seems very, very dangerous. So the way they were licensing Mythos was a small, select group of people were allowed to use it.
00:21:29.000 --> 00:21:44.000
Um, I know like, um, uh, people who are in the security department of projects, uh, for instance, uh, Daniel Stenberg, the guy that runs, uh, uh, the curl project, he has access to Mythos.
00:21:45.000 --> 00:22:02.000
Um, and that lets him run it against his code, and I guess they tightly control what you can do with it, but he runs it against the curl code, looking for exploits, and he found a couple of exploits, uh, nothing major, because that code is so tight anyway, and people are looking at it so carefully.
00:22:02.000 --> 00:22:20.000
But he has access to that, and other people, like the people in the Linux kernel, have access to it, and corporations and stuff. Well, Fable 5 is a model that's basically Mythos, but they put some pretty strict guardrails on it, so you're really limited at what you can do.
00:22:20.000 --> 00:22:32.000
There's an ongoing feud between Anthropic and the U.S. government, and the U.S. government has basically put a stop to that.
00:22:32.000 --> 00:22:46.000
Nobody can use either Mythos or Fable 5 right now. Um, and there's a pretty interesting article on the Anthropic site, sort of explain… explaining their stand on this. Uh, and they basically, in order to comply.
00:22:44.000 --> 00:22:46.000
They will be painting.
00:22:46.000 --> 00:22:52.000
They will be painting both of those models American Flag Blue.
00:22:46.000 --> 00:22:48.000
I'm sorry?
00:22:51.000 --> 00:23:07.000
That's right. The article is pretty interesting explaining their stand on that. And they've basically made it so nobody can use either of those two models just in order to stay in compliance.
00:23:07.000 --> 00:23:22.000
with the federal government. And that's too bad. I think that'll probably change. There'll probably be court cases and all that kind of stuff. One of the downsides is the NSA, National Security Agency of the U.S. government.
00:23:22.000 --> 00:23:31.000
uh, was relying on Mythos for a lot of stuff they do, and they don't get to do that anymore. By…
00:23:29.000 --> 00:23:31.000
That's a downside.
00:23:31.000 --> 00:23:35.000
Uh, well, to them.
00:23:35.000 --> 00:23:47.000
Right? They no longer have access, and they're a bit upset about that, but OpenAI says, or I'm sorry, Anthropic says, hey, too bad, you said we can't license it?
00:23:47.000 --> 00:23:49.000
That includes you.
00:23:49.000 --> 00:23:55.000
And I think that's — I sort of admire Anthropic for being that way.
00:23:55.000 --> 00:24:02.000
But, you know, there are other commercial models as well. I haven't looked into them.
00:24:02.000 --> 00:24:10.000
But they're out there. You can find them easily enough by googling. But what I really want to talk about today is local models.
00:24:10.000 --> 00:24:19.000
You can actually install and run local models on your computer, and it turns out you don't need that much of a computer to do it.
00:24:19.000 --> 00:24:26.000
Now, I have a Mac Studio with an M1 processor, that's Apple Silicon.
00:24:26.000 --> 00:24:41.000
And, um, 64 gigs of RAM. It's not a huge machine. Um, it, it, it, right now you can get an M4. Uh, unfortunately, Apple is limiting the RAM to 64 gigs, I think, uh, because of the shortage of RAM out there.
00:24:41.000 --> 00:24:44.000
But my little M1…
00:24:42.000 --> 00:24:46.000
Uh, I, wait a second, you're M4 Studio, you mean?
00:24:46.000 --> 00:24:48.000
My M4 studio.
00:24:48.000 --> 00:24:54.000
Right, like, because that you can get an M5 laptop.
00:24:48.000 --> 00:24:49.000
Yes.
00:24:52.000 --> 00:25:00.000
You can now get an M5 MacBook Pro with 128GB of RAM. I looked at it.
00:25:00.000 --> 00:25:02.000
Boy, is that pricey.
00:25:02.000 --> 00:25:07.000
And it's gonna be more expensive, Apple says.
00:25:04.000 --> 00:25:19.000
I'm wondering if they already increased the price, because if you want a pretty well-decked-out MacBook Pro with an M5 and 128GB of RAM and 4TB of storage.
00:25:19.000 --> 00:25:24.000
I kinda do want that. I'm not gonna ever have it, but…
00:25:19.000 --> 00:25:23.000
Um, it, it, it, it's about.
00:25:23.000 --> 00:25:25.000
It, it, it's about eight grand.
00:25:25.000 --> 00:25:27.000
Yes.
00:25:27.000 --> 00:25:32.000
Can you imagine that? $8,000 for a, for a, a, a laptop.
00:25:29.000 --> 00:25:33.000
Um, yeah, I can imagine that pretty hard.
00:25:33.000 --> 00:25:34.000
Oh.
00:25:35.000 --> 00:25:53.000
I'll be in the market for a new machine this fall, hopefully. I was sort of hoping that the studio would have an M5, but… Uh, between the prices and the limited RAM, I'm not sure I'm gonna make any, any change at all, uh, for the foreseeable future, because, man, 8 grand, that's a lot of money.
00:25:53.000 --> 00:26:01.000
Anyway, let's get into the models. There's a number of models you can install and run locally.
00:26:01.000 --> 00:26:16.000
And it turns out, it's very simple to do. Extremely easy to do. I did it in 15 minutes. It was so easy. There's a program called Ollama. Now, I want to make a distinction here. Meta has models called Llama.
00:26:16.000 --> 00:26:19.000
That's L-L-A-M-A.
00:26:19.000 --> 00:26:41.000
There's a program called Ollama, O-L-L-A-M-A. That's a program for downloading and managing your local models. The Llama models are… some of those models you can run in Ollama. It's confusing. I'm not quite sure why they named things like that, but anyway, Ollama is the program you want to install, and it's a really simple install.
00:26:41.000 --> 00:26:55.000
Um, I, I won't get into the instructions on how to install it here, that's, it's all out on the internet. Uh, but you, you install Ollama, and, and all you gotta do is, is, uh, if you wanna run, like, uh, Meta's Llama 3.
00:26:55.000 --> 00:27:00.000
You type ollama run llama3.
00:27:00.000 --> 00:27:27.000
it'll go download the model for you, and start it running. And these models… there… there's some big ones, and there's some small ones. I think, um… Uh, I forget the size. It seemed like Ollama was 17 gigs in size or something. It downloaded pretty quickly. Um, there, there are, there's a, that's the, I think the 8 billion parameter model. Uh, there's a 70.
00:27:27.000 --> 00:27:45.000
billion parameter model that's, like, 47 gigabytes. Fortunately, I've got a nice, fast internet connection, so these download in a matter of minutes, so it's pretty nice. And they both run, those two Llama 3 models, they both run pretty well on my little Mac Studio.
00:27:45.000 --> 00:27:47.000
I was kind of happy with that.
00:27:47.000 --> 00:27:49.000
Umm.
00:27:49.000 --> 00:27:57.000
I know you want to talk about a bunch of these models, but you did say some numbers.
00:27:53.000 --> 00:27:55.000
Yeah.
00:27:56.000 --> 00:27:58.000
Okay.
00:27:57.000 --> 00:28:03.000
And on behalf of the listeners, I feel required to ask.
00:28:03.000 --> 00:28:04.000
Eight billion.
00:28:04.000 --> 00:28:09.000
Parameters? What? 70 billion parameters? What does that mean?
00:28:09.000 --> 00:28:23.000
Yeah, okay, um… Parameters are an adjustable internal value, primarily weights and biases, that the AI uses to process the language and generate responses.
00:28:23.000 --> 00:28:39.000
um, they're like dials, or a recipe, uh, measurement, um, that you encode into the model, or they, they, the people who built the model encode these things in. And the number of parameters that a model has is, is kinda like the.
00:28:39.000 --> 00:28:41.000
power of the model. It's not the only thing.
00:28:41.000 --> 00:28:57.000
Um, but it's, it's, it's how they tune, uh, these, these models. Think of, uh, think of an 8 billion model, uh, uh, uh, an 8 billion parameter model as having 8 billion knobs that can be adjusted.
00:28:57.000 --> 00:29:07.000
to tune that… that model, uh, to do what it's want. That's… that's what training is all about. Uh, when you hear about model training, that's where the real horsepower is needed.
00:29:04.000 --> 00:29:05.000
I…
00:29:08.000 --> 00:29:09.000
And you're gonna say something?
00:29:08.000 --> 00:29:19.000
I sort of get what you're saying, um, and it sounds like the way you're saying it is way better for, uh, uh, uh… A non-expert to hear it.
00:29:19.000 --> 00:29:20.000
Yah.
00:29:19.000 --> 00:29:36.000
But, um… you know, my familiarity with how these things work is that they are not unlike a thing I've worked with for decades, um, called Markov models. And a Markov model is a graph, a big old.
00:29:33.000 --> 00:29:34.000
Right.
00:29:36.000 --> 00:29:46.000
graph. So there's nodes and edges, and typically the edges are weighted, um, and you'll move.
00:29:46.000 --> 00:30:03.000
between nodes by crossing the appropriate edge, depending on whatever thing it is you're letting guide you through the model. So if you're using it for compression, then as you get a new input symbol, you'll cross the right edge into the next.
00:30:03.000 --> 00:30:19.000
node and the edge you cross tells you, for instance, you pick that edge based on what the input was and the node you land in gives you some bits to emit or something.
00:30:19.000 --> 00:30:23.000
When you are looking at one of these giant.
00:30:24.000 --> 00:30:32.000
And there's several different ways to implement AIs. But when you're looking at one of these.
00:30:32.000 --> 00:30:36.000
It is not unlike those Markov models.
00:30:36.000 --> 00:30:42.000
And the thing that an AI expert calls a parameter?
00:30:42.000 --> 00:30:47.000
is what I would have identified as an edge.
00:30:47.000 --> 00:30:54.000
But in a lot of my models, from back in the day, compression and whatnot, the.
00:30:54.000 --> 00:31:11.000
thing that's on an edge, yeah, it's a weight, the number of times you cross it or whatever, is very simple. It might be a character or a word. It turns out that the thing that lets you pick which edge to cross, which parameter to use.
00:31:11.000 --> 00:31:32.000
is… Not even just one token, but some set of tokens, of which maybe you need any particular one, or some combination of them. And then nodes are parts of the answer, what things to put out next, because remember, these are giant predictors.
00:31:32.000 --> 00:31:45.000
So, when you say, knobs that need tweaking, that's the thing that hits me weird, and yet, it really is the right way to say it. And how you build the model?
00:31:45.000 --> 00:31:53.000
Because you start off with no nodes, no knowledge at all, um, and what you do is you acquire knowledge.
00:31:48.000 --> 00:31:49.000
Mmhm.
00:31:53.000 --> 00:32:08.000
In whatever way. You know, reading books, or stealing them, or however you got it. And you add new nodes, and as you… figure out the tokens of the thing that you're grabbing and pulling in?
00:32:08.000 --> 00:32:12.000
You build these new edges.
00:32:12.000 --> 00:32:31.000
And that's the knobs that get tweaked. That's the expensive part that you're talking about, where you need all the horsepower to identify the tokens, and pick which ones go together to make an edge, and determine what's going to be emitted at the brand new node you just made.
00:32:31.000 --> 00:32:36.000
So that's how I think of it. They're not all.
00:32:34.000 --> 00:32:36.000
Sure.
00:32:39.000 --> 00:32:41.000
Go ahead.
00:32:39.000 --> 00:32:54.000
They're not all shaped like this. Um, there's… there's many different ways to do it, and a neural net is not like this, and uh… you know, it all depends on what you're making, but the thing I just described is, um…
00:32:41.000 --> 00:32:43.000
Yeah.
00:32:43.000 --> 00:32:44.000
Yeah, okay.
00:32:54.000 --> 00:33:09.000
a way that these two things Markov models and the general form of one of these multi-parameter AI graphs is built. This is how they kind of correspond.
00:33:09.000 --> 00:33:16.000
They're not the same, but they're so much in the same family that you can think about them that way.
00:33:11.000 --> 00:33:12.000
Yah.
00:33:12.000 --> 00:33:41.000
Well, this… this is… this is the part that I just still don't understand. And I'm… I'm… I'm gonna work my way towards it slowly. Uh, it… I… I do think it's pretty interesting. Um… anyway, think of, uh, you know, the smaller models are 8 billion parameters. Think of 8 billion knobs that… that can be tweaked. And, you know, there's… there's not a human, uh, looking at each one of those and saying, oh, I think this should be a 7.
00:33:41.000 --> 00:33:51.000
Right? It's all… it's all done automatically. Um… in… when they're doing it, they are… they first, they initialize.
00:33:51.000 --> 00:34:08.000
all 8 billion parameters to random numbers. So the output then is just gibberish. And then they, as part of the training, they go through something called forward propagation, where they make a guess at what each of those numbers should be.
00:34:08.000 --> 00:34:13.000
or where each knob should be set.
00:34:12.000 --> 00:34:22.000
And that should sound super familiar to anybody who listened to our Bayes episode, because when you are doing Bayesian thinking.
00:34:18.000 --> 00:34:20.000
Yes, yes.
00:34:22.000 --> 00:34:37.000
and you don't have any priors, um, you start with arbitrary knowledge, you know, sometimes 50-50, or, you know, whatever kind of decision you're making. This is setting up priors when you don't have any.
00:34:30.000 --> 00:34:33.000
Right. Right.
00:34:37.000 --> 00:34:55.000
Yeah. They'll feed it a sentence. For instance, the example that I saw on the internet was, they'll feed it a sentence like, the cat sat on the… and they leave the last word out, and then they use the model to try to predict what the next word is going to be. And before they've.
00:34:55.000 --> 00:35:10.000
tuned anything, uh, it's gonna just come up with some random word, like, imagine it came up with the word banana. Uh, the cat sat on the banana. Now, that doesn't make any sense. So they go through and they calculate how far off they are, and I think that's where they're using the Bayesian.
00:35:10.000 --> 00:35:27.000
thinking, and they keep working through this and working through this, they run these 8 billion parameters. They might run through this model billions of times, trying to refine the model over and over and over again.
00:35:27.000 --> 00:35:44.000
And this process can take months. I think this is why, like, like, uh, Opus comes out, you know, uh, uh… Anthropic comes out with a new version of Opus, like, every two months, because it takes a long time to… Retrain the model on, you know, they, they gather new data.
00:35:41.000 --> 00:35:48.000
Lots of time, lots of electricity, and lots of GPUs!
00:35:48.000 --> 00:36:04.000
Yes, yes, they run it over… Anthropic won't say just what the size of their compute farm is, but it's tens, if not hundreds of thousands of GPUs, maybe millions at this point, and it just runs over and over and over again.
00:36:04.000 --> 00:36:12.000
refining the model and refining it until they finally come up with something where it can predict what the next word is going to be.
00:36:13.000 --> 00:36:24.000
And I still don't know how they get from that to what I'm doing, you know, by generating code with Claude Code. It's still magic to me, right?
00:36:24.000 --> 00:36:34.000
Um… Anyway, so we sort of talked about, yeah, Llama, Meta has their Llama models, which are.
00:36:34.000 --> 00:36:49.000
Pretty capable. There's Llama 3 that I played with quite a bit and Llama 4, which is newer and larger. That the Llama 4 is like 17 billion parameters. In fact, it's really they did something.
00:36:50.000 --> 00:36:54.000
I've got it here in my notes. They did something called MOE.
00:36:54.000 --> 00:36:55.000
Umm.
00:36:56.000 --> 00:37:02.000
Um, I don't see that right. Anyway, um, this thing, um, um.
00:37:02.000 --> 00:37:13.000
I'm sorry. Anyway, they break the model up into smaller pieces, so that it's not really processing the whole model at once.
00:37:13.000 --> 00:37:26.000
it, uh, it, it, they break it down. So, um, for instance, uh, Llama 4 is 108 billion parameters, but there's only 17 billion active at any given time.
00:37:26.000 --> 00:37:37.000
So they can make much, much larger models that are much more capable, but they sort of partition it, and they look at small pieces of it at once, and those small pieces can fit into memory.
00:37:35.000 --> 00:37:43.000
So this sounds like… You are starting to answer the very next question I wanted to ask you.
00:37:42.000 --> 00:37:44.000
Yes.
00:37:43.000 --> 00:37:50.000
And that is, you know, $70 billion of anything is a lot.
00:37:49.000 --> 00:37:50.000
Sure.
00:37:50.000 --> 00:37:51.000
Yah.
00:37:50.000 --> 00:37:59.000
Um, and we could do a little math, but that would have big numbers in it, and we already know how many episodes I've talked about hating big numbers. Uh…
00:37:57.000 --> 00:37:59.000
Yep.
00:37:59.000 --> 00:38:04.000
No matter what you say, if it starts with $70 billion, there's gonna be a lot of RAM involved.
00:38:04.000 --> 00:38:16.000
So my question is, how on earth do you get these giant models to run on your computer, your personal computer?
00:38:05.000 --> 00:38:06.000
Right.
00:38:16.000 --> 00:38:31.000
Okay, you're right. You know, an 8 billion parameter model, you know, if a parameter was one byte, that's 8 gigabytes of space, right? Am I doing the math right? That's a fairly simple one. 70 billion parameters, that's going to be 6 billion.
00:38:31.000 --> 00:38:49.000
Uh, 70 gigs. And let me tell you, there's models that are much, much, much larger than that. Um, Anthropic won't release, uh, the, the size of their models. They won't tell us how many parameters there, but it's estimated, uh, uh, Opus is estimated to be between 800 billion.
00:38:49.000 --> 00:38:52.000
and 2 trillion parameters.
00:38:52.000 --> 00:39:07.000
Uh, now they're using this MOE thing so that they break it down into smaller pieces so that it can run and not, not have to load the whole 2 trillion in memory at once. And again, that 2 trillion is just a guess. Um, uh, uh, open AI with their GPT 5.5.
00:39:07.000 --> 00:39:13.000
That's estimated to be between 1.5 trillion and 9 trillion parameters.
00:39:13.000 --> 00:39:28.000
Um, so yeah, how do they, no matter what, what the number is, 8 billion, 17 billion, 70 billion, whatever, how do they fit it into memory? Uh, well, Wolf talked about these, these nodes, uh, in, in the, um, the, the 8 billion.
00:39:28.000 --> 00:39:32.000
parameters, or nodes in the… in what you call it, the Markov model?
00:39:32.000 --> 00:39:36.000
Right, the parameters are the edges.
00:39:32.000 --> 00:39:33.000
Up.
00:39:33.000 --> 00:39:35.000
Okay.
00:39:36.000 --> 00:39:51.000
Yeah, okay, so the edges, if you've got 8 billion of those, well, when they're building these models, those are usually a 32-bit floating point number. So imagine 32-bit, that's 42 bits.
00:39:51.000 --> 00:40:06.000
4 bytes times 8 billion. That's a lot. That's an awful lot of memory right there. So they do something called quantization. I think I'm saying it right. Where they will reduce that number.
00:40:06.000 --> 00:40:15.000
Uh, mathematically, down to something much smaller, and it's very common for the, for the models you would run on your, on your own computer to, to use a 4-bit.
00:40:15.000 --> 00:40:24.000
to have 4-bit quantization. That means each one of those nodes, or each one of those edges, is only 4 bits.
00:40:18.000 --> 00:40:21.000
They're putting it into buckets.
00:40:24.000 --> 00:40:26.000
So, 8 billion…
00:40:24.000 --> 00:40:27.000
Cause you're picking which bucket it is.
00:40:26.000 --> 00:40:28.000
Yeah, okay.
00:40:28.000 --> 00:40:44.000
You know a lot more about this than I do. So anyway, that's how they do it. It's a method of compression. They take that 32-bit number, and they shrink it down to 4 bits, because that's enough to give you what you need for.
00:40:44.000 --> 00:40:53.000
for these models to do what they do. Now, if you had more bits, I think it would be a more powerful model.
00:40:53.000 --> 00:41:38.000
Uh, it's just one of the things, right? Maybe a more accurate model? Um, but that's… that's how they do it. They… they, uh, instead of a 32-bit number, it's a 4-bit number. Um… And, you know, because we don't have that much RAM on our machines. You know, like I said, my machine here is 64 gigs of RAM. And it's funny because I tried a bunch of different models and I tried the… Llama 4 17 billion parameter model. It's, there's a name for it. I'm sorry. Scout. No, no, Behemoth isn't available. Scout.
00:41:33.000 --> 00:41:34.000
Behemoth?
00:41:35.000 --> 00:41:36.000
Scout.
00:41:38.000 --> 00:41:52.000
is a 17 billion, uh, active parameter model. Maverick is a 400 billion. Uh, Behemoth, which isn't available to us, is 2 trillion parameters. Um… So I ran the Scout model.
00:41:52.000 --> 00:41:57.000
And… I pushed my system so hard.
00:41:57.000 --> 00:42:12.000
It's a 67 gigabyte download. So that took about 30 minutes to download. I was pegging my… I've got a 1.2 gigabit internet connection, but unfortunately my router maxes out at about 450 megabits.
00:42:12.000 --> 00:42:13.000
Ouch.
00:42:13.000 --> 00:42:30.000
Um, yeah, I've got to replace it one of these days. It's just a, it's a Netgate. It has great software on it, but it can't handle the full bandwidth of my internet connection. So I pegged that thing for 30 minutes to the point where I had a VPN running to a customer.
00:42:30.000 --> 00:42:51.000
That died. I had other things running that just wouldn't work. Uh… that was just the download. Once I finished downloading it, and I started running it, um… I pushed this little M1 64 gigabit machine so hard. Um, I, I.
00:42:51.000 --> 00:43:20.000
I tried asking a question. In fact, I did ask this question of several models. I was driving in the car, uh… last week, you know? When I drive out to Ann Arbor for lunch, I have, like, 50 minutes in the car by myself. So I have all kinds of weird thoughts, and I… somehow, this random thought entered my mind, and that is… Um, if water is H2O, that is, uh, two hydrogen atoms and one oxygen atom, and they're both gaseous at room temperature.
00:43:20.000 --> 00:43:24.000
Why, when they combine, is it liquid?
00:43:24.000 --> 00:43:33.000
I didn't understand, okay? So I thought, hey, there's a question to ask AI. So I did. I downloaded these various models, and I asked that question.
00:43:33.000 --> 00:43:38.000
And boy, I asked that question of Llama 4 Scout.
00:43:39.000 --> 00:43:59.000
And it took, uh, maybe… 10 or 15 minutes. I didn't count exactly how long. I didn't want to run it again to count it. It took at least 10 minutes to run and it maxed out my RAM to the point where I was using up almost all my swap. And I've never pushed my studio this hard. You know, I run.
00:43:59.000 --> 00:44:14.000
I run Xcode, which is kind of a pig, and that goes through short bursts of heavy usage, but this thing, that fan was running like crazy on that CPU, and it used up, like.
00:44:14.000 --> 00:44:29.000
every bit of the memory that it could, and it finally came out with an answer, and I was surprised. All these models came out with great answers to that question, and it describes covalent bonding, and positive-negative charges, and why it's liquid, and it was pretty interesting.
00:44:29.000 --> 00:44:33.000
But all these models were capable of doing that, all of them that I tried.
00:44:33.000 --> 00:45:03.000
Uh, I… I think the best model, uh, at least what I thought was the best answer, the most complete answer, was when I used QWEN. That's Q-W-E-N. And that's… that's a model with, uh, 36 billion parameters. And… I was surprised. Asking that question only took 34 seconds for it to answer. And that's on my M1. Now I also, you know, Ollama is not limited to Mac OS. It runs fine on Linux and Windows.
00:45:03.000 --> 00:45:08.000
So, I fired up a Linux virtual machine.
00:45:08.000 --> 00:45:24.000
And I ran Qwen there with Ollama, and it took 121 seconds. So, yeah, it's slower. Now, that machine is just a VM. It doesn't have access to any GPU at all. It's just CPU.
00:45:25.000 --> 00:45:29.000
And Ollama will do what it can with the hardware you have.
00:45:29.000 --> 00:45:32.000
Uh, now, you do have to have some memory.
00:45:29.000 --> 00:45:35.000
So… If you, um…
00:45:32.000 --> 00:45:33.000
But yeah.
00:45:35.000 --> 00:45:39.000
I assume Quen is not the only.
00:45:39.000 --> 00:45:53.000
good, fast model. And yeah, it's big, you described big numbers, but if you can run a good, fast model that gives you a great answer on.
00:45:50.000 --> 00:45:51.000
Okay.
00:45:53.000 --> 00:45:54.000
Your own machine?
00:45:54.000 --> 00:46:00.000
Ummm… Why would you spend money on?
00:46:00.000 --> 00:46:05.000
ChatGPT or Anthropic or Copilot or why?
00:46:06.000 --> 00:46:08.000
Well, you know the the.
00:46:10.000 --> 00:46:20.000
First of all, those models, the Sonnet and Opus and GPT-5.5 and Copilot, they all run out in the cloud. They don't run locally on your machine.
00:46:20.000 --> 00:46:21.000
Umm.
00:46:22.000 --> 00:46:48.000
And there's some benefits to running locally, but there's some downsides as well, and that is the sheer horsepower you need to actually run those. Now, the… the bulk of the processing is done when they train the models. And by the time we're using the model, it's trained. You don't care about that anymore. But the other thing that you need the horsepower for is inference. That is, taking what I type and trying to figure out what it is that I typed, and turning that into something that means something.
00:46:48.000 --> 00:46:59.000
so that it can process it with a model, and then come up with an answer, and spit that out. All that stuff takes a lot of horsepower.
00:46:59.000 --> 00:47:16.000
Um… And yeah, these models… you know, I did mention that I used the… it was the Llama 4 model that I ran that took, like, 10 minutes to run. Llama 3 is much, much faster, and Llama 3, the 8 billion parameter model, came out with a great answer.
00:47:16.000 --> 00:47:44.000
in 31 seconds or something, um, which is comparable with the Qwen model. Um, but back to Wolf's question, why would you… uh, pay, uh, Anthropic to use Claude Code, uh, to do your work. And, and I think… like I said, to do the real horsepower, to do the real heavy lifting, you gotta run the model on really big hardware. So if I had… I don't have access to a Linux box with a GPU.
00:47:44.000 --> 00:48:01.000
If I had access to that Linux box with lots of RAM and a GPU, I think you could do really well running these. Now, one of the things that I learned in this process, I was always it was a little bit muddy to me what the difference was between.
00:48:01.000 --> 00:48:06.000
Claude, Sonnet, Opus, uh, Claude Code.
00:48:06.000 --> 00:48:22.000
And then over the OpenAI world, ChatGPT, Codex using the GPT models. I didn't really understand what all those things really meant. And it turns out that, like, Claude Code is just an application. It's not a model. It's an application.
00:48:22.000 --> 00:48:32.000
That lets you… use models. Codex is an application, and it lets you use a model.
00:48:32.000 --> 00:48:42.000
Interesting thing is, you can use Claude Code, and sure, you can pick between Opus and Sonnet, and what's the thing you used again? Sonnet…
00:48:40.000 --> 00:48:44.000
Uh, there is Haiku, as well.
00:48:43.000 --> 00:48:48.000
Yeah, right, but there's a thing that you use where it sort of switches back and forth automatically.
00:48:45.000 --> 00:48:56.000
Oh, so the name of the thing I'm using is Opus Plan, but that's not a new model, that just picks between models depending upon the situation.
00:48:50.000 --> 00:48:51.000
Yah.
00:48:51.000 --> 00:48:53.000
Okay.
00:48:53.000 --> 00:49:01.000
Oh, it's… That's probably a feature of Claude Code that's doing that switching for you.
00:49:00.000 --> 00:49:02.000
Yes, it is.
00:49:01.000 --> 00:49:09.000
I'm guessing. Okay, well, it turns out that if you download these local models, you can tell Claude Code to use the local model.
00:49:02.000 --> 00:49:03.000
It is.
00:49:09.000 --> 00:49:13.000
So you can tell Claude Code to use Quinn or Llama.
00:49:13.000 --> 00:49:30.000
Uh, which means you can do your programming with that. I didn't… I did run it, just to see that it works, and yes, it does. Um, but I didn't try it in an actual programming exercise to see the quality of what it came out with. Uh, I think that would be a really interesting thing to do. Maybe I'll give it a try.
00:49:30.000 --> 00:49:34.000
And give, uh, some feedback, uh, next episode.
00:49:34.000 --> 00:50:07.000
But, again, back to Wolf's question, why would you pay? I think, right now, we're paying for the quality of the model. In my head, anyway, I think the… the Opus model is probably gonna be far, far superior to anything I could run locally. And, you know, I know, Wolf, you've got 128 gigs of RAM and an M4. I think… uh, running a model, uh, in the cloud, uh, like at Anthropic is still gonna give you better results. Maybe a local model is good enough results. Uh, I, I…
00:50:06.000 --> 00:50:15.000
Well, to me, the deciding question is not, why is H2O a liquid? The deciding question is.
00:50:15.000 --> 00:50:26.000
can you write a plan and us iterate on it? Um, that's what Opus does for me, is I want to do some incredibly complicated task.
00:50:26.000 --> 00:50:42.000
that's going to have many steps, and I want to figure out what steps, some of which I'm going to do, some of which I'm going to let Claude do, some of which I'm going to decide shouldn't happen at all, some of which.
00:50:42.000 --> 00:50:53.000
Claude won't think of, and I'm gonna say, by the way, we also need to do this, and it's gonna take me 12 tries to get it, and the plan is, you know, 8 pages long.
00:50:49.000 --> 00:50:51.000
Sure.
00:50:53.000 --> 00:51:00.000
Um… I want to know, can a local model do that?
00:51:01.000 --> 00:51:03.000
Is it even in the running?
00:51:03.000 --> 00:51:13.000
Okay, my understanding is Claude Code is what provides you with that ability to do a plan. That's a feature, a function of.
00:51:03.000 --> 00:51:04.000
Umm.
00:51:13.000 --> 00:51:17.000
your app, whether it's Cloud Code or Codex.
00:51:16.000 --> 00:51:19.000
Well, the idea… well…
00:51:17.000 --> 00:51:18.000
Um.
00:51:19.000 --> 00:51:21.000
The interacting, the…
00:51:19.000 --> 00:51:24.000
It's a really big answer that came from the model.
00:51:24.000 --> 00:51:31.000
The fact that I can look at that answer, edit it, and iterate on it, that's a feature of the app.
00:51:31.000 --> 00:51:46.000
Right, that's not the model, that's the app. So I think if you're using like Claude Code and you're used to, you know, doing like slash plan to start your plan going, that's Claude Code doing that. Now, if you have Claude Code plugged into Llama or Quinn.
00:51:31.000 --> 00:51:33.000
Um…
00:51:46.000 --> 00:51:54.000
I don't see why it wouldn't work just as well. Now, would the results be as good? Would the results be good enough?
00:51:52.000 --> 00:51:58.000
That's what I want to know. Can it give me a sensible 8-page plan?
00:51:54.000 --> 00:51:56.000
That's worth trying.
00:51:58.000 --> 00:52:11.000
Yeah, that's worth trying, and we should do that. And like I said, it's like a 15-minute install to get Llama up and running, or Olama up and running. You ought to give it a try with your big monster MacBook Pro.
00:52:12.000 --> 00:52:19.000
Because only you can really say whether it's good enough for you or not. I can certainly give it a try.
00:52:19.000 --> 00:52:22.000
So yeah, I just gave you some homework.
00:52:24.000 --> 00:52:37.000
Alright, so… I'm amazed at what these… what these, uh, local models can do. It's pretty… pretty neat. I just sort of touched the, you know, the tip of the iceberg here.
00:52:32.000 --> 00:52:33.000
Well.
00:52:37.000 --> 00:52:44.000
This… this gives me… maybe this is my final question, because I don't want us to go so… so long, but…
00:52:37.000 --> 00:52:38.000
Umm.
00:52:44.000 --> 00:52:56.000
Um, what if these models aren't good enough? What if I want something that knows about the stuff I care about? Can I build my own models?
00:52:55.000 --> 00:53:04.000
Turns out you can. Um, there's a couple of ways. You can, uh, you can fine-tune an existing model.
00:53:04.000 --> 00:53:33.000
Or you can start from scratch and build your own model. That's a huge task, but there are tools out there to let you… to help you with that. But I'm more interested in the first one, because, uh… And so after looking at all this, I'm kind of wondering, what does all this mean? How can I actually use this? What can I do with it? And, you know, I run a company where we do software for doctors.
00:53:33.000 --> 00:53:49.000
And there's a query builder where people can just go in and point and click and add what they want. But the serious people, there's a couple of them, they write SQL to do this and it's complex. And, you know, my customer, my big customer is a.
00:53:49.000 --> 00:53:53.000
They're a cancer place, a place that detects breast cancer.
00:53:54.000 --> 00:54:10.000
And the queries are pretty complex, and I love writing queries. I think I'm pretty good at it, and I help whenever I can, because I get so much joy from it. But I'm kind of wondering, is there a way that I could provide an AI model.
00:54:10.000 --> 00:54:22.000
to help them create their own queries. Now, like I mentioned, I can't give access to my database to anything running in the cloud. I just can't do that.
00:54:22.000 --> 00:54:35.000
Uh, but what if I could run my own model locally that has access to the schema of my database, and it has access to the library of over 2,000 queries that we've written to, to, to, you know.
00:54:35.000 --> 00:54:49.000
do cancer research. That's a lot of what they do there. They have tons of data, they have data on millions of patients. Wouldn't it be cool if I could say, hey, here's this AI model that can do that for you?
00:54:48.000 --> 00:54:50.000
And can you?
00:54:49.000 --> 00:55:04.000
Uh, well, it turns out, yes, I can. I have not tried yet, but I'm very, very interested in doing this, because imagine going into the report creator, and the point where you go to create a query. What if you could just ask.
00:55:04.000 --> 00:55:19.000
Um, my LLM to write the query for you with its knowledge of, and now my, you know, the knowledge isn't great. 2000 queries is not a lot to learn on, but it's a start. Uh, the database schema is obviously crucial.
00:55:19.000 --> 00:55:24.000
Uh, for it to understand. But I could fine-tune one of the models.
00:55:24.000 --> 00:55:40.000
to do exactly that. And then the neat thing about Ollama is, when you run Ollama, it sets up a server listening on a port, and you can connect to that port. It's a RESTful API. You can connect to that port with curl, if you want, and you can feed it a query.
00:55:40.000 --> 00:55:56.000
You know, you can feed it your the text of your query, you know, something like, write me an SQL query that will find the number of cancers in a patient population of 40 to 50 year old women.
00:55:56.000 --> 00:56:07.000
Uh, who are African American. That's my goal, is to be able to, through an API, ask that question, and the output should be an SQL query that'll run in Postgres.
00:56:07.000 --> 00:56:20.000
Um, the tools are there. Uh, I just have to find the time to, to play with it. And I think that's, that's where the power really comes, with your own models, uh, running them locally.
00:56:20.000 --> 00:56:24.000
Training your own model or fine tuning an existing model.
00:56:24.000 --> 00:56:38.000
And, you know, I mentioned, uh, a whole bunch of models. Ollama, or, uh, Llama, um, Quinn, uh, there's DeepSeek, that's a Chinese model that got a lot of attention several months ago. That's out there. I downloaded it, I ran it, it works.
00:56:38.000 --> 00:56:55.000
But there's a, there's like a whole place to go on the internet called huggingface.com, which is just chock full of models. And they have models for all kinds of things that specialize in all kinds of neat things. Like maybe you want to generate graphic images.
00:56:55.000 --> 00:57:23.000
There's a model for that, or there's a whole bunch of models for that. Maybe you want to feed it graphic images and have it describe what's in that image. There's models to do that. You can download those things and run them locally. It's… It's a huge world, and it's only getting bigger, and I'm kind of anxious to see where it takes us, and I'm really interested in seeing how I can integrate this local thing into my application.
00:57:23.000 --> 00:57:26.000
Any more questions, Wolf?
00:57:27.000 --> 00:57:31.000
I am dry. You used me up.
00:57:32.000 --> 00:57:37.000
This was a good one. I was pretty excited by the stuff you had to talk about.
00:57:33.000 --> 00:57:50.000
You said… Yeah, it's pretty neat stuff, and I don't totally understand it, but I'm trying to learn, and I'll tell you what, as I learn more, I'll be happy to share it with you and with the world through this podcast. So yeah, it was a lot of fun. I really enjoyed it.
00:57:50.000 --> 00:57:53.000
So… Why don't you take us out?
00:57:53.000 --> 00:58:10.000
Let me thank everybody who made it this far. In fact, let me thank the people who didn't even make it this far. We're super happy you listened and we hope that we brought something to the table that you.
00:58:10.000 --> 00:58:21.000
uh, enjoyed. Uh, we love feedback. Please send us feedback. You can email us, feedback at runtimearguments.fm.
00:58:21.000 --> 00:58:29.000
There's a website. It is http colon slash slash runtime arguments dot fm.
00:58:29.000 --> 00:58:45.000
Um, you can reach us on Mastodon, at runtimearguments at hackyderm.io. Uh, Jim is on Mastodon, and let's see if I can get it right this time. It would be a first, after a year.
00:58:45.000 --> 00:58:55.000
Uh, let's try at jammcq at, uh, hackyderm.io? Is that right, Jim?
00:58:54.000 --> 00:58:56.000
That's exactly right. That's it.
00:58:55.000 --> 00:59:03.000
Oh my god, I got it right. And I am @yesjustwolf@hackyderm.io.
00:58:56.000 --> 00:58:58.000
Good job.
00:59:03.000 --> 00:59:36.000
Um, we'd love to hear from you. We want your opinions. What did we do right? What did we do wrong? What things would you like us to talk about in the future? Um… Anything you have to think about, anything we've done anytime in the past, uh… When you go to our website, you will see there's a transcript, there's show notes, there's gonna be links, um… Thanks for listening, and I think that's it from me. Goodbye, everybody. Jim?
00:59:35.000 --> 00:59:38.000
Yeah, thanks everybody. Appreciate it.
