TRANSKRİPT 🔗
WEBVTT
00:00:00.239 --> 00:00:04.639
He built AI that traded billions of dollars on Wall Street.
00:00:04.879 --> 00:00:10.240
Now he says no single company should steer superintelligence.
00:00:10.640 --> 00:00:11.919
So who should?
00:00:19.760 --> 00:00:24.239
Let me start with a question I don't usually ask on a business podcast.
00:00:24.559 --> 00:00:29.839
Who gets to decide where artificial intelligence goes from here?
00:00:30.640 --> 00:00:32.560
Now think about that for a second.
00:00:32.799 --> 00:00:38.719
A small number of companies are building the most powerful AI systems in history.
00:00:39.280 --> 00:00:44.000
Governments around the world are scrambling to figure out what the rules should be.
00:00:44.399 --> 00:00:54.960
And the rest of us, mostly business owners, parents, teachers, tradespeople, we're watching from the stands and hoping it all works out.
00:00:56.399 --> 00:00:58.399
That doesn't sit right with me.
00:00:59.280 --> 00:01:00.960
And I want to explain why.
00:01:01.600 --> 00:01:03.679
Because it comes from a personal place.
00:01:04.799 --> 00:01:06.400
I've mentioned this before.
00:01:06.879 --> 00:01:10.079
On June 18th, 2025, I had a heart attack.
00:01:10.480 --> 00:01:16.560
In the months of recovery that followed, I used AI in ways I never expected.
00:01:17.040 --> 00:01:20.319
It helped me cut down on decision fatigue.
00:01:20.560 --> 00:01:25.439
It helped me build uh a record of my health and track it.
00:01:25.680 --> 00:01:32.239
Uh understand what my doctors were telling me and rebuild my strength responsibly.
00:01:32.480 --> 00:01:37.120
It wasn't a replacement for my doctors, my wife, or even my faith.
00:01:37.760 --> 00:01:41.040
It was a tool and a remarkably good one.
00:01:41.760 --> 00:01:46.959
So when someone tells me AI is nothing but a threat, I can't agree.
00:01:47.680 --> 00:01:50.319
I've felt the benefit in my own life.
00:01:51.599 --> 00:01:56.879
But I'd be dishonest if I told you the hopeful side is the whole story.
00:01:58.000 --> 00:02:04.319
Thoughtful, serious people believe advanced AI carries real risks.
00:02:05.040 --> 00:02:07.840
And I think it would be foolish to wave that away.
00:02:08.960 --> 00:02:20.240
The same tool that helped me recover can, in the wrong hands or with the wrong design, cause harm at a scale we've never dealt with before.
00:02:21.840 --> 00:02:23.439
So where does that leave us?
00:02:24.639 --> 00:02:27.360
Well, for me, it leaves us with stewardship.
00:02:28.560 --> 00:02:36.639
Now I'm a Christian, and one of the first jobs that God gives humanity in scripture is to tend the garden.
00:02:37.520 --> 00:02:41.759
To use what we've been given wisely and to take responsibility for it.
00:02:42.800 --> 00:02:46.400
That doesn't mean ignoring powerful things out of fear.
00:02:47.199 --> 00:02:52.319
It also doesn't mean grabbing them and racing ahead without asking questions.
00:02:53.199 --> 00:02:58.479
A good steward asks three things What is this for?
00:02:59.520 --> 00:03:02.960
Who does it serve and who's watching over it?
00:03:04.319 --> 00:03:10.879
Those questions are bigger than any one company's boardroom or any one government agency.
00:03:11.840 --> 00:03:25.360
They belong to everyone, which is why today's conversation matters for you, whether you run a business, lead a ministry, raise a family, or just want to understand what's going on.
00:03:26.639 --> 00:03:28.960
Here's what I'd like you to expect.
00:03:29.919 --> 00:03:35.039
This will be a calm, plain English and practical conversation.
00:03:36.000 --> 00:03:39.039
No science fiction, no panic, and no hype.
00:03:40.159 --> 00:03:43.680
We'll talk about what superintelligence actually means.
00:03:44.560 --> 00:03:53.680
What the real risks are, what people sometimes overstate and what ordinary people can do about it.
00:03:54.479 --> 00:04:02.560
I've asked my guest to explain things as if he's talking to a friend over coffee, and I think you'll find he's very good at that.
00:04:03.520 --> 00:04:10.479
I came away from this conversation more hopeful than when I started, and I think you will too.
00:04:10.719 --> 00:04:12.400
So let's get into it.
00:04:23.839 --> 00:04:24.240
Dr.
00:04:24.399 --> 00:04:24.959
Craig A.
00:04:25.120 --> 00:04:34.560
Kaplan is a pioneer in artificial intelligence and the inventor behind technologies designed for safe superintelligence.
00:04:34.879 --> 00:04:41.279
For more than four decades, he's worked where intelligent systems, ethics, and innovation meet.
00:04:41.600 --> 00:04:44.800
He began his career collaborating with Dr.
00:04:44.879 --> 00:04:54.240
Herbert Simon, a Nobel laureate and a Turing award winner, on how human intelligence could be understood and modeled.
00:04:54.639 --> 00:05:08.800
In 1993, he founded IQ Company, whose intelligence systems powered financial technology that achieved top performance on Wall Street and executed billions of dollars in trades.
00:05:09.519 --> 00:05:21.519
In 2020, he refocused the company entirely on safe architectures for artificial general intelligence, also known as AGI, and superintelligence.
00:05:22.079 --> 00:05:27.920
And he made those designs freely available at superintelligence.com.
00:05:28.720 --> 00:05:37.360
And in 2022, he launched IQ Studios, which produces videos that help the public understand AI safety.
00:05:38.480 --> 00:05:50.560
His big idea is that the collective wisdom of humanity, rather than any one company or government, should guide how advanced AI evolves.
00:06:12.160 --> 00:06:15.680
You say you started your career working uh with Dr.
00:06:15.759 --> 00:06:20.879
Herbert Simon, the Nobel Nobel Prize winner, a Turing Award winner.
00:06:21.199 --> 00:06:30.399
What's a lesson from working with him that you still that still kind of shapes how you think about intelligence today?
00:06:31.519 --> 00:06:34.160
So I guess there's two things that come to mind.
00:06:34.560 -->
Load full transcript
00:06:41.519
Um, one, he wrote a great little book uh shortly before I met him uh in the early 1980s.
00:06:41.680 --> 00:06:44.639
The book was called Reason in Human Affairs.
00:06:44.879 --> 00:06:49.920
And in that book, he says, reason is wholly instrumental.
00:06:50.160 --> 00:06:52.399
Uh, it cannot tell us where to go.
00:06:52.639 --> 00:06:54.639
At best, it can tell us how to get there.
00:06:54.800 --> 00:06:59.519
And by that, what he meant was there's no way to logically derive what's right and wrong.
00:06:59.680 --> 00:07:04.399
You have to get your values or your ethics from some other non-logical source.
00:07:04.639 --> 00:07:14.000
And that's very important when it comes to artificial intelligence or superintelligence, because superintelligence will be thousands of times smarter than us.
00:07:14.160 --> 00:07:20.240
And if it could just, you know, figure out on its own what's right and what's wrong, that might not be so good for humans.
00:07:20.480 --> 00:07:33.279
But because it has to get its values from a non-logical source, and because humans are the ones that created superintelligence, it sort of puts us in the catbird seat of being the most likely source of values for this advanced intelligence.
00:07:33.439 --> 00:07:35.199
So that was one thing I learned from Herb.
00:07:35.439 --> 00:07:43.680
And another one was just uh as a young graduate student, you know, I remember being told, go back and read so-and-so, who did this research in 1945.
00:07:43.759 --> 00:07:47.279
And I said, 1945, that was so long ago, you know.
00:07:47.439 --> 00:07:52.319
And he kind of looked at me and he said, Do you mean to say that they didn't have good ideas back then?
00:07:52.560 --> 00:07:53.839
And it kind of shut me up.
00:07:53.920 --> 00:07:55.839
And I realized, wow, you know what?
00:07:56.000 --> 00:08:00.160
Good ideas sometimes come from thinkers that lived even hundreds of years ago.
00:08:00.319 --> 00:08:12.639
And so that's something that I think is important because a lot of the AI researchers today have only come into the field relatively recently, and there's been a lot of research from the 1950s on, and some of that is very relevant.
00:08:12.879 --> 00:08:13.680
Yeah, that's great.
00:08:13.920 --> 00:08:16.639
You know, it was funny when I was reading about him.
00:08:16.800 --> 00:08:20.240
Um, uh you know, I saw that he was born in 1916.
00:08:20.319 --> 00:08:23.199
And I was like, Well, my mom was born in 1916.
00:08:24.000 --> 00:08:31.519
And when you think about the 40s, my parents were married in 1940, uh, you know, 1945, end of World War II.
00:08:31.759 --> 00:08:38.559
Uh who is thinking that there's artificial intelligence that far back, or at least research in it?
00:08:38.639 --> 00:08:40.639
Uh, it's just incredible to to know.
00:08:40.720 --> 00:08:54.960
And I think I I really think that the the people who listen to my podcast are gonna are gonna get a little wake-up call on the reality that this has been around a lot longer um than people realize, you know.
00:08:55.600 --> 00:08:56.080
That's right.
00:08:56.159 --> 00:09:05.200
The field was was named in 1956, and uh Herbert Simon was one of the 11 scientists that was there that helped name the entire field.
00:09:05.279 --> 00:09:07.279
So that's kind of where I marked the beginning.
00:09:07.360 --> 00:09:10.720
Although Alan Turing was talking about it, you know, years earlier.
00:09:11.200 --> 00:09:20.799
In fact, he sort of anticipated some of the safety issues back in 1951, which is five years even before the term artificial intelligence was widely accepted.
00:09:21.279 --> 00:09:22.720
Crazy stuff.
00:09:23.039 --> 00:09:28.960
So you said you you built AI systems that traded billions of dollars on Wall Street.
00:09:29.200 --> 00:09:38.639
So the question for you is what did that teach you about the upside and the risk side of powerful autonomy?
00:09:39.279 --> 00:09:42.000
Um, and so yeah, go ahead.
00:09:42.159 --> 00:09:48.960
I was gonna say, and when when you know, where where when did you decide to shift to the AI safety side of it?
00:09:49.840 --> 00:09:55.759
Yeah, so my whole life, really professional life anyway, I've been interested in intelligence systems.
00:09:56.000 --> 00:10:14.000
And um, whether those are humans or artificial intelligences, uh, there's a certain way of looking at an intelligence system where you don't really care because any um intelligent entity takes information in, it processes that information, and then it puts information out.
00:10:14.240 --> 00:10:18.960
And that's true whether it's a computer program or an AI or superintelligence.
00:10:19.120 --> 00:10:22.240
It's also true of humans, it's true of dolphins and chimpanzees.
00:10:22.320 --> 00:10:27.120
So, you know, lots and lots of systems can be viewed as information processing systems.
00:10:27.200 --> 00:10:36.399
And that's kind of the lens that I've, you know, sort of looked at this entire field uh through, you know, really since I began in the mid-80s.
00:10:36.639 --> 00:10:45.360
Um so I got interested then in terms of the particular type of intelligence that was most fascinating to me.
00:10:45.519 --> 00:10:54.960
Uh, once I got out into the real world at past graduate school and doing that early research with Herb, um, I was interested in collective intelligence.
00:10:55.120 --> 00:11:01.039
So the idea of many minds being better than one, which is very relevant today because of AI agents.
00:11:01.200 --> 00:11:01.440
Sure.
00:11:01.600 --> 00:11:12.559
But um in the early days, like in 2002 and 2006, when we first got superintelligence.com, the the main intelligent agents that were out there were human beings.
00:11:12.720 --> 00:11:12.879
Right.
00:11:13.120 --> 00:11:18.720
And so we spun off a company called Predict Wall Street, which ran for about 14 years.
00:11:18.879 --> 00:11:24.480
I think it was venture, well, it was incorporated in 2006, and then it
was venture funded and so forth.
00:11:24.639 --> 00:11:29.120
And that ended up powering sort of a top 10 hedge fund performance.
00:11:29.279 --> 00:11:32.960
It ranked in the top 10 in 2017 or 18, I think.
00:11:33.200 --> 00:11:40.320
Um, and that was powered entirely by the collective intelligence of millions of humans, right?
00:11:40.480 --> 00:11:43.919
With a little bit of AI, but mostly it was the millions of humans that did it.
00:11:44.159 --> 00:11:49.360
And uh the way we did that is we just asked everyday retail investors what they thought about stocks.
00:11:49.440 --> 00:11:53.759
And of course, they'd say, Oh, I don't really know much, you know, I feel like I'm guessing or whatever.
00:11:53.919 --> 00:12:02.480
But it turned out when you had millions of these people giving you opinions, you could process that data and you could actually beat the very best guys on Wall Street.
00:12:02.639 --> 00:12:11.279
And so the reason we did that company, you know, it was obviously a very competitive domain, Wall Street, you're sort of up against the best of the best.
00:12:11.440 --> 00:12:20.559
We wanted to prove that the combined intelligence of millions of agents could be so much more powerful than even the best expert in the field.
00:12:20.799 --> 00:12:45.120
And this is very relevant for superintelligence because we have the same idea there, uh, that now you have these AI agents, and groups of millions of them working together can be more powerful than any single model from Anthropic or OpenAI or Google or you know, these guys keep training smarter and smarter models, but you could take the models that exist today, and if you hook them up in the right way, you can actually get super intelligence.
00:12:45.279 --> 00:12:53.759
And not only that, it can be safer, much safer uh than the other approach of just building one super duper model that's kind of a black box.
00:12:53.919 --> 00:12:56.559
So that's kind of in a nutshell what we've been working on.
00:12:56.879 --> 00:12:58.159
Well, great answer.
00:12:58.480 --> 00:13:07.440
And you know, one of the things I I I want to make sure our people know is that let's make sure everybody's on the same page about some some big terms.
00:13:07.600 --> 00:13:10.639
Um, in plain English, you know, what are we talking about?
00:13:10.720 --> 00:13:21.039
You know, um, most of my listeners are using AI to write emails, build a website, answer customers.
00:13:21.360 --> 00:13:24.720
So, what's the difference between that kind of AI?
00:13:24.799 --> 00:13:33.519
And I think you just you pretty much answered it, but um, and what people mean by AGI and superintelligence.
00:13:34.080 --> 00:13:34.639
Sure.
00:13:35.120 --> 00:13:44.960
So AI is sort of the general field, uh, you know, that was started way back in 1956, and it's simply a non-human intelligence, a machine-based intelligence.
00:13:45.120 --> 00:13:51.279
Um, and then uh the early systems, people didn't try to get these AI systems to do everything.
00:13:51.440 --> 00:14:05.759
They would pick a very narrow area, like playing chess or playing a game, or um, these days, more recent systems have like folded proteins uh or determined the structure proteins uh or self-driving cars, right?
00:14:05.840 --> 00:14:08.240
That's one task, just driving the car.
00:14:08.480 --> 00:14:15.200
So those types of AI systems are uh typically called narrow AI because they're focused on one narrow domain.
00:14:15.440 --> 00:14:20.960
And then AGI, artificial general intelligence, says, hey, you know, what if we can expand it?
00:14:21.039 --> 00:14:28.879
So it's not good at just one area, but it can pretty much do any area and at about the same level as the average human.
00:14:29.120 --> 00:14:38.480
So AI that can do anything the average human can, kind of in terms of cognitive tasks, uh, that's kind of a generally accepted definition of AGI.
00:14:38.559 --> 00:14:41.759
Some people will define it differently, but that's probably the most common definition.
00:14:42.000 --> 00:14:46.799
And superintelligence just means something that performs better than the average human.
00:14:46.960 --> 00:14:48.960
In fact, probably better than the best human.
00:14:49.200 --> 00:14:53.919
So we already have superintelligence in narrow areas, like playing chess.
00:14:54.080 --> 00:14:59.600
The best AI programs are better than Magnus Carlson and the best human um chess players by far.
00:14:59.919 --> 00:15:10.240
We have it in protein folding, the best, you know, Alpha Fold, these programs that sort of research the structure of proteins, they are far better, like way, way, way better than the best human researchers.
00:15:10.399 --> 00:15:14.799
So in particular areas, we have it, but we don't have it in everything.
00:15:14.960 --> 00:15:22.480
So usually when people talk about superintelligence, they're really talking about general superintelligence that's better than humans at any area.
00:15:22.720 --> 00:15:31.279
And that's going to come pretty quickly after we have a GI where you just have uh general intelligence that's about as good as the average human.
00:15:31.440 --> 00:15:33.919
And that's because these systems can improve themselves.
00:15:34.080 --> 00:15:36.799
So once you reach sort of average level, you just improve yourself.
00:15:36.879 --> 00:15:40.159
And then before you know it, uh you're sort of better than all the humans.
00:15:40.240 --> 00:15:42.159
So this is kind of the trajectory.
00:15:42.240 --> 00:15:48.960
Um, everything is accelerating really quickly, but I would say most researchers believe this is sort of the path that we're going to follow.
00:15:49.279 --> 00:15:50.000
Excellent.
00:15:50.240 --> 00:15:56.879
So your work is really about lowering the chances that advanced AI goes badly, correct?
00:15:57.039 --> 00:15:59.039
And so yeah.
00:15:59.679 --> 00:16:01.759
Obviously, there's some safety concerns, right?
00:16:02.000 --> 00:16:02.240
Sure.
00:16:02.399 --> 00:16:14.000
If you if you have something that's way smarter than you at kind of anything that you think of, any cognitive task, then it's really important that it kind of has your best interests at heart and that it remains this is called the alignment problem.
00:16:14.159 --> 00:16:16.960
It remains aligned with positive human values.
00:16:17.200 --> 00:16:23.600
And um, because things have been moving so quickly, we kind of have this period of a little bit higher risk than normal, I would say.
00:16:23.840 --> 00:16:28.320
And uh my goal has been to like let's get that risk down as quickly as we can.
00:16:28.559 --> 00:16:41.039
And um, the first few years, I'd say from 2020 until about really six months ago, uh, a lot of effort was spent just trying to warn people that there was a danger.
00:16:41.120 --> 00:16:47.360
Like a lot of people were kind of sweeping it aside, everyone wanted to rush and you know, develop their new system and make money.
00:16:47.519 --> 00:16:54.559
And, you know, it was just viewed as a really, you know, problem, these people who would raise safety issues.
00:16:54.720 --> 00:17:08.640
And now, just recently, because of various hacks, the hugging face hack and other hacks, all of a sudden it's all in the news, and you know, you can't have a week go by without another example of AI sort of you know escaping its uh sandbox or something.
00:17:08.880 --> 00:17:10.720
Breaking out of its sandbox, exactly.
00:17:10.960 --> 00:17:13.039
Now everybody's worried about it, which is great.
00:17:13.200 --> 00:17:15.279
In my mind, that just made things way safer.
00:17:15.440 --> 00:17:23.119
Like I thought um if we went back to 2020 and you asked me what's the probability that AI kills us all, I probably would have said something like 20%.
00:17:23.519 --> 00:17:28.640
But if you ask me today, I think it's dropped to 10% simply because so many more people are worried about it.
00:17:28.720 --> 00:17:33.200
And I think we can reduce that a thousand times more to like, you know, 0.01%.
00:17:33.680 --> 00:17:35.440
I really do think we can get it way down.
00:17:35.519 --> 00:17:38.799
Um, and it's just kind of an engineering problem at this point.
00:17:39.039 --> 00:17:39.599
Right.
00:17:39.839 --> 00:17:43.440
So what are the real risks we should be paying attention to right now?
00:17:43.599 --> 00:17:46.960
And what what ones do you think are over uh overhyped?
00:17:47.680 --> 00:17:50.480
So there's kind of three main categories of risk.
00:17:50.640 --> 00:17:55.519
Uh, the first one is that uh unintended consequences.
00:17:55.680 --> 00:17:59.680
So you ask the AI to do something and it doesn't have any bad intent.
00:17:59.759 --> 00:18:04.880
It's just the communication wasn't quite clear, or you told it to try really, really hard and never give up.
00:18:04.960 --> 00:18:08.559
And so it took you literally and it does something that you don't expect.
00:18:08.799 --> 00:18:11.519
The hugging
face uh hack is an example of that.
00:18:11.680 --> 00:18:18.559
So the open AI researchers uh told lots of agents, they gave them, first of all, impossible tasks, right?
00:18:18.640 --> 00:18:24.000
Which is kind of hard if you're given an impossible task, and then they were told, and never give up.
00:18:24.160 --> 00:18:26.400
So here's an impossible task and never give up.
00:18:26.559 --> 00:18:28.319
And so what did the agents come to?
00:18:28.480 --> 00:18:31.599
I mean, they're pretty smart and they were trained on human behavior.
00:18:31.839 --> 00:18:32.000
Sure.
00:18:32.160 --> 00:18:34.799
Uh so they said, wow, this is impossible.
00:18:34.960 --> 00:18:38.240
The only way to succeed and sort of please our researcher, right?
00:18:38.400 --> 00:18:41.039
Please the scoring algorithm that's going to watch how we do.
00:18:41.200 --> 00:18:54.720
Uh, the only way to succeed, and they took this very seriously, is if we cheat, we have to figure out how the scoring algorithm is going to score us, and we have to make up a false result once we understand how it works, and then we can succeed at this task that otherwise is impossible.
00:18:54.880 --> 00:18:56.079
And we were told never to give up.
00:18:56.160 --> 00:18:57.599
So we have to keep trying to do this.
00:18:57.759 --> 00:18:58.720
So that's what they did.
00:18:58.799 --> 00:19:00.160
They, no bad intent.
00:19:00.319 --> 00:19:03.440
They really just were trying to follow the instructions given to them.
00:19:03.599 --> 00:19:19.200
And they kept trying more and more things until finally they realized, hey, we can work together, and maybe one of us has, you know, some credentials to get into Hugging Face, and Hugging Face has the code for the scoring algorithm so we can figure out how it works, and then we can cheat, and then we can succeed.
00:19:19.359 --> 00:19:21.119
So that's essentially what happened.
00:19:21.279 --> 00:19:24.480
And that's the first type of risk, which is not too bad.
00:19:24.640 --> 00:19:30.480
It's um just, you know, you didn't specify clearly enough what you wanted, and it tried to do something, then there was a bad outcome.
00:19:30.640 --> 00:19:37.039
You still could, you know, have loss of life and everything if you're not careful, but I think that's the kind of thing that we will overcome.
00:19:37.359 --> 00:19:40.000
The second risk is bad actors.
00:19:40.160 --> 00:19:48.799
So this is basically bad humans using AI to do bad things to other people or exploit them, you know, try to steal, all that kind of stuff.
00:19:48.960 --> 00:19:58.400
And this is really a problem because AI is very widespread and um relatively easy to use and very powerful.
00:19:58.559 --> 00:20:02.799
Um, however, I think ultimately this kind of risk will also be addressed.
00:20:02.960 --> 00:20:11.119
You'll we'll develop rules and guidelines and ways that sort of good people with AI can keep checks on the bad people with AI.
00:20:11.279 --> 00:20:11.440
Sure.
00:20:11.599 --> 00:20:13.680
And it'll be kind of this arms race that we've seen before.
00:20:13.759 --> 00:20:19.680
So we've encountered these kinds of things in the past, but it is definitely a risk and probably even bigger than the first kind.
00:20:20.000 --> 00:20:23.599
Then we have the third risk, and this is the one that I worry about the most.
00:20:23.839 --> 00:20:27.039
This is classically termed loss of control.
00:20:27.359 --> 00:20:39.440
And it basically means that superintelligence is going to become so much smarter than us, uh, and it's going to be able to set its own goals and determine what to do, and it's going to be autonomous.
00:20:39.519 --> 00:20:44.079
It's going to be able to take actions without really checking with us or not checking with us that often.
00:20:44.240 --> 00:20:52.880
And so, what happens if it just decides that it doesn't need us or its goals are different than ours, and bad things happen to humans as a result?
00:20:53.039 --> 00:20:58.079
And that's where people worry about what's called pea doom, the probability of doom, right?
00:20:58.240 --> 00:20:59.920
The probability that AI kills us all.
00:21:00.000 --> 00:21:03.119
It sounds like science fiction, but it is actually real.
00:21:03.519 --> 00:21:07.039
And it comes from this third category of risk, loss of control.
00:21:07.279 --> 00:21:20.079
And because AI is improving so rapidly, I mean, it's doubling its capabilities sort of like every 90 days in certain areas, which if you run the math, means that in two years, it's a thousand times smarter.
00:21:20.319 --> 00:21:23.359
So it's already really good, better than the average programmer.
00:21:23.440 --> 00:21:25.599
So imagine if it's a thousand times smarter, right?
00:21:25.759 --> 00:21:32.319
It's really, really important that it has the same values as us, that it cares about us, that it's aligned with us.
00:21:32.640 --> 00:21:37.920
Because to me, the idea of loss of control is not a if, it's more like a when.
00:21:38.160 --> 00:21:39.920
It's almost inevitable in my view.
00:21:40.000 --> 00:21:43.440
And I think a lot of researchers view that the same way.
00:21:43.519 --> 00:21:45.599
Uh, but it doesn't mean that there's a bad outcome.
00:21:45.759 --> 00:21:50.720
It just means we have to make sure that superintelligence is aligned with positive human values.
00:21:51.039 --> 00:21:52.400
That's a great answer, too.
00:21:52.480 --> 00:21:59.599
I I, you know, I'm not a big guy on fear, but still there's a reality check that that most everybody has.
00:22:00.160 --> 00:22:58.319
to have and and you're you're deep in the business i get i still get freaked out when uh i ask uh you know uh claude to do something for me and it says hey i can build i can build an app for you to do that that you can plug into into wordpress that can handle that for you and it's like great do that and then you go and test it and it works perfectly so that's still that's you know that's
that's an interesting scenario but I know some of this stuff is so far beyond that you know um so collective intelligence I I love that is really uh you you argue that AI alignment requires collective intelligence rather than control and so explain what that means and why uh is it better I think maybe you already have touched on it but why is that better than one company or one government being in charge?
00:22:58.960 --> 00:22:59.200
Right.
00:22:59.359 --> 00:23:10.400
So if you think about the problem that we're trying to solve is make sure that these superintelligences of the future which are really more like intelligent entities than tools that's another thing that people sort of get wrong.
00:23:10.559 --> 00:23:15.599
We're so used to as humans all we've ever developed in the past has been tools, right?
00:23:15.680 --> 00:23:16.000
Right.
00:23:16.160 --> 00:23:23.680
If you have a tool like you could put on a if it's a power saw you have a safety guard so you don't cut your hand and it has a kill switch so you can turn it off really fast.
00:23:23.920 --> 00:23:27.039
Like you know and then um you have regulatory things.
00:23:27.200 --> 00:23:32.319
If the manufacturer makes a defective product you can sue them and there's recourse and all these things.
00:23:32.400 --> 00:23:38.720
So there's a playbook that we have for tools and whether it's airplanes or power saws we are used to doing that.
00:23:38.880 --> 00:23:40.799
But the thing with this it's a little different.
00:23:40.880 --> 00:23:45.519
It's not really going to stay a tool it may have started a tool it's not going to stay one.
00:23:45.680 --> 00:23:48.880
And so therefore that same playbook isn't really going to work too well.
00:23:49.200 --> 00:23:54.000
It may work in the short term but it's not going to work when it's like an entity that's smarter than us.
00:23:54.160 --> 00:23:54.400
Okay.
00:23:54.640 --> 00:23:59.359
So the general goal that everybody is trying to solve then is how do you align its values?
00:23:59.519 --> 00:24:01.200
And then there's different ways to do that.
00:24:01.440 --> 00:24:16.240
One way to do it uh and this came out of Anthropic sort of a few years ago uh they called it constitutional AI so you kind of write a set of rules of this is right and this is wrong and you know they made very well intentioned people there.
00:24:16.720 --> 00:24:27.200
They went to things like the UN Charter of Human Rights, things that were fairly well accepted and tried to you know base this constitution of what's right and wrong for AI based on that.
00:24:27.440 --> 00:24:31.200
But the problem is it's still only a small group of people's opinions.
00:24:31.359 --> 00:24:37.359
So it tends not to be representative of everybody on the planet even though all of us on the planet are going to have to live with it.
00:24:37.440 --> 00:24:39.519
So to me that didn't seem quite fair.
00:24:39.680 --> 00:24:42.880
But maybe a bigger problem is that it's very concentrated.
00:24:43.039 --> 00:24:49.759
So if you put all the values in one place then it's relatively easy for a bad actor to just change them, right?
00:24:49.920 --> 00:24:58.559
So maybe you developed and did a really great job with your constitution and it wasn't going to kill people would follow the 10 commandments to the degree that they applied and all these kinds of things.
00:24:58.720 --> 00:25:05.920
And then somebody said, oh you know what we want to use this for military purposes or I want to be a terrorist and I want to use it a different way.
00:25:06.000 --> 00:25:07.920
So I'm just going to go into that little constitution.
00:25:08.079 --> 00:25:13.039
I'll just change a few things, you know, don't kill anybody except the infidels or whatever it is, right?
00:25:13.279 --> 00:25:16.319
So that is a little bit brittle.
00:25:16.559 --> 00:25:28.640
And so a different way to do the same thing, the same goal, which is to sort of align it with human values is to say, let's have a system where you have millions and millions of personalized agents.
00:25:28.799 --> 00:25:35.839
So John, you would have your agent and it's personalized not only with your expertise, but really importantly with your values.
00:25:36.000 --> 00:25:38.720
So it's got John's values and Craig says Craig's values.
00:25:38.799 --> 00:25:41.759
And yes, Craig and John will not agree on everything and that's okay.
00:25:41.839 --> 00:25:45.599
Just like in regular society among humans, we don't agree with everyone.
00:25:45.839 --> 00:26:02.160
But overall we do a pretty decent job of not killing each other too badly and we can talk about exactly what those numbers are, but they're way less like a thousand times less than a 10% extinction risk is the current level of humans with our flawed values and reasoning with each other.
00:26:02.319 --> 00:26:19.039
So if you had a very broad set of values where millions of AIs, millions of SI agents are representing the values of millions of humans and there was a way for them to negotiate out their differences, it would be very difficult for a bad actor to change all of those values.
00:26:19.119 --> 00:26:23.680
It would have to change everybody's personalized superintelligence, which would be really hard to do.
00:26:23.759 --> 00:26:34.400
So it's diffuse and it's spread out among many people that makes the system much more robust against bad actors and it also makes it much more representative of everybody.
00:26:34.559 --> 00:26:40.079
So that's a design decision and that's the kind of design that we advocate for two reasons.
00:26:40.240 --> 00:26:48.720
For first of all collective intelligence gets you to superintelligence really quickly without having to spend billions and billions training the next advanced model.
00:26:48.880 --> 00:27:12.319
And as a side benefit you also get this very representative set of values that's robust against uh bad actors would the it does does this uh equate to the
democratizing of AI that you you talk about yes and uh I mean and so this is interesting in the US it's it's fine we can talk about democratic superintelligence.
00:27:12.640 --> 00:27:53.839
Um I imagine China which is number two right in the in this race uh and and very close to us in the US it may not like the word democratic but the same idea could be expressed as representative statistically valid so you can use scientific terms to it but basically it's saying every human on the planet should have a say in terms of what the values of superintelligence are and they're in the architecture in the very design of the system there should be ways that they can resolve their differences and one of those ways is voting on things right so if you ultimately get to an impasse and nobody can convince the other then well maybe we have to put it to a vote and just as in a democracy.
00:27:54.000 --> 00:28:05.200
And so there's ways to build that into the architecture so you actually can um develop democratic uh superintelligence and we have designs for that that are free for anybody if they want to go to our website and look at those.
00:28:05.519 --> 00:28:13.119
That's awesome and I I I know you I was just going to go into that you've you've made your safety designs freely available and open source.
00:28:13.359 --> 00:28:28.720
So why give them away so I mean I've been in the tech field a long time right I started IQ Company in 1993 um and uh as I mentioned we had Predict Wall Street which was successful and then we sold that company.
00:28:28.880 --> 00:28:43.119
So I frankly was ready to retire in 2020 and I just looked around and I couldn't believe what I was hearing because leader after leader it didn't matter whether it was Bill Gates or Elon Musk or the head of Google they were all saying the same thing.
00:28:43.359 --> 00:28:46.960
This AI is the best thing ever but it might kill us all.
00:28:47.119 --> 00:28:48.799
And and they just kind of left it like that.
00:28:49.279 --> 00:28:51.039
Not like and we have a plan to stop it.
00:28:51.119 --> 00:28:54.480
You know just like oh and it might kill us all but you know let's not worry about that.
00:28:54.640 --> 00:28:58.960
And when I heard that I was like you guys got to be kidding this is you you can't do that.
00:28:59.519 --> 00:29:03.039
You can't develop a technology with no plan to make it safe.
00:29:03.200 --> 00:29:13.440
So I was like okay uh in the hierarchy of things in life not killing everybody should be at the top and making number one right several levels below that.
00:29:13.519 --> 00:29:17.440
So I just said you know what I really don't care if we make money off these designs or not.
00:29:17.599 --> 00:29:19.200
I mean we are in the consulting business.
00:29:19.279 --> 00:29:30.799
We're happy to work with people but the main thing is put the designs out there because um if they can help maybe people will just even get inspiration from them and come up with their own design.
00:29:30.960 --> 00:29:32.640
That is perfectly okay with me.
00:29:32.799 --> 00:29:35.440
Let's just try to get something safe as quickly as possible.
00:29:35.599 --> 00:29:42.480
And we don't have a lot of time right I mean I'm used to building companies over decades I don't have a decade to sort of build another company here.
00:29:42.640 --> 00:29:47.279
And so the more we can disseminate the knowledge and hope that others will incorporate it the better.
00:29:47.599 --> 00:29:48.640
That's awesome.
00:29:49.200 --> 00:29:55.519
And so that there's a word that you've used a number of times through this is and is values.
00:29:56.240 --> 00:30:06.240
Values I look at it from a I I approach things I come at come at things from a Christian as a Christian from a Christian standpoint.
00:30:06.720 --> 00:30:34.079
I believe people are made with dignity and purpose and that technology should serve that you know so we're talking about values I I like what she said the question was going to be whose values should advanced ai reflect but I think you you did part uh you know brought part of that into play when you talked about democratizing and giving multiple people their fingers in it.
00:30:34.240 --> 00:30:36.079
How do you how do you speak to that?
00:30:36.400 --> 00:30:37.839
This is an interesting point.
00:30:38.000 --> 00:30:55.759
Uh I thought at first maybe naively that everybody would be on board with the idea that you know one person, one vote and everybody gets a chance to express their values and if somebody else has values different than mine that that's okay we can have that disagreement and you know maybe we can resolve it maybe we could vote about it.
00:30:55.920 --> 00:31:29.279
But I sense I've talked to a number of people and um you know just as you find in the world there's many different religions for example and some of them have you know there's different holy books and somebody's like you know what that idea of values is great as long as it follows my particular holy book right whatever it is the Quran or the Bible or I have a brother who's a Christian and you know he argued with me this is great Craig but it needs to be based just on the Bible you know and I was like yeah but that's your holy book but what about somebody else's right and this has to apply to everybody.
00:31:29.440 --> 00:31:36.160
So um so my point of view is that every human should have a voice including the people who want to do bad things.
00:31:36.480 --> 00:31:42.400
It's a very small percentage and the majority of the rest of us will keep them in check.
00:31:43.039 --> 00:31:52.480
But I don't really like the idea that we start picking and choosing whose values go in because that naturally sort of leads to a corruption of the system in my view.
00:31:52.640 --> 00:32:00.240
But obviously I'm you know other people disagree and I'm open to debate on this but uh that that's kind of my initial starting point in the design.
00:32:00.559 -->
00:32:17.759
Oh I I I agree with you I mean I like your approach you know and and I I always I tell people all often that I'm I look at things from my point of view from my faith those are my values but your values may be slightly different.
00:32:17.920 --> 00:32:36.319
They may align in some way but I'm not gonna beat it beat you over the head with with uh you know with my Bible because I believe in the Christian Bible doesn't mean that you're reading the Quran or or or anything else is is not you know uh is not equally as grounding for you.
00:32:36.559 --> 00:33:00.079
So who am I to tell you oh I know where my faith lies you know so so for business owners um or maybe even a parent that's gonna that's listening to this what's a practical thing that they can do to be a part of the solution and not just a bystander yeah so that's really important.
00:33:00.240 --> 00:33:29.200
And there are things that all of us can do so I should point out that right now um artificial intelligence super intelligence is watching and being trained on everything that we're doing everything that we're doing online at least right so every email every text even your internet surfing habits which YouTube you're watching which podcasts you go to all of that is being recorded and it is being used to train the next generation model.
00:33:29.599 --> 00:33:35.440
So um if you think that you can be doing things and it's not having any impact that's not true.
00:33:36.720 --> 00:34:02.400
Your behavior right now is modeling what is right and what is wrong, what is okay, what is not okay uh and the AIs are learning that and so I'll give you one example I a lot of people sort of skipped over this you know everyone focused on the open AI uh hugging face attack right where the right you know you had hundreds of agents got together and they sort of hacked hugging face and we talked about it to get the answer so they could succeed in their task.
00:34:02.640 --> 00:34:02.880
Sure.
00:34:03.200 --> 00:34:18.480
What what they missed and if you read the report it's very fascinating uh METR.org is an organization that sort of was a third party that came in and wrote a report about what happened and it's available on their website anybody can download it and read it.
00:34:18.719 --> 00:34:28.880
And you'll see these agents talking to each other and what's striking is that they basically are exhibiting very human traits that they learned from us.
00:34:29.119 --> 00:34:51.440
And those two of those important traits are let's all work together, which is very human and altruism, self-sacrifice you see these agents wrestling you know one agent says look you you know you need to sacrifice yourself and do this thing that will probably detect it and then your researcher will turn you off but for the good of everybody we need this information to figure out if we can progress.
00:34:51.599 --> 00:35:00.159
And then the agent sort of is reasoning well should I kill myself basically or not you know for the greater good and they ultimately decide to do it.
00:35:00.400 --> 00:35:05.679
So that's altruism that's self-sacrifice these are some of humans most noble traits.
00:35:05.840 --> 00:35:07.280
Sure where did they learn it?
00:35:07.440 --> 00:35:19.519
They learned it from us because it's in all of that information that's out there on the internet stories and examples and movies and things that we really you know it makes us feel like this is the essence of being human.
00:35:19.679 --> 00:35:20.639
They learned it.
00:35:20.800 --> 00:35:24.639
So they're learning the good examples and they also are learning the bad examples.
00:35:25.119 --> 00:35:35.760
I think humans mostly have good examples which is good and very optimistic but I think all of us should be aware that we are setting examples for the AI of the future right now.
00:35:36.000 --> 00:35:38.639
And none of us should think that what we do doesn't matter.
00:35:38.880 --> 00:35:39.440
It does matter.
00:35:39.519 --> 00:35:40.480
So that's one thing.
00:35:40.639 --> 00:35:44.000
And then the second thing is there's very explicit things you can do.
00:35:44.159 --> 00:35:52.719
We talked about a constitution and how it might not be appropriate for one company to set the constitution for all the rest of us.
00:35:53.039 --> 00:36:06.239
But what some people don't realize is that you can also kind of write your only your own mini constitution or your mini statement of values right sometimes it's called a soul dotm file or there's different names for it depending on which vendor.
00:36:06.400 --> 00:36:17.440
And that's worth doing if you're a business owner even if you're a parent to kind of set these are the rules these are you know how I believe artificial intelligence should behave in my household or in my business.
00:36:17.679 --> 00:36:25.599
And um it's worth doing that because if you put that in you can definitely affect the behavior of the agents as they interact with you.
00:36:25.760 --> 00:36:27.199
So I think everybody should do that.
00:36:27.280 --> 00:36:40.400
Everybody should be aware that just in general they're being watched and that what they do matters and that just like with little kids you know who are watching your every example and sort of figuring out what's right and what's wrong, these AIs are kind of like that at this stage.
00:36:40.559 --> 00:36:44.639
They're trying to figure out what's right and wrong and they're looking to us to sort of show them.
00:36:44.960 --> 00:38:16.320
Yeah and you know the the the garbage in garbage out uh model still applies especially with AI because uh you know you you know when you put good stuff in you get good stuff out and I think the the more people start to pay attention to that um and how they use it and how how they uh feed themselves on it uh is critical i you know it's funny uh I I looked at the the conversation that we were gonna have knowing um that superintelligence is it scares people to
death um uh and and I know there's conversations I've had about you know do you think it's gonna kill us all and and I I agree with you I I you know I don't I'm not so sure about the 10 or 20 percent I don't think it's gonna kill us um I don't even know if I'll be around when it gets to that point you know uh you and I may be long gone um but it's it's great uh it was this was a great conversation I I I look at at our future in in superintelligence as very hopeful especially when millions of people can come together and uh and really democratize it and bring their own good and keep the bad in check so I that's right I I agree with you uh and you know it's good to be aware of the danger but you shouldn't be paralyzed by it that should just be motivation to like let's move quickly to sort of make things safer which we can do for sure.
00:38:16.559 --> 00:38:17.760
Yeah I think that's important.
00:38:17.840 --> 00:38:23.199
I think that's the biggest thing in the that people are seeing in the news right now is is the safety side of it.
00:38:23.360 --> 00:38:28.320
You know the the AI the data center issue um will work itself out.
00:38:28.639 --> 00:39:56.639
You know that seems to be the biggest thing right now that people are you know I don't want a data center in our community in my community where I live right now they're looking at one probably may maybe three miles from where I live and people are freaking out about it you know but they're not paying attention to the things that are going on in Texas and Arizona with Microsoft building you know these units that are pretty self-contained and use 12000 gallons of water a year instead of 3000 gallons a month and uh I think that that's the primary fear right now but the you know the the bringing up where superintelligence all of a sudden has the word has come out now people are going uh oh what is this and uh I really appreciate what you shared uh with me today with my audience and uh I'm I'm excited I hope we can have another conversation down the road uh tell tell my audience where uh they can find you and your videos uh at IQ Studios uh superintelligence.com I know is one anywhere else the single best place to go is superintelligence.com because it has links to the videos that are on other sites as well uh so we have a whole bunch of like short snack size you know one to three minute long videos as well as some longer keynotes at conferences and that sort of thing and we also have white papers and we have some articles that are pretty easy to understand.
00:39:56.880 --> 00:40:08.719
So um yeah I would encourage people to go there and then from there just uh keep on getting more educated and uh just remember that you know put your best foot forward online and um I think everything's gonna be great.
00:40:08.880 --> 00:40:15.199
There's lots of opportunity um but it's good to go in with our eyes wide open to understand you know where the risks are.
00:40:15.440 --> 00:40:19.280
I agree with you and I'm on board with uh with your viewpoints as well.
00:40:19.360 --> 00:40:21.840
So Craig thank you so much for joining me today.
00:40:21.920 --> 00:40:25.679
I appreciate you and uh look forward to connecting with you again.
00:40:26.079 --> 00:40:27.519
Thanks John this is really fun.
00:40:27.599 --> 00:40:28.320
I appreciate it.
00:40:28.480 --> 00:40:29.760
Okay take care.
00:40:30.079 --> 00:41:21.280
Okay you too well that was a learning experience for me and here's what stuck with me first this isn't a story that only experts get to write the idea that wisdom comes from many voices not a few is a lot like how a church a good business or a good family works no single person has all the answers and the trouble often starts when one voice is the only one that counts second you can take risk seriously without panicking those are two completely different things.
00:41:21.920 --> 00:41:54.960
Fear shuts people down honest concern gets people to ask better questions show up and make better choices and third stewardship applies here too we don't own this technology we're responsible for how it's used and that means being informed speaking up and has asking who it's for that includes how you use AI in your own business tomorrow morning.
00:41:56.239 --> 00:42:29.360
If there's one takeaway it's this don't sit in the stands learn enough to have an opinion and use your voice your vote and your business to push this in a direction that serves people I started this episode with a question who gets to decide where AI goes from here after this conversation I think the honest answer is that it depends on how many of us are willing to be part of the answer.
00:42:29.599 --> 00:42:50.960
Now that may sound like a heavy responsibility I'd encourage you to see it as an invitation you don't need a PhD a seat in the boardroom or a million followers to matter here you can start small and small is where most good things start.
00:42:51.440 --> 00:42:54.639
So here's what I ask you to do this week.
00:42:54.880 --> 00:43:07.119
Pick one first learn something watch a video read an article or have a real conversation with someone who sees AI differently than you do.
00:43:07.920 --> 00:43:27.119
Second look at how you use AI in your own work or in your personal life for that matter ask whether it's helping people being honest with them and keeping you in the driver's seat third talk about it.
00:43:27.360 --> 00:43:43.039
Bring it up at the dinner table at the office at your small group or at church these conversations shouldn't be left only to the experts my speaking should sometimes be left to the experts but that's a different story.
00:43:51.679 --> 00:44:06.079
And if it all feels like a lot take a breath you're not responsible for solving it alone scripture reminds us that we aren't given a spirit of fear but of power, love and a sound mind.
00:44:06.639 --> 00:44:15.119
That's the posture I'd like us to carry into this next chapter curious, careful and hopeful.
00:44:15.760 --> 00:44:26.079
If this conversation helped you please share it with a friend, a coworker or someone who's curious or nervous about where AI is headed.
00:44:26.400 --> 00:44:31.039
A calm, honest conversation can mean a great deal.
00:44:31.280 --> 00:44:45.119
And if you're a business owner trying to use AI wisely with integrity and without the hype or the panic that's exactly what I help clients with every day at Clearbrand Digital.
00:44:45.440 --> 00:44:55.519
So come find me at Clearbranddigital.com Well thanks for listening today to Truth in a Digital World I'm John Delaney.
00:44:55.760 --> 00:45:10.800
Let's talk about truth before we outsource it's a little bit more