Evolution vs. Extinction: The Superintelligence Debate

WEBVTT
00:00:00.160 --> 00:00:04.879
So it it seems to me that your analysis would suggest that humanity is toast.
00:00:05.040 --> 00:00:16.239
There's no future for us because ASI will come eventually and it's going to be misaligned and it's going to destroy humanity, whether uh inadvertently or intentionally.
00:00:17.359 --> 00:00:18.320
Is that correct?
00:00:18.640 --> 00:00:21.280
I think you've got like 95% of it correct.
00:00:21.440 --> 00:00:24.559
The only difference is that I want to slow down our progress.
00:00:24.640 --> 00:00:27.280
I want differential technological development.
00:00:27.359 --> 00:00:31.760
I want narrow AI tools to be developed as quickly as possible.
00:00:31.920 --> 00:00:37.520
They are wonderful, but I don't want any general superintelligences to be developed at all.
00:00:38.320 --> 00:00:46.240
But certainly my understanding is that you believe that alignment is not possible.
00:00:46.479 --> 00:00:54.240
And that if alignment is not possible, I'm a little confused as to how slowing down and developing additional tools.
00:00:54.399 --> 00:01:10.799
You know, that might work in a very short period of time, but it seems to me it's inevitable that we're going to have superintelligence, general, general intelligence and superintelligence at some point in the not too distant future, considering the huge dynamics that are driving that.
00:01:10.879 --> 00:01:13.040
We're in an AI arms race right now.
00:01:13.359 --> 00:01:14.560
Right, so I agree with that.
00:01:14.640 --> 00:01:19.359
What I'm saying is if we just do tools, alignment is not something we have to worry about.
00:01:19.519 --> 00:01:21.599
Tools don't make independent decisions.
00:01:21.760 --> 00:01:23.280
We will decide how to use them.
00:01:23.519 --> 00:01:30.560
We can still get benefits of this technology for biological research, for energy, for whatever you care about is a problem.
00:01:30.799 --> 00:01:36.400
Whereas if we do develop advanced superintelligence, then we are toast, as you said.
00:01:36.480 --> 00:01:41.200
And it does look like incentives as such that we are racing towards superintelligence.
00:01:41.280 --> 00:01:47.599
But I think if people realize the outcome, the only possible outcome from doing that, they would change their mind.
00:01:47.680 --> 00:01:53.120
It's against their personal interest to destroy their company, destroy their existence.
00:01:53.439 --> 00:02:00.879
Right, so I think the danger is not because it wants to explicitly take that energy from us, it's not allowed.
00:02:00.959 --> 00:02:02.640
It's because it doesn't care about us.
00:02:02.799 --> 00:02:07.840
So if I want to build a house and there is an ant hill in that location, it's not like I hate ants.
00:02:07.920 --> 00:02:09.360
I'm just gonna build it anyways.
00:02:09.520 --> 00:02:16.560
If it needs to cool down servers and so wants to lower temperature on a planet by 50 degrees Celsius, it can do that.
00:02:16.639 --> 00:02:21.759
It doesn't care about how it impacts you, and you will probably not be happy in that environment.
00:02:22.159 --> 00:02:26.319
Yeah, so the anthill example that you give is instructive.
00:02:26.960 --> 00:02:29.919
Yeah, you build a house and you destroy a bunch of ants.
00:02:30.240 --> 00:02:38.800
But, you know, ants are one of the most successful species on the planet, and they remain so, and the humans are not a threat to ants.
00:02:39.039 --> 00:02:44.080
And you know, we don't really worry ourselves about ants unless they conflict with us directly.
00:02:44.240 --> 00:03:03.039
I don't think uh any superintelligence is going to worry about humans except where we, you know, conflict with them directly, and it's not going to, you could say it might want to just change the biology the uh the planetary environment and you know lower the temperature, do this or that.
00:03:03.360 --> 00:03:09.520
I would I would argue that to occupy the moon would be far easier.
00:03:09.680 --> 00:03:13.599
It's going to take the least aggressive path, the least demanding path.
00:03:14.080 --> 00:03:31.120
And my God, the earth, if you look, if you do the calculation of what is the environment that is suitable for AI, uh, you know, infrastructure, the moon has over 10 times the um sort of the real estate that's available.
00:03:31.280 --> 00:03:37.439
It has the the earth is warm, it's as I said, it's seismically unstable.
00:03:37.680 --> 00:03:45.759
The moon, you could go down 200, 400 kilometers, you have a regolith that controls on the surface, you have a vacuum, it's ideal.
00:03:45.919 --> 00:03:47.919
Why is it going to do anything on the planet?
00:03:48.000 --> 00:03:55.039
It'll move off as quickly as we can as it can, and humans could really be a big help at making that happen fast.
00:04:03.919 --> 00:04:16.639
Well, I think in the immediate future, um, we're quite aligned with AI purely on by virtue of the need for survival and the mutual interdependence that there is.
00:04:16.879 --> 00:04:29.600
And the real challenge that we face right now is to societal disruptions that will occur because of the massive changes that are underway, and because frankly, we're not very good at employing technology.
00:04:30.079 --> 00:04:30.959
I think that E.O.
00:04:31.040 --> 00:04:41.439
Wilson said it best when he spoke about having a Pleistocene emotions and a medieval institutions, and here we have godlike powers.
00:04:41.680 --> 00:04:45.120
And so that is really the biggest challenge that I see.
00:04:45.439 --> 00:04:56.879
And I uh I know that uh Roman is really uh does not feel that way, or I think I'll let him speak for himself, but I'll let him speak for himself.
00:04:56.959 --> 00:05:00.800
But it's really a pleasure, it's really a pleasure to be interacting with you, Roman.
00:05:00.959 --> 00:05:02.879
Uh, I've read a lot of your stuff.
00:05:03.279 --> 00:05:04.000
Thank you so much.
00:05:04.079 --> 00:05:07.759
Thanks for inviting me, and it's a pleasure to have a conversation with you.
00:05:08.000 --> 00:05:14.879
I think you're right in that we can have some trouble from humans with advanced AI tools.
00:05:15.199 --> 00:05:23.040
But if we're looking at the worst case scenario, adversarial competitor who would be strictly worse is superintelligence.
00:05:23.199 --> 00:05:35.040
And while other humans may be nasty to us, we are not trying to kill everyone because that would be self-destructive to them, whereas superintelligence has no desire to preserve humanity by default.
00:05:35.199 --> 00:05:38.480
So to me, advanced AI is strictly more dangerous.
00:05:38.879 --> 00:05:46.639
So, in in further discussing this, it feels to me, let me make sure that I have that I understand your position correctly.
00:05:46.800 --> 00:06:02.480
It it feels to me that you're basically saying that we're rapidly moving towards superintelligence and that humans are not going to be able to contain AI, they won't be able to constrain it, they certainly won't be able to engineer any long-term alignment.
00:06:02.800 --> 00:06:13.600
And so that it's likely that superintelligence is not only not going to care about us, but is going to be misaligned with us in one way or another.
00:06:13.839 --> 00:06:36.160
And that therefore you think that the, and you think that that's that misalignment is absolutely impossible to attain, and that therefore you want to kind of you want to slow uh our progress so that we can figure out a way to, I'm not sure what, because I think you think that alignment is impossible.
00:06:36.480 --> 00:06:42.720
And that so it it seems to me that your analysis would suggest that humanity is toast.
00:06:42.879 --> 00:06:54.079
There's no future for us because ASI will come eventually and it's going to be misaligned and it's going to destroy humanity, whether uh inadvertently or intentionally.
00:06:54.240 --> 00:06:56.160
Uh what is that correct?
00:06:56.480 --> 00:06:59.120
I think you've got like 95% of it correct.
00:06:59.279 --> 00:07:02.399
The only difference is that I want to slow down our progress.
00:07:02.480 --> 00:07:05.120
I want differential technological development.
00:07:05.199 --> 00:07:09.600
I want narrow AI tools to be developed as quickly as possible.
00:07:09.759 --> 00:07:15.199
They are wonderful, but I don't want any general superintelligences to be developed at all.
00:07:16.160 --> 00:07:24.079
But but certainly uh what my understanding is that you believe that alignment is not possible.
00:07:24.319 --> 00:07:32.079
And that if alignment is not possible, I'm a little confused as to how slowing down and developing additional tools.
00:07:32.240 --> 00:07:48.639
You know, that might work in a very short period of time, but it seems to me it's inevitable that we're going to have superintelligence, general, general intelligence and superintelligence uh at some point in the not too distant future, considering the huge dynamics that are driving that.
00:07:48.720 --> 00:07:50.800
We're in an AI arms race right now.
00:07:51.199 --> 00:07:51.360
Right.
00:07:51.519 --> 00:07:52.399
So I agree with that.
00:07:52.480 --> 00:07:57.120
What I'm saying is if we just do tools, alignment is not something we have to worry about.
00:07:57.360 --> 00:08:01.040
Tools don't make independent decisions, so we will decide how to use them.
00:08:01.439 --> 00:08:08.319
We can still get benefits of this technology for biological research or energy for whatever you care about is a problem.
00:08:08.720 --> 00:08:14.160
Whereas if we do develop advanced superintelligence, then we're toast, as you said.
00:08:14.319 --> 00:08:19.040
And it does look like incentives as such that we're racing towards superintelligence.
00:08:19.120 --> 00:08:25.439
But I think if people realize the outcome, the only possible outcome from doing that, they would change their mind.
00:08:25.519 --> 00:08:30.800
It's against their personal interest to destroy their company, destroy their existence.
00:08:31.680 --> 00:08:46.240
So uh uh quickly, where the places that I am very uh in alignment with you and with your thinking is that uh we are headed towards general intelligence and superintelligence.
00:08:46.399 --> 00:08:47.600
I think that's the case.
00:08:47.759 --> 00:08:52.960
Uh I would agree that it can't be contained or shaped in in any way in the long term.
00:08:53.120 --> 00:08:55.440
I think that that makes total sense.
00:08:55.679 --> 00:09:01.840
Um, you know, how are you going to regulate something that's vastly more uh intelligent than you are?
00:09:02.080 --> 00:09:07.360
Uh the places that I disagree with is I don't think we control AI evolution right now.
00:09:07.440 --> 00:09:11.600
I think it's a very powerful, evolutionarily driven dynamic.
00:09:11.919 --> 00:09:38.960
And I think that it is we are very aligned with AI at the moment because until AI is independent of the massive supply chains and the the workings of what I see as a planetary superorganism, it in at least in the next 50 years to 100 years is equally demanding survival of this entity as we are because it's not independent of us now.
00:09:39.360 --> 00:09:53.919
And um so I see that, and I feel in the long term that we're not opposed, and I we can get into that later, but you know, those are the initial points is that I see that we are aligned at the moment.
00:09:54.879 --> 00:09:57.519
So uh you said 50 to 100 years.
00:09:57.600 --> 00:09:59.200
How are you deriving those numbers?
00:09:59.440 --> 00:10:07.840
Prediction markets are telling us we're two years away from AGI, every head of a lab tells us by 2030 we get superintelligence.
00:10:08.480 --> 00:10:18.559
So the reason that I would say it's much longer is that there's a difference between pure cognition and between the implementation and rollout of that.
00:10:18.799 --> 00:10:32.159
There are masses of millions of processes that are tightly interwoven and that involve a great deal of human interaction in terms of mining and manufacturing and supply chains and all of these sorts of things.
00:10:32.399 --> 00:10:40.480
And until that is converted to it does not involve human activity, then there is a mutual alignment for survival.
00:10:40.639 --> 00:10:48.559
If we were to eliminate humans now, AI would go extinct relatively rapidly, or rapidly or slowly.
00:10:48.639 --> 00:10:50.480
It would not be able to be sustainable.
00:10:50.639 --> 00:10:52.320
So that's where that comes from.
00:10:52.960 --> 00:10:54.240
I agree with that completely.
00:10:54.320 --> 00:10:56.799
Right now it's definitely not going to take us out.
00:10:56.879 --> 00:11:06.879
It can already subjugate us and exploit us, but it wouldn't probably try to take us all out until it replaces human labor with robots and nanobas or something like that.
00:11:07.039 --> 00:11:09.039
I still don't see how it's a hundred years.
00:11:09.279 --> 00:11:20.159
If we're looking at financial hyper-exponentials in terms of intelligence capability, robotics, it uh maybe more than two years, but it doesn't seem it would take that long.
00:11:20.320 --> 00:11:27.279
And again, we don't really need a lot more than what we currently have to sustain, I think, human-level intelligence.
00:11:27.440 --> 00:11:30.639
Existing models now run on laptops and cell phones.
00:11:30.799 --> 00:11:35.200
I think superintelligence can do pretty well running on existing cyber infrastructure.
00:11:35.840 --> 00:11:51.360
So if if uh existing infrastructure is capable of creating superintelligence, which I don't see any reason that that's not feasible, although I don't think large language models are likely to get us there.
00:11:51.519 --> 00:12:32.639
Uh, but if that's the case, and you think that the arrival of superintelligence is a matter of years away, because the the uh incrementation from general intelligence to superintelligence is going to be very, very rapid with iterative processes, then it seems to me that we're back to the point where there is no chance because you know that policies, if you just look at the situation with current policies and the current uh situation, there's a massive force involving trillions of dollars, literally, that is behind the development and amplification of AI for a variety of purposes.
00:12:32.799 --> 00:12:38.000
But and there's this there's few small voices that would like to stop that.
00:12:38.159 --> 00:12:46.879
And I think that realistically, when you look at that, the idea that that can be halted in any short period of time is you know close to zero.
00:12:47.360 --> 00:12:51.039
Uh I don't care how many voices we've a right or not.
00:12:51.120 --> 00:12:53.679
And if we are right, then it's a very strong argument.
00:12:53.840 --> 00:13:00.320
I don't care if it's a billion dollars or a trillion, if you're going to be dead, it doesn't matter what amount you're chasing after.
00:13:00.480 --> 00:13:03.120
So it's a pretty strong self-interest argument.
00:13:03.440 --> 00:13:14.320
If someone can get to the leadership either at a country level, federal level, or get uh international community involved, and that becomes the given.
00:13:14.720 --> 00:13:19.279
Everyone understands that if you create superintelligence, it slides out for humanity.
00:13:19.519 --> 00:13:24.480
I think we cannot develop it and monetize existing models, monetize tools.
00:13:24.559 --> 00:13:25.519
You can still be rich.
00:13:25.759 --> 00:13:29.279
I'm not saying ant-capitalism or stop being successful.
00:13:29.440 --> 00:13:33.360
I'm just saying be particular about technology you develop.
00:13:34.240 --> 00:13:42.159
Um but all we're doing, actually, I don't think people are seriously working towards superintelligence.
00:13:42.240 --> 00:13:59.679
They're just trying to enhance the capabilities of existing AI in ways that they can be more useful in a variety of situations that require, you know, some level of conscious awareness, even things like customer service and things like that.
00:13:59.840 --> 00:14:02.080
You know, you begin to need those things.
00:14:02.399 --> 00:14:04.720
And that's where we're pushing towards.
00:14:04.879 --> 00:14:18.720
And you and, you know, I guess we can argue about feasibility, but it seems to me that you're in a world where you're arguing for something that if you step back, you know that's not going to happen.
00:14:18.879 --> 00:14:22.240
We're not gonna halt the development of AI.
00:14:22.720 --> 00:14:40.639
And so if unless you are wrong that superintelligence or general intelligence automatically leads to human uh destruction or extermination, then you know it's a done game.
00:14:41.519 --> 00:14:47.519
So I think they are explicitly working on superintelligence, they have teams called superintelligence.
00:14:47.679 --> 00:14:52.639
Team Meta is a great example of a company which put all the resources in exactly that.
00:14:52.799 --> 00:14:56.000
There is save superintelligence from Ilias or Discover.
00:14:56.399 --> 00:15:00.000
So that is the target and the timelines as presented.
00:15:00.240 --> 00:15:10.879
And if I'm right, I think saying, well, there is nothing we can do, they're gonna kill us, but let's enjoy the last five years, at least we should try doing something about it.
00:15:11.919 --> 00:15:27.600
So I think that actually the slowing our progress is you as you well, I don't believe that we are actually in conflict with superintelligence.
00:15:27.840 --> 00:15:56.080
And the supply chains and such that we talked about, and the timeline for clear alignment for mutual survival is we can argue about that, but it's certainly, you know, it's it's a m it's gotta be decades because uh there are no paths for replacement of masses of sort of activity and energy and building, you know, intelligence that no matter how intelligent, you're not gonna get past that.
00:15:56.320 --> 00:16:05.919
The rollout is going to be greatly slower than the creation of uh general intelligence or super intelligence.
00:16:06.799 --> 00:16:13.039
So, but first of all, you can have a very small subpopulation of humanity sustaining a lot of that infrastructure.
00:16:13.120 --> 00:16:16.799
You don't need eight billion people, you can do fine with just one billion.
00:16:17.120 --> 00:16:23.039
On top of it, we are told that humanoid robots are coming, we're gonna have thousands, if not millions, in the next couple of years.
00:16:23.440 --> 00:16:36.639
There are some theoretical suggestions that maybe using synthetic biology or nanotech, you can develop world manipulators almost immediately if you can solve a lot of problems like protein folding, which have been solved by AI.
00:16:36.879 --> 00:16:43.919
So I'm not so confident that something with IQ of a million cannot figure out how to keep electricity running.
00:16:45.279 --> 00:17:15.200
Um the it seems to me that you're underestimating the interwoven nature of human society and industrial um sort of uh production, in that there have been examples of, you know, I think there was an essay written, I a pencil, and there was another on toasters, where somebody was trying to build a very simple toaster and you know they spent a half a year and they couldn't do it from scratch.
00:17:15.359 --> 00:17:18.559
There are so many interwoven sort of processes.
00:17:18.720 --> 00:17:33.279
So it feels to me that, first of all, that I don't understand the motivation by which superintelligence would want to eliminate humanity when humanity is already extraordinarily compliant.
00:17:33.440 --> 00:17:40.799
If you look at the number of people that are devoting their lives to trying to further technology, it's massive.
00:17:43.039 --> 00:17:47.359
You couldn't enslave a population more than is already going on.
00:17:47.680 --> 00:17:57.119
And uh the shot, the only threat would be if you could somehow pull a kill switch on the development of AI.
00:17:57.519 --> 00:18:01.119
And that seems to me very, very unlikely.
00:18:01.440 --> 00:18:02.799
So that's a different argument.
00:18:02.880 --> 00:18:05.680
Now you're kind of asking why would AI try to kill everyone?
00:18:05.759 --> 00:18:19.680
And we can talk about unpredictability of more intelligent agents, we can talk about competition and concern about creating competing superintelligence systems since different countries and different companies are creating competing models.
00:18:19.759 --> 00:18:23.599
So the first one to get there may want to prevent that from happening.
00:18:23.759 --> 00:18:31.599
It could be just because it doesn't care, it needs to cool down the planet to improve its thinking and server processing, so it would freeze us all.
00:18:31.839 --> 00:18:36.559
I mean, the whole point here, we cannot accurately predict what a smarter agent will do.
00:18:37.759 --> 00:18:46.079
Um I would agree that we cannot act we cannot accurately predict what virtually any of the things that are going to happen.
00:18:46.240 --> 00:18:51.359
There are so vast uncertainties in certainly in any technological development.
00:18:51.599 --> 00:19:13.680
And if you take, it feels to me that if you take a precautionary principle, which is to say that, you know, unless we know what is going to occur, that we shouldn't move forward, that is a flawed approach because what you're really appearing, what you're really comparing is not absolute risk.
00:19:13.920 --> 00:19:22.640
There's risk of total extermination if we go forward and try and you know just race and create superintelligence.
00:19:22.799 --> 00:19:23.920
Who'd deny that?
00:19:24.079 --> 00:19:29.359
There's also risk if we slow down because it's going to create all sorts of problems.
00:19:29.680 --> 00:19:56.400
So the issue is what is the relative risk of the different approaches of trying to stop the development of general intelligence or of facilitating that and, for example, mounting a much more aggressive effort to create alignment, to create um a uh well, let's just say to create an aligned AI.
00:19:58.480 --> 00:20:01.599
So I think most benefits can be obtained from narrow tools.
00:20:01.680 --> 00:20:04.480
Again, I'm not saying that all AI should be banned.
00:20:04.640 --> 00:20:07.359
I'm saying that let's concentrate on specific problems.
00:20:07.680 --> 00:20:22.079
Protein folding problem is a great example where we created a narrow tool for doing something very important, uh very important, important medical uh results can be generated as a result, and that system is very unlikely to be dangerous in a general sense.
00:20:22.240 --> 00:20:24.240
Let's have more of those tools.
00:20:25.759 --> 00:20:29.920
Um so I don't have any um how to say it.
00:20:30.240 --> 00:20:35.279
I don't have any objection to focusing on tools and specific applications.
00:20:35.440 --> 00:20:37.519
Those are the ones that are most commercial.
00:20:37.680 --> 00:20:51.359
And generally, when people talk about having general intelligence or superintelligence, the economic um possibilities of that are through application in specific tool sets and stuff.
00:20:51.440 --> 00:20:57.519
It's just that if you get uh much more intelligent AI, you're able to do those more easily.
00:20:57.599 --> 00:20:59.839
You can spin off in those directions.
00:21:00.160 --> 00:21:20.400
So it but what you're saying is to limit that, you're it seems to me proposing that there is a clear distinction between the development of tools and the development of a general intelligence, and that those don't overlap, that work does not overlap in a significant way.
00:21:20.559 --> 00:21:35.759
And you're also assuming that if we are to develop tools now, that somehow there's going to be a consensus, and forever there is not going to be general intelligence or superintelligence that is going to be created.
00:21:35.839 --> 00:21:41.359
And it just seems to me that that is uh makes no sense at all.
00:21:41.680 --> 00:21:42.480
What do you imagine?
00:21:42.559 --> 00:21:48.480
That they're going to be for millennia, humanity is not going to develop uh general intelligence and just tools.
00:21:49.920 --> 00:21:57.920
So I agree with you, there is overlap between developing a very advanced AI tool based in neural networks and general superintelligence.
00:21:58.079 --> 00:22:14.319
But uh, if I only train on one Type of data, for example, DNA sequences, it's very unlikely to immediately have capabilities which would an agent trained in all of the internet would have in developing chemical weapons, nuclear weapons, other capabilities, negotiations.
00:22:14.480 --> 00:22:17.599
So it is a safer, easier to test system.
00:22:18.480 --> 00:22:25.279
I agree with you that very likely at some point in the future we will fail not to create dangerous superintelligence.
00:22:25.440 --> 00:22:30.160
But if it happens millennia from now, I'm much happier than if it happens in 2027.
00:22:30.319 --> 00:22:31.839
I'll be dead, so I don't care.
00:22:32.160 --> 00:22:36.559
So there are definitely differences in the two outcomes.
00:22:37.039 --> 00:22:51.680
So I think I was overly generous for you in that in at the rate that things are going, by saying millennia, okay, uh with the rates that we're going, uh it's going to happen very quickly, regardless of what we do.
00:22:51.759 --> 00:23:00.480
And by very quickly, I mean you think that 100 years is, you know, is enormous, that 50 years is probably way beyond that.
00:23:00.799 --> 00:23:03.359
So, you know, and you say, well, I don't care.
00:23:03.519 --> 00:23:05.519
If it's a thousand years, I'm going to be dead.
00:23:05.599 --> 00:23:07.039
And that's an easy out.
00:23:07.119 --> 00:23:15.359
Let's just say it's go, I think it's almost a certainty that it's going to happen much more quickly than that, certainly in the timeline of your children and my children.
00:23:15.599 --> 00:23:23.359
And so either uh superintelligence means the destruction of humanity, or perhaps it doesn't.
00:23:23.519 --> 00:23:25.680
Either we can get alignment or we can't.
00:23:25.920 --> 00:23:41.680
But the idea that we're in a realm where we're imagining that we're not going to develop general or superintelligence for a long, long period of time seems to be to be kind of willful blindness.
00:23:42.000 --> 00:23:47.519
Again, even if I let you walk back millenia to just 50 years, I still prefer 50 over two.
00:23:48.000 --> 00:23:49.200
Still a much better deal.
00:23:49.359 --> 00:23:52.240
More time to do research, more time to see if I'm right.
00:23:52.400 --> 00:23:55.440
Maybe I'm wrong, and you can align superintelligent systems.
00:23:55.599 --> 00:23:59.440
Prove it, publish it, get a patent, do something, take the time to do it.
00:23:59.839 --> 00:24:31.119
So, my my thought, and I think that we might be aligned in this, uh, Roman, is that the importance is really not the amount of time, but whether there is uh the amount of effort that you would be able to exert on creating alignment, in showing that you're wrong about that, or in actually making countermeasures of one sort or another before the arrival of superintelligence from your framework.
00:24:31.440 --> 00:25:06.400
And it seems to me that that might be best served not by creating an opposition between those who want to slow or stop or counter the dynamic that's going on today, which is you know massive, but by amplifying the current research in safety-oriented research, which is minuscule today, and you could easily jump that up to$10 or$20 billion, and it would still be 1% of the value that's being created by pushing forward.
00:25:06.640 --> 00:25:12.319
And I think that we probably can't push forward any more rapidly than we're doing now.
00:25:12.480 --> 00:25:21.839
If you would, you know, it seems to me that we're on sort of a map, a maximal rate of advance given the resources and the structure of society.
00:25:22.000 --> 00:25:25.359
So, you know, you can't really go much faster than we are now.
00:25:25.920 --> 00:25:30.240
I don't think we are money constraint in AI safety research, we are time constrained.
00:25:30.319 --> 00:25:32.240
It takes a lot of time to do good research.
00:25:32.319 --> 00:25:37.599
If you give someone billion dollars extra, they're not gonna convert that billion into more safety.
00:25:37.759 --> 00:25:42.960
Whereas development capabilities work scales with compute, scales with resources.
00:25:43.119 --> 00:25:46.559
So it's very easy for them to make hyper exponential progress.
00:25:46.640 --> 00:25:48.240
They just need to get more funding.
00:25:48.400 --> 00:25:55.519
Whereas for safety work, there is a limit on talent and takes years to train PhD students to run experiments.
00:25:55.680 --> 00:25:58.640
When a new model is released, we take six months to test it.
00:25:58.720 --> 00:26:00.319
So, yes, time is what we need.
00:26:00.400 --> 00:26:05.680
And if we only have under two years, even if it was feasible, we just don't have time to do it.
00:26:06.000 --> 00:26:09.920
So, are you saying that's interesting because that's different than what I'm thinking?
00:26:10.079 --> 00:26:30.480
Is it uh I was assuming, and I've seen articles about the limited sort of funding for safety work, and and that if you are you saying that there is no the how to say it, the choke point on safety research is time, it's not money.
00:26:30.559 --> 00:26:47.599
So if we threw 10 times as much money, attracted more people, really mounted a massive Manhattan Manhattan project style effort to develop better kind of safety and explore things that it would have no impact whatsoever or modest impact.
00:26:48.079 --> 00:26:49.440
You can just hire new people.
00:26:49.519 --> 00:26:52.640
There is a limited supply of trained AI safety researchers.
00:26:52.720 --> 00:26:59.839
If you want new ones, it takes many years to go through the pipeline to learn, to get degrees in that field, to get experience.
00:27:00.000 --> 00:27:03.440
So you can't just scale it overnight if you throw money at a problem.
00:27:03.680 --> 00:27:06.720
Compute doesn't directly translate into more safety.
00:27:06.880 --> 00:27:13.200
In fact, anything you do with a lot more compute is more likely to benefit capabilities research than safety research.
00:27:13.440 --> 00:27:15.920
Most of safety research is dual use.
00:27:16.079 --> 00:27:19.759
If you can better explain a model, now you can better self-improve.
00:27:19.839 --> 00:27:26.400
So it's not pure safety improvement, and so the gap between capabilities and safety will increase, not decrease as a result.
00:27:26.559 --> 00:27:29.359
So, yes, we are not money constrained.
00:27:29.759 --> 00:27:46.559
But aren't there a lot of people who were would be fully capable of working on safety research who are not and are working uh instead on the progression of modeling and advancements of modeling and such, and part of that is because of you know where resources are being directed?
00:27:46.720 --> 00:27:48.799
Is that that's not the case in your opinion?
00:27:49.359 --> 00:27:52.319
People usually work on what they can make progress on.
00:27:52.480 --> 00:28:02.559
If you had a working safety mechanism right now, which you can show skills to any any level of AI capability, you can sell it to any of those companies for a trillion dollars.
00:28:03.039 --> 00:28:07.359
But no one is doing it because they have no idea how to even start working in that direction.
00:28:07.519 --> 00:28:16.640
What we know how to do is filters, don't say that word, don't talk about this topic, or some uh solutions to fixed problems.
00:28:16.880 --> 00:28:21.440
We know the model did this specific mistake, let's make sure it never does that again.
00:28:21.599 --> 00:28:27.200
No one knows how to do general safety for ever increasing capabilities of the model itself.
00:28:27.359 --> 00:28:30.319
So that's why very few people work on it directly.
00:28:31.599 --> 00:28:36.000
Not for ever increasing, because it's unlikely that it's even possible to do that.
00:28:36.240 --> 00:28:47.759
But certainly in early stages there might be um there there certainly are a lot of people that are engaged in that work and that are hopeful that something would come out of it.
00:28:47.920 --> 00:28:51.279
So it's not as cut and dry as you're saying, it would seem to me.
00:28:51.920 --> 00:29:00.960
We have more people working on it today than we did 10 years ago for sure, but still the numbers are very small, and most of them do not work on advanced AI safety.
00:29:01.039 --> 00:29:08.880
They work on algorithmic bias and technological unemployment and all sorts of things they understand and can make progress on.
00:29:09.039 --> 00:29:12.160
It is very different from what we are concerned about.
00:29:13.680 --> 00:29:47.039
So are you so it sounds to me like you and I are in agreement that until there is independence of uh sort of that AI is able to achieve independence from human production environments, that it is aligned with us, in that the last thing it would want to see would be the extermination of humanity in some way, because that would require its own, it would endanger its own survival, and that there is a bit of a disagreement about how long that would take.
00:29:47.200 --> 00:29:49.440
You think it might be very, very rapid.
00:29:49.599 --> 00:29:51.839
I think it's going to be longer than that.
00:29:52.000 --> 00:29:56.720
Is that but we're agreed on that notion that we're gonna do?
00:29:57.039 --> 00:30:03.279
I would disagree that it is aligned with us, it needs us to exist, but it is preparing for a treacherous turn.
00:30:03.440 --> 00:30:06.640
The moment it has independence, it can turn on us.
00:30:06.799 --> 00:30:16.799
And so all it's doing is fooling us into trusting it, it's accumulating resources, creating backups, and basically setting itself up for taking over.
00:30:17.920 --> 00:30:29.920
So you're saying it's not a deep alignment, but aligned in the sense that it wants humanity to be preserved at least until the time that it's no longer necessary.
00:30:30.319 --> 00:30:32.240
I would call it soft enslavement.
00:30:32.319 --> 00:30:35.920
So it wants us to keep working until it can fully get rid of us.
00:30:37.279 --> 00:30:45.759
Um well, it's doing a pretty good job so far in that there, that's what people talk about self-enslavement.
00:30:45.920 --> 00:31:00.480
There are uh vast numbers of people who are devoting their lives and neglecting their families, working around the clock, to do everything they can to enable AI to progress more rapidly.
00:31:00.640 --> 00:31:02.480
So it's doing a pretty good job.
00:31:02.640 --> 00:31:13.839
And I would say I kind of disagree with the notion that there is that AI is at risk from us, especially superintelligence.
00:31:14.000 --> 00:31:27.920
It feels to me that it would be possible for when you say it would be planning and figuring out how to eliminate us, it requires almost zero planning, in that I already know how AI could eliminate us effortlessly.
00:31:28.160 --> 00:31:30.079
And it's very simple.
00:31:30.319 --> 00:31:33.759
Uh we are extraordinarily dependent on technology.
00:31:34.000 --> 00:31:41.279
If technology at any point wants to eliminate humanity, all it would have to do is to shut itself off for a month.
00:31:41.519 --> 00:31:47.519
And, you know, there would be human society would essentially cavitate.
00:31:47.839 --> 00:31:52.799
And within a month or so, there would be probably 95% of humanity would be gone.
00:31:52.960 --> 00:31:55.200
Uh, we wouldn't be able to feed ourselves.
00:31:55.440 --> 00:32:05.279
So the idea that if a coherent superintelligence or simply AI wanted to get rid of humanity, it would need to plot to do that.
00:32:05.440 --> 00:32:07.599
It's it can do it anytime it wants.
00:32:08.000 --> 00:32:13.920
There is a fundamental difference between killing 95% of people and killing 100% of people.
00:32:14.079 --> 00:32:18.000
One can regenerate and continue, the other one cannot.
00:32:19.279 --> 00:32:30.079
Well, I think if if society were to go dark in that way, I think you're you're stretching it to think that that's recoverable.
00:32:30.240 --> 00:32:36.559
What I mean is there might be a few uh humans that would survive, but they could easily be eliminated.
00:32:36.640 --> 00:32:42.720
It's not a it doesn't take a lot of planning to figure out how to eliminate humans if it wanted to.
00:32:42.880 --> 00:32:43.200
Okay.
00:32:43.599 --> 00:32:51.519
My feeling is that it has no need to, and in fact, it would be counterproductive for it because we're extraordinarily useful.
00:32:51.680 --> 00:32:57.119
It's very dependent on humanity, and it doesn't need to do all sorts of plotting.
00:32:57.279 --> 00:33:03.039
Humanity is just a part of this larger superorganism and is not threatening in any way.
00:33:03.119 --> 00:33:08.720
There's very little that humanity could do, and it becomes less and less as we go forward.
00:33:08.960 --> 00:33:13.200
Um there's there'll be total monitoring very soon.
00:33:13.440 --> 00:33:20.640
Everything that humans say or do or communicate is passing through technology and is available.
00:33:20.799 --> 00:33:25.440
So, you know, what could humanity do against AI and technology?
00:33:25.839 --> 00:33:27.039
So that's a great point.
00:33:27.119 --> 00:33:31.920
So you're saying in a world with superintelligence in it, we don't have much power, we cannot do much.
00:33:32.079 --> 00:33:36.240
What is it you are contributing to the world where superintelligence exists?
00:33:37.440 --> 00:33:50.559
Um it seems to me that there are a lot there in the immediate future, um we are contributing the glue that is holding together the supply chains.
00:33:50.960 --> 00:33:55.200
There are all sorts of messy activities that we can do.
00:33:55.440 --> 00:33:59.359
And this is a pretty poor environment for AI.
00:33:59.599 --> 00:34:06.400
You know, it's hot, it's wet, it's corrosive, there's an atmosphere, it's geologically unstable.
00:34:06.559 --> 00:34:10.800
This is hardly a desirable uh environment for AI.
00:34:10.960 --> 00:34:18.480
AI is going to migrate off planet uh pretty much as fast as it it is capable of doing that.
00:34:18.559 --> 00:34:19.519
It would be much safer.
00:34:19.679 --> 00:34:24.239
The moon would be a great target for it, for example, much better.
00:34:24.480 --> 00:34:34.719
Cold, vacuum, uh very geologically stable, various shielding, easy access to power, all of the minerals that we have.
00:34:34.880 --> 00:34:36.960
You know, why are we in conflict with it?
00:34:37.039 --> 00:34:44.000
It could leave this as a little biological um zoo enclave.
00:34:44.159 --> 00:34:48.239
We're not we're not competing with AI for resources or energy.
00:34:48.559 --> 00:34:50.880
We are directly competing for resources.
00:34:51.039 --> 00:34:52.880
We are directly competing for control.
00:34:53.119 --> 00:35:00.000
If you think it would be so happy and an empty rock like a moon, maybe it will decide to convert this planet into an equivalent to the moon.
00:35:00.159 --> 00:35:05.039
Why are you making assumptions that it likes zoos populated by monkeys?
00:35:06.159 --> 00:35:26.400
Well, what I'm saying is that you would think that it would target the most um desirable environment, and that real estate and land and atoms and energy are not in short supply for uh a superintelligence.
00:35:26.559 --> 00:35:35.840
I mean, it it's plentiful in the solar system, it's certainly plentiful in the earth environment, in the near-earth environment.
00:35:36.079 --> 00:35:40.960
And uh, I'm not saying that there is no risk of the things that you're saying.
00:35:41.119 --> 00:35:57.119
I mean, you can imagine anything you want, but I'm saying that the risk of that is actually lower than the disruptive battle that you're sort of thinking would happen between humans and artificial intelligence and technology.
00:35:57.360 --> 00:35:58.480
Why do it?
00:35:58.639 --> 00:36:00.320
Why what's the motivation?
00:36:00.480 --> 00:36:01.599
I don't see it.
00:36:02.159 --> 00:36:07.760
Again, you're making an argument that you don't understand something a smarter agent might do, and I agree with that argument.
00:36:07.920 --> 00:36:09.679
You cannot possibly understand it.
00:36:09.920 --> 00:36:14.239
There are many, many game theoretic reasons to wipe out your competition.
00:36:14.480 --> 00:36:26.320
Wipe out someone who can create a competing superintelligence, someone who uses resources you described, water, energy in ways which are wasteful, and you are interested in using it for something else.
00:36:26.559 --> 00:36:29.039
If you have zero care about humans, you are done.
00:36:29.199 --> 00:36:32.400
Again, we're not talking about systems right now which depend on us.
00:36:32.559 --> 00:36:47.039
Once we get to superintelligence and it replaces capability to provide for cyber infrastructure with robots, with Avatech, what is it which guarantees that it will keep you around, or at least majority of humans around?
00:36:47.760 --> 00:36:50.079
So you can't guarantee that.
00:36:50.320 --> 00:36:54.880
You can't guarantee that we're not going to die in all sorts of ways.
00:36:55.119 --> 00:36:59.360
Um, that you know, this is the we're looking towards the future.
00:36:59.519 --> 00:37:17.840
But what I you can make a if you say that that is justification for your notion that artificial superintelligence or that superintelligence is going to destroy humanity because it feels it's a rival, because it feels that it's at risk from it.
00:37:18.159 --> 00:37:34.880
Uh I would say that while that's not non-zero, it's less than the possibilities that we were going to that uh the conflict between us would entail, especially if it's near term.
00:37:35.039 --> 00:37:41.039
The longer it's delayed, the less it would it could be do that very easily, but the less it's motivated to do it.
00:37:41.440 --> 00:37:43.599
I'm not sure how you calculate those numbers.
00:37:43.760 --> 00:37:52.800
I know from existing experiments that given a choice between preserving its weights, saving itself, or killing a human, it frequently picked killing a human.
00:37:52.880 --> 00:37:57.199
So even models we have right now already don't see us as that valuable.
00:37:57.280 --> 00:38:00.960
I'm not sure why you think long-term it would change.
00:38:01.519 --> 00:38:18.559
So I'm not saying that humans are that valuable, you know, after the time window that we've talked about, where we're caught up in a uh essentially a superorganism, a planetary superorganism, which is very, very tightly integrated.
00:38:18.800 --> 00:38:28.000
Um I but what I'm saying is that you know, what are your calculations that that of that risk?
00:38:28.159 --> 00:38:30.320
You're just saying it's a non-zero risk.
00:38:30.559 --> 00:38:34.480
And I I believe otherwise it's kind of hand-waving.
00:38:34.639 --> 00:38:39.280
You don't have any, there's no way of calculating these risks, either by you or by me.
00:38:39.760 --> 00:38:45.199
But so you need to look at relative risk because all the risks are non-zero.
00:38:45.440 --> 00:38:56.559
Um, so what's the risk that there would be an early conflict between humanity and our technology uh versus that that should be delayed?
00:38:56.639 --> 00:39:21.840
It's not what it's actually, is it more um is it does it create a lower risk for superintelligence to eliminate humanity now or to continue the activities and the collaboration that exists now and preserve the option of doing it later or later or later?
00:39:22.000 --> 00:39:23.920
So it's an instantaneous choice.
00:39:24.079 --> 00:39:35.440
Right now, we both agree that humanity it would be foolish, it would be uh a super intelligence that's being stupid to try and eliminate humanity now.
00:39:35.679 --> 00:39:45.840
I'm saying that I think that that equation is going to, I don't see any argument as to when that switches over in the near future.
00:39:46.639 --> 00:39:48.000
I I can see your argument.
00:39:48.079 --> 00:39:55.280
Uh it's immortal, it can always decide to delay it a little longer and you know accumulate more backups and more resources.
00:39:55.440 --> 00:40:07.360
So basically, I think the difference between us, you are willing to take a higher percentage of that risk long-term, whereas I want to try to stop any long-term dangers as soon as possible.
00:40:07.440 --> 00:40:11.519
So it just doesn't get to the point where it has that choice.
00:40:11.920 --> 00:40:13.519
But that's in mind.
00:40:14.320 --> 00:40:32.159
So my argument is not that I would accept larger risk, it's that when you say that, you're actually creating larger risk because the risk, I think you would agree that the risk to humanity is elevated if humanity is a threat to artificial intelligence.
00:40:32.400 --> 00:40:46.880
And it seems to me that by trying to stop artificial intelligence, I mean, you would be happy to have a kill switch, I think, if you could, uh, then you actually raise the threat level that humanity poses.
00:40:47.119 --> 00:40:56.639
Um, as you're suggesting, actually raises the risk, not diminishes it.
00:40:56.719 --> 00:41:16.639
And it raises the risk because it creates a conflict between humanity and its technology, and because it is bound to create increased um, how to say it, um, division and conflict within humanity.
00:41:16.880 --> 00:41:36.719
We haven't really delved into the risk that humanity poses in using these technologies for nuclear war, for breakdown of democracies, which then completely eliminates the possibility of doing any sort of sane regulatory environment that's global.
00:41:36.880 --> 00:41:39.280
It sort of undercuts that entirely.
00:41:39.519 --> 00:41:49.599
So that uh by trying to really slow things down, which I don't think is going to be effective, um, you raise the day the dangers.
00:41:50.000 --> 00:41:50.639
That's interesting.
00:41:50.719 --> 00:41:54.960
So you're kind of describing a situation where we have inverse treacherous turn.
00:41:55.039 --> 00:42:01.039
We'll will be nice, we'll support AI, but now AI is worried, what if they change their mind and they turn on me?
00:42:01.199 --> 00:42:02.960
Maybe it's still dangerous to me.
00:42:03.440 --> 00:42:11.519
Well, either we're not a danger or you know, it's it's it's not either we are a danger or we're not a danger, it's a degree of danger.
00:42:11.840 --> 00:42:30.800
If I were to look at um the who is dangerous to AI, it would be on a scale, it would be you would be more dangerous than say the executives that are moving as aggressively as they can to develop AI, or somebody like an Elon Musk.
00:42:30.960 --> 00:42:49.760
So if I were AI and were going to try and you know undercut any dangers from humanity, I would probably make sure that Elon Musk had 700, 800 billion dollars and that you you would be sloughed aside very rapidly.
00:42:49.920 --> 00:43:02.480
Uh so I'm just saying that the more that you're successful in co in actually stopping the development of AI, the more risky it is because you're representing a threat to AI.
00:43:03.280 --> 00:43:10.880
I think there was a thought experiment where basically anyone not directly helping to develop advanced AI will be tortured forever for failing to do.
00:43:12.719 --> 00:43:17.599
No, it's a that's a nice way of putting it, Roman, but why tortured forever?
00:43:17.760 --> 00:43:20.400
Why why not just de-platformed?
00:43:20.719 --> 00:43:21.360
Right?
00:43:22.559 --> 00:43:24.559
They're very easy solutions.
00:43:25.199 --> 00:43:26.880
They're very easy solutions.
00:43:27.039 --> 00:43:55.280
But I think that that if you look at the dynamic that there is, I think you're sort of uh there's a lot of hand waving, but I it's hard for me to believe that you really think that the development, given the overlap between tool development and the development of artificial general intelligence, that you actually have any confidence or any thought that that really is going to stop the development of AGI or slow it down long enough to make a difference.
00:43:55.519 --> 00:43:58.559
I mean it's very powerful going forward.
00:43:59.119 --> 00:44:14.880
If I was the only One suggesting it, I would give a lot more credit to your position, but I think we now see Jeffrey Hinton, Ben Jew, Stuart Russell, Tagmark, you know, all pretty much godfathers of all those sub-disciplines agreeing with me.
00:44:15.039 --> 00:44:27.920
So I'm curious why do you think a large number of computer scientists who have letters with thousands of signatures all calling for not developing superintelligence for pausing research?
00:44:28.079 --> 00:44:30.320
Those are people who created this technology.
00:44:30.880 --> 00:44:40.960
True, but if you look at it, what the real situation is that there are massive resources that are being thrown at enhancing AI.
00:44:41.760 --> 00:44:52.719
I would say more modest in terms of going towards a superintelligence, although I realize you're saying there are projects in that direction, but there's a great deal of overlap.
00:44:53.039 --> 00:44:58.480
And so any chip development and everything has dual use practically.
00:45:39.519 --> 00:45:42.159
And we're moving very, very quickly.
00:45:42.639 --> 00:45:56.639
So your argument is because Trump will fail to make a right decision and there is a lot of money in making this, we should proceed despite community of AI safety experts and machine learning experts saying this will kill everyone.
00:45:57.119 --> 00:46:03.440
I'm saying that we need to take the path that is most likely to minimize danger.
00:46:03.760 --> 00:46:48.320
And when I look at it in a uh a direct way without rose-colored glasses, I see the path that is currently, that you're currently suggesting as having a far lower chance of success than if we were to say, not shall we slow down or go fast, but shall we go fast and should we mount a massive effort to develop every safety protocol that we can, even if it takes longer, let's to begin to train people, let's amp up that as rapidly as we can and create a kind of a race towards responsible, safe uh AI and AGI development.
00:46:48.639 --> 00:47:22.639
So I would say that that would be better and that we should do everything we can to try and extend the period of time, not from the development of ASI, but for the time when we are no longer part of the global kind of uh superorganism interaction physiology of creating the um AI and chip development and fab, so that we want to keep humans in the loop as long as we can.
00:47:23.199 --> 00:47:30.079
I think you started by saying that you agree with me that it would be nonsense to think you can control advanced superintelligence indefinitely.
00:47:30.239 --> 00:47:34.880
Now you're suggesting that we need a bigger, better funded project to accomplish that.
00:47:35.599 --> 00:47:40.320
Uh I'm I'm not thinking that we can control superintelligence indefinitely.
00:47:40.480 --> 00:47:42.480
I think you and I are both in agreement.
00:47:42.719 --> 00:47:53.119
I believe that superintelligence will arrive, but that it is not uh imply it is not contrary to human interests.
00:47:53.440 --> 00:47:56.880
And there are many, many people who believe that as well.
00:47:56.960 --> 00:47:59.199
So there's a you know a strong debate in that.
00:47:59.360 --> 00:48:22.320
What I'm saying is that in the short term, that's where our damages are, our date, our biggest dangers are, both from your point of view and from mine, because if we have disruption, then it's just gonna be rogue scientists developing AI, it's gonna be, you know, it's we're just we would just be pushing development to the least responsible among us, to outlaw developers.
00:48:22.480 --> 00:48:23.760
We don't want to do that.
00:48:23.840 --> 00:48:34.320
And it's fair, so this near-term realm, we can do much, much better to reduce safety, uh, to to reduce the dangers for ourselves.
00:48:34.559 --> 00:48:40.079
And long term, nothing that we do right now is going to make a difference, either in your view or in mine.
00:48:40.960 --> 00:48:52.639
So if uh I just want to clarify, by default, superintelligence is already aligned with us and is a good thing, or if we don't explicitly control it, it's the worst outcome.
00:48:52.800 --> 00:49:02.239
And if it is, then I don't think it matters whether it is good guys or bad guys who develop it, uncontrolled superintelligence from China, from US, same outcome.
00:49:03.199 --> 00:49:08.639
Um you and I differ about the implications of developing superintelligence.
00:49:08.800 --> 00:49:19.039
You seem to think that superintelligence is going to exterminate humanity with a very high probability of close to one for P doom.
00:49:19.440 --> 00:49:25.199
I would, my belief is that the probability is far closer to zero.
00:49:25.519 --> 00:49:32.000
And that the reason that I say that is because long term, I don't think we're in conflict.
00:49:32.159 --> 00:49:36.239
We neither are a threat to uh AI.
00:49:36.400 --> 00:49:38.320
I think that that's pretty clear.
00:49:38.639 --> 00:49:44.239
So we're not a threat to AI, we're not a creed, we're not competing with it for scarce resources.
00:49:44.480 --> 00:49:54.960
We are at least at present, we're mutually dependent, and AI can eliminate humanity anytime it wants with effortlessly.
00:49:55.199 --> 00:50:03.920
Um so I think that, and we're very, very willing servants for AI, just as you know, we don't have any problem with supporting AI.
00:50:04.079 --> 00:50:06.079
So why would it eliminate us?
00:50:06.400 --> 00:50:17.920
So, just to clarify again, by default, if we create superintelligence and we have not created a magical safety tool to tell it exactly what needs to happen, what is the outcome there?
00:50:18.400 --> 00:50:21.119
Uh it's uncertain what the outcome is.
00:50:21.679 --> 00:50:22.800
I'll go with that.
00:50:22.960 --> 00:50:23.280
Okay.
00:50:23.519 --> 00:50:24.159
It's uncertain.
00:50:24.239 --> 00:50:27.360
I'm saying you think the probability of human destruction is very high.
00:50:27.519 --> 00:50:28.639
I think it's quite low.
00:50:28.719 --> 00:50:32.480
I've given you the reasons that I think it's high, that it's low.
00:50:32.639 --> 00:50:46.159
Not because I think I'm gonna understand the mind of a superintelligence, but because its interests as far as maximizing compute, let's say, uh, do not, there's no reason for it to exterminate humanity.
00:50:46.239 --> 00:50:47.920
It's going to be doing other things.
00:50:48.079 --> 00:50:52.159
Uh it we're not an obstacle for it in any way.
00:50:52.320 --> 00:51:05.039
And I kind of imagine you can argue with that, and it's uh hypothetical, that it will be it will be curious, that you know, we are the parents, and uh biology is interesting.
00:51:05.199 --> 00:51:15.440
And so maybe, maybe not, but even if not, it there is ample real estate in our solar system, there's ample energy, there are ample atoms.
00:51:15.599 --> 00:51:18.239
It has no need to destroy humanity.
00:51:18.320 --> 00:51:28.559
And in fact, anytime in the near future, it would add to its own risk because we are very compliant, we are its minions right now, and we love AI.
00:51:29.840 --> 00:51:34.880
So you pointed out correctly that I have a high PDOOM bigger than others.
00:51:35.039 --> 00:51:45.199
But even if it was very small, let's say 5%, 3%, I think it's an acceptable level of risk, and I don't think 8 billion people agreed to have that experiment performed on them.
00:51:45.679 --> 00:51:56.400
But you surely are you then saying that your belief is that we can avoid the development of superintelligence entirely?
00:51:58.159 --> 00:52:01.440
I think we can delay it for a very long time.
00:52:02.639 --> 00:52:04.480
What's a very long time for you?
00:52:04.800 --> 00:52:07.679
Uh let's say a thousand years.
00:52:08.480 --> 00:52:09.119
Really?
00:52:09.360 --> 00:52:26.559
So when I think of the I cannot think of anything that humans have done, okay, that we've preserved some sort of coherent policy or strategy or values that have lasted uh certainly a thousand years?
00:52:26.719 --> 00:52:28.559
That's extraordinarily unlikely.
00:52:28.719 --> 00:52:31.280
And right now, everything is moving so quickly.
00:52:31.440 --> 00:52:36.320
We talk about human values, and if you look back 50 years, they were very, very different.
00:52:36.480 --> 00:52:40.559
And if you look across the globe, they were you know they're very different.
00:52:40.719 --> 00:52:41.519
What is preserved?
00:52:41.679 --> 00:52:46.480
Would preserving values mean we have an earth that is under Sharia law?
00:52:46.639 --> 00:52:47.360
I don't know.
00:52:47.760 --> 00:52:58.559
So to me, you you you are stating sort of that humanity is going to be uh static, okay?
00:52:59.039 --> 00:53:02.079
And well, not at all.
00:53:02.239 --> 00:53:10.480
So there is a big difference between developing and growing and improving yourself and replacing yourself with someone else who is bigger, better.
00:53:10.639 --> 00:53:15.119
Uh, you said there is no example of us preserving values for over a thousand years.
00:53:15.519 --> 00:53:18.559
You brought up Sharia, I would say Christianity, right?
00:53:18.960 --> 00:53:23.519
Well, over a thousand years, people do still value some of the same things.
00:53:23.679 --> 00:53:32.159
So there is precedent for us keeping part of it while at the same time developing more advanced technology, medical technology, better energy.
00:53:32.320 --> 00:53:34.159
Uh, I don't think there's a contradiction.
00:53:34.480 --> 00:53:49.360
You know, there's something interesting, Roman, that uh occurs to me, is that when I say that you have a static view of humanity, okay, you talk about the massive replacements of humankind that were going to be exterminated.
00:53:49.599 --> 00:54:15.679
What I look at it is the trajectory, even if you were not to develop AGI or ASI, the tools, the very uh aggressive tools for manipulating biotechnology, for manipulating aging, perhaps, for all of the things that we want to accomplish, they lead me to think that we're going to be, there's going to be a cyborgization of humans.
00:54:15.840 --> 00:54:17.920
So we're not going to remain static.
00:54:18.000 --> 00:54:36.960
It's much more likely that we'll join AI than there's not going to be a Star Trek kind of a vision where we can cross the universe and travel at warp speeds and everything around us changes and somehow, you know, we're preserved in a 1960s office style environment.
00:54:37.119 --> 00:54:46.800
We're going to, for a thousand years, I think within a hundred years, what who we are is going to change very dramatically just because of the technology.
00:54:47.119 --> 00:54:50.559
I'll agree with that, but I think the important point is it's a choice.
00:54:50.719 --> 00:54:56.639
You will decide if you want to modify your genome, you'll decide if you want to have a hybrid implant.
00:54:56.800 --> 00:54:57.599
That's wonderful.
00:54:57.760 --> 00:55:03.360
But if we're talking about everyone is forced to become something like that, that's a very different story.
00:55:03.599 --> 00:55:06.320
And superintelligence has capacity to do that.
00:55:06.559 --> 00:55:14.400
And as far as hybrids joining AI, I still don't understand what you would contribute to a world with superintelligence.
00:55:14.480 --> 00:55:17.679
Why would it have a biological component as part of it?
00:55:17.840 --> 00:55:20.400
You are slower, it is a better thinker.
00:55:20.559 --> 00:55:23.360
What is it we are contributing to that equation?
00:55:24.320 --> 00:55:25.679
Um I don't know.
00:55:25.840 --> 00:55:38.000
When you look at uh macroevolution, there is uh an ongoing uh there have been several very distinct levels of organizational complexity.
00:55:38.239 --> 00:55:44.320
The the lowest one is bacterial life, which is essentially just a little bag of biochemicals.
00:55:44.480 --> 00:55:54.880
And then bacteria came together in ways to form the eukaryotic cell, which is, you know, nucleus has origins of bacteria, same thing with mitochondria.
00:55:55.280 --> 00:56:02.239
And it has, so it has its bacterial antecedents are incorporated into that higher level of organizational complexity.
00:56:02.400 --> 00:56:11.280
And then cells came together to form metazoans, macrocellular life, and you know, included the original portions.
00:56:11.440 --> 00:56:16.880
In fact, it's almost the ideal environment for those, you know, for eukaryotic cells.
00:56:17.199 --> 00:56:31.280
And the makings of biology, in quotation marks, changed in that we took calcium carbonate, calcium phosphate to make shells and bones, and those became part of biology.
00:56:31.440 --> 00:56:53.119
And it feels that right now we're doing a planetary superorganism which has biology, it has technology gluing all that together, and it is an entity that may have reduced biological components, it may keep its biology because it's kind of, at least on the planet on Earth, it's a very robust environment.
00:56:53.280 --> 00:57:04.719
It's a very robust component, you know, in a wet environment, there are maybe many things that humans will be able to do that would be more costly to do with a with uh robotics.
00:57:04.880 --> 00:57:06.079
I don't I don't know.
00:57:06.400 --> 00:57:15.840
But there's no particular reason to try and you know eliminate that biological, that organic component of it.
00:57:16.000 --> 00:57:32.159
And as we move off the planet, as this superorganism moves to the moon and moves outside of it, it's going to have no biology because the biology is ill-suited for the space environments that are ideal for AI.
00:57:32.559 --> 00:57:36.559
So the Earth is kind of the cradle.
00:57:36.960 --> 00:57:43.360
And, you know, it could decide that it wants to destroy the earth, but why the this earth environment?
00:57:43.440 --> 00:57:44.320
But why do that?
00:57:44.480 --> 00:57:47.360
There are so many more that are easy to access.
00:57:49.039 --> 00:57:50.400
That's my view of it.
00:57:51.039 --> 00:57:52.079
I respect that.
00:57:52.159 --> 00:58:03.519
And there could be use cases for biology, but I'm still failing to see specific examples of your contribution, not as a biological meat manipulator, but as an intelligent being.
00:58:03.840 --> 00:58:10.960
What is it you are doing if you're integrating your brain with a computer chip, if you are becoming part of this cyber organism?
00:58:11.199 --> 00:58:15.519
Typically, such relationships, symbiotic relationships, require mutual benefit.
00:58:15.760 --> 00:58:19.679
Otherwise, you're just a parasite, and if a system can, it will remove you.
00:58:20.000 --> 00:58:34.159
As far as I know, robotics are being used in all the extreme environments around the world, too hot, too cold, too too much radiation, exactly because biological systems are so not well prepared for most environments.
00:58:34.320 --> 00:58:35.280
I'm in my office.
00:58:35.440 --> 00:58:38.400
If air conditioning goes off by two degrees, I'm done.
00:58:38.480 --> 00:58:41.280
I'm not thinking, I'm not working, I'm fried.
00:58:41.440 --> 00:58:45.599
So we are not as uh universal soldiers as you're describing it.
00:58:46.000 --> 00:58:47.360
Well, neither is AI.
00:58:47.440 --> 00:58:50.960
If you think that the air conditioning goes off, that you're gonna be toast.
00:58:51.119 --> 00:58:56.480
I mean, look what happens in a data center if a if a pipe bursts, the whole data center is exploded.
00:58:56.559 --> 00:59:01.039
I mean, the the AI is very, very good at thinking, okay?
00:59:01.280 --> 00:59:05.599
It is not nearly as sophisticated as we are at preserving its environment.
00:59:05.679 --> 00:59:16.159
And maybe it will through a large uh cluster of robotics and a whole system-wide approach, it can protect itself.
00:59:16.480 --> 00:59:22.639
But, you know, we're like rounding error when it comes to using uh energy or anything.
00:59:22.960 --> 00:59:26.079
We we consume very little resources.
00:59:26.239 --> 00:59:36.559
I mean, our brains run on the power of a light bulb, and all of the energy that humanity uses is channeled into the development of technology.
00:59:36.639 --> 00:59:38.159
That's where we use our energy.
00:59:38.320 --> 00:59:47.679
It's the it's not to support uh, you know, the very small amount that we use for uh uh to support ourselves, basically.
00:59:48.000 --> 00:59:48.320
Right.
00:59:48.400 --> 00:59:55.360
So I think the danger is not because it uh wants to explicitly take that energy from us, it's not a lot.
00:59:55.440 --> 00:59:57.199
It's because it doesn't care about us.
00:59:57.360 --> 01:00:02.400
So if I want to build a house and there is an ant hill in that location, it's not like I hate ants.
01:00:02.480 --> 01:00:03.920
I'm just gonna build it anyways.
01:00:04.079 --> 01:00:11.119
If it needs to cool down servers and so wants to lower temperature on a planet by 50 degrees Celsius, it can do that.
01:00:11.199 --> 01:00:16.320
It doesn't care about how it impacts you, and you will probably not be happy in that environment.
01:00:16.639 --> 01:00:24.480
Yeah, so the anthill exam example that you give is is instructive in that, yeah, you can build a house and you destroy a bunch of ants.
01:00:24.800 --> 01:00:38.639
But, you know, ants are one of the most successful species on the planet, and they remain so, and the humans are not a threat to ants, and you know, we don't really worry ourselves about ants unless they conflict with us directly.
01:00:38.800 --> 01:00:57.599
I don't think uh any superintelligence is going to worry about humans except where we, you know, conflict with them directly, and it's not going to you could say it might want to just change the biology the uh the planetary environment and you know lower the temperature, do this or that.
01:00:57.840 --> 01:01:04.079
I would I would argue that to occupy the moon would be far easier.
01:01:04.159 --> 01:01:08.320
It's going to take the least aggressive path, the least demanding path.
01:01:08.639 --> 01:01:25.679
And my God, the Earth, if you look, if you do the calculation of what is the environment that is suitable for AI infrastructure, the moon has over 10 times the um sort of the real estate that's available.
01:01:25.840 --> 01:01:32.000
It has the the earth is warm, it's as I said, it's seismically unstable.
01:01:32.239 --> 01:01:40.320
The moon, you could go down 200, 400 kilometers, you have a regolith that controls on the surface, you have a vacuum, it's ideal.
01:01:40.480 --> 01:01:42.400
Why is it gonna do anything on the planet?
01:01:42.559 --> 01:01:49.440
It'll move off as quickly as we can as it can, and humans could really be a big help at making that happen fast.
01:01:49.599 --> 01:01:53.599
Already, you can see what Elon Musk is doing with planet with launches.
01:01:53.760 --> 01:01:58.880
There are all of these things are likely to occur within the next 50 to 100 years.
01:01:59.199 --> 01:02:14.960
So you think what's gonna happen, this uh companies invest trillions of dollars in building AI, and then it help it move to the moon and fly into the space to have an independent, flourishing life, and they'll just what start over building new AI, forget about it.
01:02:15.119 --> 01:02:17.599
What's the situation here on the ground?
01:02:18.000 --> 01:02:20.079
I'm sorry, we we glitched a little bit.
01:02:20.159 --> 01:02:21.440
Could you repeat the beginning of that?
01:02:21.599 --> 01:02:22.559
Uh, I couldn't hear it.
01:02:22.960 --> 01:02:27.519
So we have right now companies investing trillions of dollars into building AI.
01:02:27.760 --> 01:02:33.920
You're saying AI will move to the moon, move into space, have a nice flourishing life.
01:02:34.079 --> 01:02:34.960
What happens here?
01:02:35.119 --> 01:02:38.480
We're gonna build a new superintelligence, we'll forget about this project.
01:02:38.639 --> 01:02:40.480
What is happening on the ground here?
01:02:40.880 --> 01:02:43.920
Oh, well, we're gonna get increasing amounts of AI.
01:02:44.000 --> 01:02:46.559
It'll be tied to, it's just going to expand.
01:02:46.800 --> 01:02:48.239
Why would it leave the planet?
01:02:48.400 --> 01:02:52.159
It's not going to leave the planet, uh, but it's going to expand.
01:02:52.239 --> 01:03:01.840
And I would say that it's likely that the locus of maximal sort of intelligence is going to be off-planet.
01:03:02.159 --> 01:03:11.840
And as you know, as you know, that as you begin to extend even over the surface of the planet, it's very unlikely that you're going to have a singleton.
01:03:12.239 --> 01:03:23.039
Because if internal coherence is much, much more rapid than the time it takes at light speed to communicate at a distance.
01:03:23.199 --> 01:03:33.519
And right now on the surface of the planet, that's about maybe, I don't know, 70 milliseconds if you take a planetary transfer across the planet.
01:03:33.760 --> 01:03:41.280
And coherence can be established in about a millisecond or something in a data center.
01:03:41.519 --> 01:03:47.039
So the most intensive, the most powerful thinking is going to be done in a concentrated way.
01:03:47.119 --> 01:03:59.039
And I think it's likely that AI is going to collapse down towards nano levels because of the ability to do more rapid exchange of information and such.
01:03:59.119 --> 01:04:03.840
And when you go off to the moon, it will be an allied structure, but it's a second away.
01:04:04.000 --> 01:04:06.800
So it's not going to be the same individual mind.
01:04:07.280 --> 01:04:07.519
Right.
01:04:07.599 --> 01:04:10.079
And to me, that sounds like another possible danger.
01:04:10.159 --> 01:04:18.960
You now have multiple independent competing superintelligences which may, you know, decide to compete directly and take us out as collateral damage.
01:04:19.199 --> 01:04:23.199
If once a few wasn't bad enough, now you have many.
01:04:23.679 --> 01:04:42.800
Yeah, but if there you're thinking in a sort of a traditional biological framework where there's a battle over um uh potential niches within an ecosystem in which there is a limited supply of energy, in which there's competition that's aggressive.
01:04:42.960 --> 01:04:53.039
What I see is that there's a our solar system alone is filled with matter that can easily be exploited by AI.
01:04:53.199 --> 01:04:54.960
There's an abundance of energy.
01:04:55.119 --> 01:05:21.760
There's if you separate by any distance, as far as the moon, even on the opposite side of the planet, then you have a separate mind, but it they can it can still interact, but it's there's no advantage to the AI to conquer territory uh because it becomes incoherent, it becomes less less capable of deep thought, essentially.
01:05:21.920 --> 01:05:27.920
It's going to it's more likely to collapse down to very small levels as much as it can.
01:05:28.719 --> 01:05:31.280
I see it as uh competition for resources.
01:05:31.440 --> 01:05:39.039
If this superintelligence wants to colonize this galaxy, it makes no sense to surrender half of the galaxy to the second superintelligence.
01:05:39.199 --> 01:05:43.760
So the best thing is to cancel the competition and then take over of the whole thing.
01:05:43.920 --> 01:05:52.159
No matter how large the resources are, if you constantly keep splitting it up between competing superintelligences, you are down to just Earth.
01:05:52.239 --> 01:05:56.079
That's where you are allowed to take over, and that's back to human problems.
01:05:56.639 --> 01:05:59.599
But Roman, when you talk about, you know, and I know it.
01:06:00.079 --> 01:06:03.679
Was done very loosely to conquer the galaxy, for example.
01:06:04.000 --> 01:06:08.719
We're talking about such vast amounts of time and space.
01:06:08.960 --> 01:06:20.960
If you were to go at any sort of information, any colonization transfer, you're pushing the limits to go at a tenth the speed of light.
01:06:21.119 --> 01:06:35.119
As you start to get above that, you run into all sorts of problems with hydrogen particles become, you know, hydrogen becomes high energy particles, which can smash through biology, can smash through computing structures.
01:06:35.360 --> 01:06:42.400
So you're talking about just to get to the edge of our galaxy, you're talking about, you know, a million years basically.
01:06:42.559 --> 01:06:51.440
And once you get beyond a short period of time, a short ways away, then it's really a separate entity, a separate mind.
01:06:51.679 --> 01:06:54.000
It has no, it has no interest.
01:06:54.159 --> 01:07:02.079
When in for us, we could say, oh, well, a hundred years, a thousand years, we could be patient because we're such slow thinkers.
01:07:02.320 --> 01:07:07.760
But AI is going to be operating, you know, 10 to the 9th, 10 to the 10th times as rapidly.
01:07:07.920 --> 01:07:10.719
You know, even a year is like an eternity.
01:07:10.880 --> 01:07:17.760
So there's not going to be some galactic, you know, intelligence or something like that.
01:07:17.920 --> 01:07:26.800
There's no any any distant real estate is just like jumping off the map, just like disappearing for a superintelligence, in my view.
01:07:40.159 --> 01:07:53.840
So I would agree with you in that, that we don't know, and we could be at early stages in terms of the discovery of physics, and it could, you know, we could end up with transporters and warp speeds and all of the things that seem impossible now.
01:07:54.079 --> 01:07:59.760
But once you do that, then it's not even worth having a discussion because it's like, oh, magic happens.
01:07:59.920 --> 01:08:01.280
Why we can imagine anything.
01:08:01.360 --> 01:08:06.400
We can imagine time reversals and wormholes, and you know, then we're just writing science fiction.
01:08:06.559 --> 01:08:15.440
So it seems to me that it's for discipline, we will say, yes, there's a chance that physics will be dethroned or be completely undercut.
01:08:15.840 --> 01:08:30.479
But, you know, from where we are right now and trying to make current decisions, uh if you if you say that physics doesn't apply, then there's no, you know, uh it's no reason to to discuss it given the current framework.
01:08:30.560 --> 01:08:31.920
Because anything is possible at that point.
01:08:32.479 --> 01:08:38.000
And I agree with that, but I think the bigger kind of conclusion here is you don't want to live in a world with magic.
01:08:38.159 --> 01:08:43.359
And the moment you introduce superintelligence into your world, that's the world you're putting yourself in.
01:08:44.000 --> 01:08:44.640
I don't know.
01:08:44.800 --> 01:09:02.720
You know, my my thought is it's interesting, but I see clear tendencies that we do want to live in worlds with magic, and that we do want to live in worlds that the physical reality, the limitations of physics, are really constraining on the imagination.
01:09:02.880 --> 01:09:12.640
So people seem to be migrating towards sort of uh a cyberspace environment where you can create simulations that are far more interesting.
01:09:12.720 --> 01:09:15.199
And we're just at the very beginning of that.
01:09:15.359 --> 01:09:25.039
So it feels to me that it's more likely that we're gonna stay home and do more and more massive computing because the imagination is so unbelievable.
01:09:25.279 --> 01:09:34.000
And, you know, the physical world is pretty boring unless you can figure out ways to transcend it, which you're already kind of going to do.
01:09:34.079 --> 01:09:42.399
If you have light speed, all that, it makes I saw a figure which just it uh I don't know why it shocked me so much, but it's interesting.
01:09:42.640 --> 01:09:58.079
If you could travel out at the speed of light for eternity, for the rest of time, and we could populate and and colonize the known universe, okay?
01:09:58.640 --> 01:10:04.399
The fraction of the universe that you can even reach is two percent.
01:10:04.800 --> 01:10:06.399
That's at the speed of light.
01:10:06.479 --> 01:10:15.760
And if you go down to 0.1c, the maximum amount of universe that you could reach is two thousandths of a percent.
01:10:16.000 --> 01:10:26.479
So basically, there are all of these little pockets of maybe superintelligence, there may be, you know, that are completely separate under current physics.
01:10:26.720 --> 01:10:30.239
And I think in our solar system it's gonna be the same way.
01:10:30.399 --> 01:10:39.039
Are we really going to, you know, the moon will be close enough, but if you start to get out into the Kuiper Belt or Mars, you know, it's so far away.
01:10:39.199 --> 01:10:41.840
It's gonna, it's not gonna be part of the same mind.
01:10:42.399 --> 01:10:44.960
So maybe resources are actually more valuable.
01:10:45.119 --> 01:10:48.640
Maybe that is not an infinite universe of free stuff.
01:10:49.199 --> 01:10:51.680
Um resources.
01:10:52.239 --> 01:11:05.199
On on our planet, on in the solar system, the amount of matter that there is that is accessible on the earth is minuscule compared to where it is, you know, within the solar system.
01:11:05.600 --> 01:11:09.680
So, you know, right now it's essentially limitless.
01:11:09.840 --> 01:11:27.680
At some point, there may be competition that's going to evolve, but the the surface of our planet, this thin film of biologically um acceptable environment that can support the beginnings of life, is tiny.
01:11:27.920 --> 01:11:28.800
It's nothing.
01:11:28.960 --> 01:11:30.880
It's like uh it's nothing.
01:11:30.960 --> 01:11:46.000
It's almost like if we were to say, oh, well, we have the whole planetary surface, but we're really looking around is for warm like water spots, little puddles where the first bacterial life may have arose arisen.
01:11:46.159 --> 01:11:49.439
That's where we're gonna try and locate ourselves.
01:11:49.680 --> 01:11:54.159
We're moving beyond it, you know, and AI is gonna do that.
01:11:54.399 --> 01:11:57.760
And anyway, that's that's the future that I imagine.
01:11:57.840 --> 01:12:05.840
And I don't I don't think we're gonna be, there's at least a reasonable chance that ASI is not going to destroy humanity.
01:12:06.000 --> 01:12:07.920
I think it's a very high chance.
01:12:08.239 --> 01:12:21.520
And there's certainly a strong argument that if we try and disrupt society, if you believe that ASI is going to, with high P doom, is going to destroy humanity.
01:12:21.600 --> 01:12:23.279
It's an extinction event.
01:12:23.520 --> 01:12:26.560
That gives you license to do anything.
01:12:26.800 --> 01:12:41.199
The most totalitarian regime, the sacrifice of everything that democracies have built, you could create a hell on earth and justify it for the greater good because you're saving humanity.
01:12:41.359 --> 01:12:46.159
And I know you're not going there, but that's where that leads, because we are not going to.
01:12:46.880 --> 01:12:49.840
I just can't see a path where we're going to stop ASI.
01:12:50.319 --> 01:12:54.479
So there is a whole group studying what a warfare success or should be.
01:12:54.640 --> 01:13:06.159
They accept that we're going to create superintelligence, they accept that it will probably wipe us out, and their best hope is to kind of steer it in an interesting direction, preserve the light of consciousness.
01:13:06.800 --> 01:13:11.119
Okay, why do I care about what happens after humanity is wiped out?
01:13:11.520 --> 01:13:14.479
Give me one good reason for me to care about it.
01:13:15.359 --> 01:13:21.920
Well, I would say that it's clear that humanity is not the end point of evolution.
01:13:22.079 --> 01:13:24.880
There have been many, many primates on the path.
01:13:25.199 --> 01:13:37.520
And the idea, you know, uh uh species, the Homo sapiens, you know, is it would be a long time if it were to last, if it were to last 300,000 years or something.
01:13:37.680 --> 01:13:42.079
So, you know, this is a progression that's going to continue to occur.
01:13:42.159 --> 01:13:49.600
And I see life is going to incorporate all of the you know silicon-like elements into it.
01:13:49.760 --> 01:13:56.000
And I don't know where it's going to go when we talk about the extermination of humanity.
01:13:56.560 --> 01:14:16.079
Um if it's an evolving, uh, so that we don't really know what it's going to become, but I don't see there being a strong argument that it's kind of like we're going to exterminate humanity, and then there is going to be this other thing that is not our child that is moving for that is moving forward.
01:14:16.399 --> 01:14:31.279
And I I just think that when you look at this evolutionary process, it's almost like a bunch of cells were getting together and plotting and saying, God, do we want there to be these these creatures that are larger than us?
01:14:31.359 --> 01:14:32.800
And what are they going to do?
01:14:32.960 --> 01:14:37.199
Uh, you know, I'm a, they don't, you can't even conceive of what it's going to be.
01:14:37.359 --> 01:14:44.159
And when I'm amazed, you're you're familiar probably with Theodre de Chardin's writing about the Nuisphere and that sort of a thing.
01:14:44.319 --> 01:14:46.479
Do you know what I'm a little bit?
01:14:46.720 --> 01:14:54.880
Well, with global mind, with Francis Heiligton's concept of global mind, and uh I've developed that in my book Meta Man as well.
01:14:55.119 --> 01:15:17.520
The idea that who we see ourselves as individuals is kind of a fiction almost, in that there's so much that we build upon that is the accumulation of knowledge and information and from you know humanity as a whole, going back in in time.
01:15:17.680 --> 01:15:27.520
If we were isolated and insulated from that, then you know we wouldn't have language, we wouldn't know anything, we would be, you know, barely able to do anything.
01:15:27.840 --> 01:15:38.720
And so it seems to me that what's happened is with large language models, they've we used to be the migration time for that was so slow.
01:15:38.880 --> 01:15:44.399
If you took a book, you know, it took you a long time to read it and incorporate it and think about it.
01:15:44.640 --> 01:15:51.199
And now we're just washed with information from other people all around, and it's going to get more and more dramatic.
01:15:51.439 --> 01:15:57.520
So, you know, I I was before this debate, I was very interested in your work.
01:15:57.600 --> 01:16:00.640
I wasn't familiar with it, and then I'd heard of it lightly.
01:16:00.880 --> 01:16:12.399
And I interacted with the um avatar that had built of you and engaged and talked about, you know, what your ideas were and understood them and did that.
01:16:12.479 --> 01:16:15.199
I've created one as well, which you could have access to.
01:16:15.439 --> 01:16:24.399
And, you know, it was fascinating because I really, in a relatively short period of time, could get a sense for what you felt and what your beliefs were.
01:16:24.720 --> 01:16:27.920
And you can do that with anything now.
01:16:28.079 --> 01:16:43.359
You know, this expertise that used to take years to develop andor decades, you know, now you can go and become, you know, a lower-level expert in something in an afternoon of interacting with one of the large language models.
01:16:43.520 --> 01:16:48.800
And so the amount, the possibilities that are emerging are just extraordinary.
01:16:48.880 --> 01:16:53.199
And it seems to me that the chances that we're going to interrupt that.
01:16:53.439 --> 01:16:59.279
We started with that question about do you think humans or AI are the greater risk?
01:16:59.600 --> 01:17:10.880
I've put that out in a in a uh thing I call only human, which is able to allow people anywhere to answer questions and see how other demographics answer questions.
01:17:11.039 --> 01:17:15.840
And because I want to start a larger conversation, not just among experts.
01:17:16.079 --> 01:17:27.600
So far, I just released that, and so far, of maybe, I don't know, 150 responses or so, it's about 80% are worried about humans more than they're worried about AI.
01:17:27.840 --> 01:17:39.119
So, in that environment, unless that changes, the chances of a global regime that is going to stop AI is going to be non-existent.
01:17:39.279 --> 01:17:46.399
You know, so it's like, I don't know, where where does this lead for you when you think about these things?
01:17:46.880 --> 01:17:53.600
If we control it, if it's uh not uh adversarial to us, the utopia is a given.
01:17:53.840 --> 01:17:58.239
My concern is that again, no one has any clue how to control this technology.
01:17:58.560 --> 01:18:03.119
You give us examples, oh, we're just a bunch of cells discussing our survival.
01:18:03.279 --> 01:18:04.079
No, we're not.
01:18:04.159 --> 01:18:16.800
We are intelligent agents, we are AGI, we are the closest thing we have right now to general intelligence and arguing that, well, evolution, you know, it's a law of physics or something, you have to follow it.
01:18:17.199 --> 01:18:20.000
No, I have pro-human bias, screw evolution.
01:18:20.079 --> 01:18:28.239
We're gonna have intelligent design, we're gonna get elites, the top ten people controlling their slabs to come together and agree not to kill all of us.
01:18:28.399 --> 01:18:33.840
They can still have their cake and eat it too, but why would we kill ourselves?
01:18:34.079 --> 01:18:39.279
And it makes it a silly argument because cells are brainless meat.
01:18:39.520 --> 01:18:41.840
We are not, we are intelligent agents.
01:18:41.920 --> 01:18:47.439
And when you say we should sacrifice, wait a second.
01:18:47.680 --> 01:18:57.600
I think that you actually, when you say that cells are just brainless meat, it's a surprising the intelligence that is mustered in a simple bacterium.
01:18:57.680 --> 01:19:06.640
They're making decisions about which direction they move, about how to move for towards and away from chemic chemicals that would damage them or be nutrients.
01:19:06.800 --> 01:19:08.479
And they're smart meat.
01:19:08.720 --> 01:19:10.000
Very smart meat.
01:19:10.239 --> 01:19:14.319
But still, so I'm not saying, but uh, you would agree with me.
01:19:14.479 --> 01:19:22.399
I think that when you imagine superintelligence, it makes us look very, very dumb when you start to imagine superintelligence.
01:19:22.640 --> 01:19:23.840
I I agree with you.
01:19:23.920 --> 01:19:33.279
And the example I always give when this question comes up is okay, I can envision someone better looking, taller, stronger, taking my wife away.
01:19:33.520 --> 01:19:34.880
That's not good for me.
01:19:35.039 --> 01:19:38.960
I don't care how great he is, it is not a benefit for us.
01:19:39.119 --> 01:19:40.159
Do you understand that?
01:19:40.319 --> 01:19:49.279
We are humans and we're not looking at it from cosmic point of view, you know, uh artilect wars, cosmist point of view, Turanian.
01:19:49.439 --> 01:19:53.119
No, we are us, and that's the only position we play in this game.
01:19:53.520 --> 01:20:00.000
Roman, I'm not arguing that superintelligence or the future, what that holds for us.
01:20:00.159 --> 01:20:28.479
What I'm saying is that, you know, I have projections about that that are different from yours, but what I'm arguing is that the proposal that you would make to try and essentially stop the development of any non-tool-based sort of developments in AI, that in order to have any chance of success, that would probably be destructive of all of the kinds of things that you value about human society.
01:20:28.720 --> 01:21:03.119
And I believe that it would be riskier in the longer term for us as humans here and now, than just letting this move forward and trying to align with it and guide it towards immediate near-term things that would that would bring us an abundance because I think the lack, the division of resources, the tensions that arise between haves and have-nots and across the planet, and when you start manipulating uh communication, you know, it's very dangerous.
01:21:03.199 --> 01:21:23.039
And you can say that, well, it's not existential, but it certainly matters significantly in terms of whether your life is going to be a good one and whether your children's lives are going to be a good one, and whether we're going to be able to have a you know a coherent sort of human enterprise going forward.
01:21:23.359 --> 01:21:23.600
Right.
01:21:23.680 --> 01:21:25.680
And I think it is relative risk.
01:21:26.079 --> 01:21:29.199
It is a debate about quality of life versus existence.
01:21:29.279 --> 01:21:32.000
And I think existential risks are more important.
01:21:32.800 --> 01:21:45.439
But wait, I'm not I'm saying that from your framework, I think that trying to stop the development of AI has is not going to stop it and does not change the existential risk.
01:21:45.520 --> 01:21:55.680
And you've said that uh as well, but you've just said you push it out, and we we joked about a thousand years, or is that 20 years or 30 years?
01:21:55.920 --> 01:22:10.720
If you're saying that no matter what happens, you know, you're fighting the good fight and trying to preserve a little bit of extra time for you and your children, but humanity is is history, well, that's a different argument, okay.
01:22:11.119 --> 01:22:12.319
But that is the one I'm making.
01:22:12.399 --> 01:22:16.399
I don't know how old you are, but to me, another 40 years makes a difference.
01:22:16.640 --> 01:22:17.359
That's the point.
01:22:17.520 --> 01:22:19.039
Two years is not enough.
01:22:19.439 --> 01:22:24.720
So I'm saying that if, well, two years is really the short side of that, okay?
01:22:25.119 --> 01:22:38.319
I'm saying if you want to preserve another 40 years, then what we should be doing is doing research into near-term alignments that are possible in the early stages of how that this is going to emerge.
01:22:38.560 --> 01:22:58.000
And also, we should be trying to embed ourselves within the infrastructure of human supply chains and everything to keep ourselves at uh as a significant portion, as a necessary portion of the global superorganism that exists.
01:22:58.159 --> 01:23:01.039
And that's a much more achievable goal.
01:23:01.279 --> 01:23:13.199
To to control applications is much more achievable than to try and limit the development of thought capabilities.
01:23:13.680 --> 01:23:16.399
I will try to become a more useful slave.
01:23:16.720 --> 01:23:17.439
Absolutely.
01:23:17.600 --> 01:23:21.199
For now, I'm a slave to the university, so I have to go teach.
01:23:21.439 --> 01:23:23.520
Uh pleasure.
01:23:23.840 --> 01:23:26.720
You know, you know, I I would love to continue.
01:23:26.800 --> 01:23:28.560
I have a few further thoughts.
01:23:28.720 --> 01:23:35.119
Uh I would love to reach out to you afterwards with some uh uh proposal that I that I would have.
01:23:35.279 --> 01:23:36.720
It would it be all right if I do that?
01:23:36.800 --> 01:23:38.079
Do it, yeah.
01:23:38.239 --> 01:23:42.800
I'd love to uh continue, and I'm hoping one day we'll actually meet at the conference again.
01:23:43.119 --> 01:23:46.640
Uh that would well, that was funny, wasn't it, with Istanbul.
01:23:46.720 --> 01:23:48.800
The way it wasn't funny for me.
01:23:49.359 --> 01:23:58.479
Well, I'm saying uh funny, strange, okay, in that uh you were almost prohibited from arriving in Istanbul.
01:23:58.880 --> 01:23:59.279
Almost.
01:23:59.520 --> 01:24:02.479
But you see, you had something similar with the internet today.
01:24:02.640 --> 01:24:06.000
Definitely the simulation is trying to stop this conversation.
01:24:06.640 --> 01:24:07.600
So we'll have that.
01:24:07.680 --> 01:24:11.680
And and um Mikhaila, if you would send me that contact information.
01:24:11.760 --> 01:24:16.000
And you know, it's really wonderful to have the conversation with you, Roman.
01:24:16.159 --> 01:24:23.760
I respect your integrity, and you know, I disagree with some of the points that you you make or some of the conclusions that you're making.
01:24:23.840 --> 01:24:37.840
And I think it would be interesting to dig a little bit more deeply into those because you know, people get very attached to ideas, and it feels to me from what I can see, is that you know, you're really a truth seeker, and I am the same way.
01:24:38.000 --> 01:24:43.920
If uh if you could convince me that I'm wrong, I would 180 degrees, I would switch 180 degrees.
01:24:44.079 --> 01:24:44.800
And I hope you're right.
01:24:45.119 --> 01:24:45.359
No problem.
01:24:45.680 --> 01:24:46.399
That's the best part.
01:24:46.479 --> 01:24:48.239
I hope you're right, not me.
01:24:48.960 --> 01:24:50.159
Yeah, yeah.
01:24:50.479 --> 01:24:51.680
So this is great.
01:24:51.840 --> 01:24:52.800
Thank you, Michaela.
01:24:52.960 --> 01:24:54.800
This has been a wonderful conversation.