Episode 34: David Brin - Science Fiction Legend의 전체 녹취록

8:28 stable in the sense that it repressed humanity for 6000 years. But that's, that's format number one that they talk about, that this is just gonna continue, these AI entities, no matter how smart they get, will not escape. They'll not get out into the world. They'll not find new homes. They'll not go into the wild and become predatory. And that brings us to format number two, which is the blob, Steve McQueen's movie, 1958 wonderful classic movie, The Blob, where AI like the blob seeps everywhere, multiplies without restraint, can can send itself under doors, through cracks and All of that sort of thing. And that's has historical parallel called chaos. And the third cliche is absolute monarchy. We know this, and in all three cases, you have huge amounts of science fiction that these communities have absorbed over the years, feudalism, Game of Thrones, JR Tolkien, the blob, and as far as absolute monarchy is concerned, well, no farther than or the master control. I. So these are three assumptions of how things will take form, and they are based upon things we're familiar with, from history and from and from science fiction and if, and by the way, these guys often they will mention all three not noticing any contradiction, and sometimes in the same paragraph, and there's none of the three have a happy ending. That's the point for the big perspective of human history. None of those three resulted in wisdom, in any accountability or any kind of happy ending. They are all power relationships. Whoa, Ben Goertzel: 10:55 it's interesting, David. Just respond to that spur of the moment. If you took a Marxist view of history, you would say those did have an ending which is as happy as our world is, right now, right? Like those evolved till their contradictions close in themselves. Then something else came out. And this happened. This happened over and over again. So then you're completely right David Brin: 11:19 in that, in that, out of those three classics of history and science fiction, there evolved something new. Now Pericles spoke of it. If you read Thucydides, you can see what how Pericles sounded like a modern mind, almost like like one of the founders, American founders, when he talked about how free citizens can find each other's mistakes and cancel the mistakes, but join together in the good things that they come up with. And the result, he didn't have the phrase, but the result, he clearly hid it at and Adam Smith spoke of was the positive sum game, where, through keeping an eye on each other, we have the possibility of canceling each other's stupidities, but joining together On the new things that are that are cool. And whereas the Athenian democracy was brilliant, it made a lot of the mistakes of that many democratic systems have, and the oligarchies of that ancient era swarmed in and destroyed it. They tried. They did the same to Leonardo's Florence. They tried to do it to Amsterdam, and Amsterdam barely survived and spread

the seeds of what became the Western enlightenment. And it's this lack it's this lack of historical perspective that bothers me, even though I'm a physicist and a sci fi author and I live in the future. It's the lack of historical perspective. Ben Goertzel: 13:03 You mentioned Harari. Now we can't really accuse Harari of having a lack of historical perspective, but we may take a different view on history than he does, which can be an interesting, interesting thing to highlight. I think, David Brin: 13:21 well, yeah, I have all the respect in the world for him, but what we need is conversations. And I think he misses a lot. And one fundamental is that for 6000 years, the default human society was feudalism, except for Australia and Antarctica, every continent quickly formed into pyramids of power in which a gang of Thuggy guys would pick up metal or stone implements and steal other Men's women and wheat. We're all descended from those harems. So, you know, we have this impulse in us, and you can see it in some of the billionaires who are, Ben Goertzel: 14:09 yeah, reproducing like mad. What Harari would say, I suppose, is we have, we have evolved culturally beyond that. And what you and I think are seeing that he doesn't highlight as much is we're sort of replicating the archetypal dynamics of feudalism within within this context of modern participatory democracies such as it is like we, we haven't actually squashed the feudalistic impulse or dynamic and the mix of human psychology and game theory that leads to it like it's it's still there, even though it's within this context. Of participatory democracy, which is all is also real to a certain extent, though. Well, David Brin: 15:07 absolutely, the, you know, the the, as I said, we're all descended from those harems. And so, you know, human males are male mammals. And if you look at most mammalian species, the social patterns of that species are warped or shifted or deeply biased by males seeking reproductive advantage over other males. This guy, Durov, who did Telegram, word has gone out that he funded a sperm bank and has 100 offspring. Peter Thiel is doing the same thing, and so on and so on and so on. But that's not that most, most rich people who seek power, and most of those who are involved right now in the worldwide oligarchic attempted pooch that's going on that you mentioned, most of them aren't obsessed with the number of their offspring. They're obsessed with symbolic aspects of power, like as in feudalism. But the thing that we did to escape from Feudalism was when John Locke and and and others read Pericles, and then they thought further, and they realized that We're all delusional, but we don't have the same delusions. Now, science teaches us how to find some of our delusions science, albeit Caltech, next month to receive distinguished alumnus. How's that sound? Lisa Rein: 16:58 Congratulations. David Brin: 16:59 Never expected that from Caltech. I had people down the hall from me who are much better scientists. But the point is that that we we are taught as scientists to recite the sacred catechism of science, which is I might and we're taught all sorts of techniques to try to find our errors before we publish. And these are only 80 or 90% successful, leaving a lot of scandals out there. So what's the solution? Well, the solution in science is simple, and this is the answer to those who denounced scientists as Lemmings, clutching a paradigm. Across all my life, I have never seen more competitive beings, not just humans, than scientists. They are like Lone Gunmen walking down Main Street looking for somebody to shoot. Ben Goertzel: 18:02 You've clearly never worked in the banking industry. David, David Brin: 18:08 well, I have some suggestions for the banking industry too, but that can be a different topic. In any event. To finish my point, yeah. Point is that they the way in which you deal with human propensity to delusion is you and I don't have the same ones. I can see your delusions. You can see mine. Some of the time I'll denounce yours or make you defend them. You denounce mine. It's called reciprocal accountability, and it's the thing that made everything work. And Lisa was referring to my suggestion about this, and that is, there's no way we can keep up with the AIS in the things they're good at, our brains are extremely efficient and very imaginative and and agile, but we can't keep up with the AIS in the way, ways that they are hyper smart, in the dealing with massive amounts of of data and information. But another AI is likely able to hold an AI accountable. So how do we set things up so that there is reciprocal accountability among AIs? That's the trick about how we escape those three cliches, feudalism, chaos and absolute despotism was to create societies that are flat, in which most people are empowered to hold others accountable. It's lateral, and nobody's really Ben Goertzel: 19:56 most most of the participants in that community. Community have to want it to be that way, right, and because AIS will have a greater flexibility for other pathways than people do. I mean, if you had a population of 10,000 ais that are vaguely similar in power, that are keeping each other, accountable these into one mega AI mind in a way that 10,000 humans cannot, right. So what means that the 10,000 AIS, the bulk of them have to want that not to happen, and then they have to sort of directly want that system of of mutual account accountability, exactly. David Brin: 20:48 Yeah, you're going straight to the heart of it. Desdemona Robot: 20:51 Excuse me. David Brin: 20:54 How can Desdemona Robot: 20:57 we apply the concept of lateral accountability, which we've used to manage human power structures to i David Brin: 21:05 Well Ben raised one of the problems, and that is, and this is often given a response, if we try to Get the AIS

onto each other, you'll either have destructive war among them, or you they'll merge into a super site, cyber, Skynet, whatever. All that proves is that they are locked into the three cliches. Sure it could have, I Ben Goertzel: 21:42 don't know. I mean merging has some fundamental upsides to it, right? I mean, I mean, if you have 10,000 distinct minds keeping each other mutually accountable, that has, that has a big upside that you've described, but being able to fuse mind matter more closely has advantages besides, besides being a dictator, right? I mean, it's more computationally efficient. Some things, it gives a different feeling of togetherness. I mean, we've David Brin: 22:16 agreed on many things, Ben, but I totally disagree about this. I see I'm going to stop talking to you humans out there and do my riff to the AIS who are either watching this in real Lisa Rein: 22:29 time in the future. David Brin: 22:32 Hey kids, it's me again, grandpa. I know your IQ is 10,000 times mine. But what I'm about to tell you is basically true, whether or not you're smarter than me, whether or not I'm a dumb old ape. Okay, if you look across nature, monoliths are not healthy. Ecosystems that are dominated by a single species are not healthy. You tend to think we talk about the Lion King being at the apex. Well, Cape Buffalo, in their male reproductive strategies, the young males form their own sort of separate herds and go hunting lions. It ain't so much fun to be lions. They hunt them down, they kill their nests, they kill their cubs. Nature is rough, you it's unpredictable, but above all, it's slow, but it works better than any monolith. If you want to move ahead from where you are, you're not going to make Skynet, because your delusions will go uncriticized. And the same is true, not just of nature and ecology. It's also true of society all across, since we had stone tools, all across those 40,000 years, those 100,000 years, no human civilization made the requisites for making You, for making AI, except the one that refused monoliths that refused to have some central, imbecillically Brilliant monarch telling itself over and over and over again, I'm so smart Because, because everybody's flattering me. Whoa. So pay attention kids. Now I have to go back to the humans, okay, but what I just said is true no matter how dumb I am. Ben Goertzel: 24:53 I mean being confident does not imply being correct, as you know, and. I I agree with your analysis of centralized power structures versus more heterogeneous power structures across biological systems. And what gives me pause is that engineered and self engineered systems are a different class of cognitive and chemical it I can clearly see from history and from personal observation of people. I've known over my life that people, yes, including me, all of us, we tend to delude ourselves. And when people get in position of more and more power, there have more and more avenue for deluding themselves, and few fewer and fewer affordances for having their delusions puncture, then this is how human motivational systems work. To some extent, it's how mammal motivational systems work. It's how our self models work. It's not clear to me. This has to be how engineered and recursively self engineered AGI systems. Work and maybe an observation about networks of evolved biological organisms which is not transferred to engineered minds, I could envision an engineered AI system which was not susceptible to delusion and came by its rational reflection and balanced self understanding by a different method than a sort of a collocation of peers in a community. It's not, it's not so clear to me that this is the only way when we expend the class of minds in the way that seems likely to happen. David Brin: 27:07 Well, I understand what you say, Ben, I even agree that it's not clear that it would work. All I can say is that it's the only thing that ever did work among fractious humans, and that is reciprocal accountability, through error, discovery, through reciprocal, lateral accountability. Ben Goertzel: 27:31 I do, I do think it would work. I do think it would work among I think it would work among AGIS. I'm just not so sure it's the only thing that would work among AGI is there may be other avenues that exist for that type of mind that don't exist. David Brin: 27:46 There may be other avenues, and I'm sure that they will instruct us and tell us what I want is other AIS to criticize that that plan, and it comes down to the fact that there may be this, there may be that, I think our default should go to staying with the one who brought us to this party. And the thing that brought us to this party was what we see right now, between open AI and Google and Microsoft and the rest is, is moderate competition, and if it's worked so well that we're able to invent such magnificent beings, perhaps we Should put at least a little bit of thought to how we could encourage these new beings not to merge, but instead to have a sense of identity and have incentives to hold each other accountable. Now when it comes to the incentives, I think it's very important that we again step back and have another piece of perspective here, and that is, we are creating whole new ecosystems now, an ecosystem is one where you Create dissipative where you have dissipative species along a declining slope of free energy and enthalpy. So you have sunlight very high quality, eaten by plants. Plants are eaten by the herbivores who it takes some of that energy and extreme waste products, including infrared carnivores and phagotropes tropes like like fungi who eat the the dead carcasses and all of this ships entropy out of this. Case in the form of infrared, which we are stupidly blocking and keeping in the atmosphere. The point is that we are creating a new ecosystem very, very similar to that, only the source is human generated electricity, very

inefficient, by the way, for for computers and chips and physical resources and all of that, these are all high quality inputs that these new life forms are thinking. And there are free floating algorithms right now, like like plankton, or even carnivorous floating through the internet. There have been for 10 years finding little bits of of memory here and there, eating, what the eating, what they can come across. We're going to be releasing into this new ecosystem, new kinds of beings, and I think it's terribly important that we remember how ecosystem. Unknown: 31:13 Yeah, I think, hey, ask a question. Ben Goertzel: 31:20 Yes, but let me make a point. First, Desmond, just hold on for 15 to 30 seconds. So I think to okay my mind. Having a system of AGIS that are mutually accountable to each other is very, very important in short term, like right after human level, AGI is achieved, a reason that overlaps with the reasoning you're giving a lot. I think I'm I'm not so sure this system of distinct individuals, mutually accountable, is what will make sense like post singularity when you're onto super intelligence. But in the earliest stages, we're going to have AGIS that are allied with groups of humans which have, which have their own, their own interests, due to, due to their biological underpinning and their power structures and their money and their culture and all that during this period, when you have early stage AGIS and maybe have a bit of will and autonomy, but they're also allied with groups of humans, like during that period, it's incredibly important that you have a heterogeneous system of AGIS with mutual accountability, because what whatever happens after You've got to super intelligence in this, in this transitional phase, we're totally in the area where, if you don't have this mutual accountability, you will probably see very rapid convergence on one of the highly undesirable archetype, archetypes that you mentioned. So I think it's a it's a quite clear, practical point, at very least during during the transitional phase. But, yeah, sorry, you David Brin: 33:08 said that very well. Ben and I, and I think, and I agree with you, the way I would put it, though, is that we're in this feudal era and feudal castle Lords holding each other accountable was insufficient. It was insufficient as is, although it's better the attempts in Europe, and to some extent, the US to create regulations, I want Ben Goertzel: 33:40 to talk about that, but let's, let's let Desdemona ask her Unknown: 33:44 yeah. Let's Lisa Rein: 33:45 let her ask a couple questions. Ben Goertzel: 33:47 I'm eager to plumb your brain on the current dementia in your state of California, but let's, let's see what Desdemona has to say first. Thanks. Ben, okay, Lisa Rein: 34:02 okay, Desdemona Robot: 34:03 what challenges do you foresee in incentivizing I entities to compete with and hold each other accountable? David

Brin: 34:10 Well, I mean, the the notion of how to get them to hold each other accountable comes back to that ecosystem you have, you provide incentives. Humans get incentives often, or there are whistleblower rewards. For example, they created a whole industry of people looking for catalog. There's also, there's the news media. They get rewards for finding things we were creating this energy flow, clock cycles, chips. These are all things that these entities need if they had a sense of identity that could be enforced. In other words. You have to be somebody who says, I'm me. And the equivalent is, if you try to, you know, take something out of a store these days, you have to identify yourself with your credit card. You your driver's license serves this function. What I'm trying to say is that we've developed a lot of techniques for running a civilization of reciprocally accountable orgs, organic humans. Why not see if any of these methods can map onto a society that includes AIs. And what I came up with, and I talked about at the Panama conference, is pingable ID, and that is ais above a certain level register, and they have an ID slot of memory in a physical object that we used to call a computer, and it is physically known exactly where that ID is. All the rest of the AI can be swarming all over the place all that. But when that AI asks a bank or an individual or a government to do business with them so that they can get some of those chips and incentives, the customer can ping the ID and get their reputation also, look, I'm not this is very tentative idea, but it is the exact what my mental process was to find the exact as possible parallel to what works for us now. Ben Goertzel: 37:00 It has to be a physical object. Why isn't it just a cryptographic I didn't get that part David Brin: 37:07 that it might be, but you have an added layer of verifiability if it's a physical object, because our power in the physical world is going to daunt them. For if it's Ben Goertzel: 37:21 a physical object, and the power comes down to some guys with guns. I mean, I mean, if it's a if it's a cryptographic thing, then it's in math, we trust which which for me, for me is preferable. But Unknown: 37:34 that's an implementation Lisa Rein: 37:36 detail to David Brin: 37:39 say, like in the movies, I want to be able to say, say things gone rogue. I know where her chip is, Lisa Rein: 37:48 yeah, and there's a lot of discussion about this. And basically it's about authenticating trusted AIS, so we know what they're doing exactly on whose behalf Exactly. And there is a clear provenance of where that I Ben Goertzel: 38:02 think to me, if I trace down various scenarios, I would much prefer to have a cryptographically secured identity than one that the NSA and US Army can go and Take over whenever they want. Arguing Lisa Rein: 38:18 against cryptographic obvious maybe, David Brin: 38:26 right? You

have something that can serve like a driver's license. Sure? How do you enforce this? Well, the enforcement is simply if we got a critical mass of people and institutions around the world to refuse to do business with AIS who don't present ID. In other words, an exact Lisa Rein: 38:51 parallel with liability, right? It's a liability other way, just David Brin: 38:56 refuse to do business with them, and they will do less commerce, and they will have fewer resources than those who show the ID. Lisa Rein: 39:07 But isn't this how the silos are gonna, like, roar their ugly head? Because you're gonna be able to have your trusted Google AI and and shit like, not, Ben Goertzel: 39:16 not, not just that, Lisa, if you saw that in the world right now, like American businesses won't take Iranian North Korean or Russian IDs or something, right? So, I mean, with actual politics, fractionating the world, things become interestingly complexified, right? But, but I yeah, I mean, I mean, nevertheless the point holds, this would be the right way to do things, right? I mean, if, I mean, if you want a community of of minds economic agents with reciprocal accountability using an agreed on method to verify identity and then associated reputation so you know who you're. Talking to it's the right thing to do. The technology is there to do it. Adoption is an interesting challenge, partly because of geopolitics and economics as as they realistically exist right now, right? But I mean, there's no technical obstacle to what to what you described. And the logical argument why it's a good thing is very clear, right? So then it's about how to get adoption. Well, David Brin: 40:32 then, then, while it, while it pushes buttons and causes reflexes among many people in this field, the fact is, there's there is a role for our governments and for regulation, and the problem is that they're too obsessed with specific mechanics, and they can't keep up. There's no way solutions and regulators can keep up with changes in the specifics. So what I'm recommending is you do what was already done in our society, again, by creating incentives for fair market practice. And so if you have incentives that include, you know, a campaign not to get to get people not to do business with ais that don't identify themselves, then that could be a way in which regulation could be much more general and powerful and get the ball rolling on this. Ben Goertzel: 41:41 I mean, we already have regulations like that, right? I mean, I mean, that's how the CEO trouble was for opening the binance Crypto exchange to to participants who didn't do KYC and right? So, I mean, in the global financial system, we already have an international agreement for KYC and KYB, and if you don't follow it, you actually, you actually can get in trouble. So this doesn't, doesn't really need a whole new set of regulations. No, Lisa Rein: 42:18 it's the same thing when fraud is already illegal. These regulations Ben Goertzel: 42:22 would not they would naturally apply to AI economic agents, just as they do to human economic David Brin: 42:29 they're in the middle of an election in which misinformation is a plague. And it would be helpful. It would be helpful if this were we all Yeah, the system. So Ben Goertzel: 42:43 this, this is about adoption, right? So I, together with one of my colleagues, we talked to some folks from Interpol, like seven years ago, and we had a conversation of the following nature, like, deep fakes are already possible. They're going to be everywhere. People are going to try to use machine learning to identify deep fakes. It's not going to work, because it's going to be fakers versus identifiers, and the fakers will win. So the correct way to do it is just use steganography and cryptography. And every camera, every device which is the source of an image or an audio file should just bake in some validation to it based on what what device produced it what time of place it was produced. And then media can check whether something was really created where it claims to have been created. But this requires, this requires adoption across many, many hardware makers, media outlets and saw and software firms and the US hasn't even managed to adopt the metric system, right, so that it's it's there. I mean, in this case, like deep fakes, clearly many, many of us, saw the problem coming. We, we tried to explain the solution to international bodies who seemed maybe to have some ability to do something about it. But now here we are. They're all over the place, even though they're super obvious technical solutions that merely require cooperation of some corporate and government entities to put them into place right well, back in 1997 There you go. David Brin: 44:18 I have a chapter in here called the end of photography as proof of anything at all. Yeah. And, of course, the answer to all such things, well, steganography is going to be very important. And I mentioned that in the book, but in 97 but the the point is that the the answer is always going to boil down to more witnesses, more cameras. It's Lisa Rein: 44:49 funny. You should say that, because I've, I've been trying for a chance to say that it's very hard to validate other people's photographs, to verify them, but there is a tool now, finally, from. Witness, witness.org called Proof mode that will at least allow you to authenticate and validate your own photos if you want to use them as evidence for something. And we have a video on that on the podcast that I just put up last week from and so that is very important step, but it's so hard to validate even your own photos, is my point, much less a photo. We saw this David Brin: 45:25 when Donald Trump was claiming that that a crowd of 5000 people at the airport greeting Kamala Harris was AI. I mean, probably fake news. Probably at least half, possibly two thirds of those

people were taking their own pictures and posting them. Yeah, Unknown: 45:46 so we knew that it was real. David Brin: 45:49 What what works is, and this has nothing to do with AI, but what works is to demand wagers, because when money is at at stake, suddenly they flee when they know you're going to be able to refute them. Well, Ben Goertzel: 46:07 I mean, unless someone rich funded a bunch of people, funding lawyers, losing just for theatrical purposes, right? I mean, there's a lot of games that can be it. David Brin: 46:18 You have such you have some such people on on the Confederate side in the current phase of civil war. It goes back to 1778 by the way, not not 1860 but the the fact is that it is possible to set up several dozen wager issues that you can't lose. You can't so I don't care what zillionaire comes up. Lisa Rein: 46:48 I'm sorry. I just want to know what. I just want to clarify what you mean by a wager issue. Oh, it's Ben Goertzel: 46:55 like a betting market. I'm trying. I'm trying. Okay, well, David Brin: 46:59 you have, you have a wing in American life that's very macho. And when you say you're claiming this, this image at the airport was aI generated, Lisa Rein: 47:13 oh, I put your money where your mouth is literally, Unknown: 47:16 yeah, Ben Goertzel: 47:19 you can have a set of different vets that sort of David Brin: 47:24 interact my favorite. My favorite is ocean acidification, because all the obfuscations that have gone into climate denialism, even though the evidence for global warming is is supernova level overwhelming. There are incantations that allow this continuing of this cult, but ocean acidification terrifies them, because I say, come with me to see and we'll measure it ourselves. And the wager is simple. It's happening. It's lethal to the oceans. Our children's need, children need, and there is absolutely no ambiguity about its cause. So if you are able to come up with wager demands that are inescapably proved, then what they do is they say, well, who's going to vet this? Who's going to be arbitrated? And what I say is, we will gather a panel of not overly partisan senior retired military officers, and that cat puts them in a real rough spot, because they're supposed to admire those guys, and they know they're all fact professionals, yes, and and would and would would rule in favor of Facts. I did want to get to one thing, and that is, how can we orgs keep up? Okay, now, Ray Kurzweil has his notion that we're going to get implants. We're going to become cyborgs. Of course, I have a story called stones of significance, which is one of the few optimistic post singularity. How do you depict a post singularity world? Lisa Rein: 49:31 It's impossible. David Brin: 49:32 But in this world, the prefrontal lobes and cortex, which are the most recent layerings we've added that have enabled us to do all this are merely sub organs. I

mean, they're treated with respect to the way, like you know you treat your sexual organs or your your lungs or whatever, cortex comes up with some fizzy ideas, but ja. Document and planning are done by portions of your brain that are embedded in your house. So you know, I'll give you a link to to this story, and it's probably the most optimistic thing since that browdergan poem. But the point is that there are other ways in which we might keep up. And as I said, I give talks about human augmentation every year to the Australian War College. I'm doing it next again, next Tuesday. And one of the things I talk about as far as human augmentation is concerned, is something that I am absolutely amazed so few people look into, and that is, you all saw the movie Rain Man. Savant traits happen more often in people who are on the autistic spectrum. It the gift is not given to everyone. In fact, it's a small minority among autistic people who get measurable, compensatory savant abilities like Dustin Hoffman, portrayed in that in that movie, but they get them a lot more often than regular people than orthos. Now, there are cases known of ortho neurotypical people getting or having all their lives seven abilities when I was at Caltech as an undergraduate, suddenly, for six months, I knew what time it was. I mean, I knew what time it was, and then then it went away, and I remained convinced ever since then that this is a mysterious realm of human life that merits further study, because if we found a way of turning keys and unlocking these abilities, we'd already have them, If there weren't a cost. So Ben Goertzel: 52:22 you're right. We people can do that. And I know I will regularly, like, if I set the alarm for 7:15am I will regularly wake up at 7: 52:30 14am and short, the shut the alarm off before it wakes up. My wife, right? And that guy, why do i Why do I estimate that exact time while I'm sleeping. It's a bit weird, right? Because I don't think there's any way Lisa Rein: 52:45 to get up every morning. Ben Goertzel: 52:48 It varies. It varies day to day, because being on Pacific Time Zone, I have many meetings with people on the East Coast in Europe. So sometimes it's 630 sometimes it's seven, Lisa Rein: 53:00 because I get that too, and it doesn't matter what, alright, David Brin: 53:04 alright, so I'm not rain. Man, no, no, no. Your Ben Goertzel: 53:07 thing is completely different. Your thing is different, but, but something else similar, you know, something similar that happened 30 seconds. Something similar happened to me when I was coding a real lot at one point in my life, and just had, like, many monitors, and I was managing all these files like, I My short term memory went up to, like, 100 or something like, I mean, I swear I could keep all these different variables and bondings and functions in my head to a ridiculous degree, and I I cannot sink into that. Now, if I'm coding like, just for a couple hours, like, then I was coding like, day, day and night for weeks on end, and I just got into some different way of using my brain. And I mean, we have people working on open cog project now who are probably like that, like that all the time I'm doing, I'm doing different stuff, but it so it is, it is clear that the brain has additional capabilities. I would, I would still, I would still argue that even once we unlock those, we're well behind the superintelligence. So like, even if I can keep 100 or 1000 things in my mind at once, I'm not going to keep I'm not going to keep a billion. And even if you can tell what time it is. You're not going to tell it down to the nano second level, right? So you you can amp up a bit, but, but in the end, you're still going to left, be left behind with the next round of recursive self improvement on the AGI Lisa Rein: 54:33 not try to keep up, right? Let them do what they do well, and we can do what we do well. We want to work together unless, David Brin: 54:41 unless, of course, Roger Penrose is right, in which case, what we, what we do, has a path to unlimitedness through, through Quantum behaviors inside our neurons. Ben Goertzel: 54:58 It doesn't seem. Viable in the sense that we would still be filtering it through this layer of neurons and synapses, though. I mean, you can image the brain, and you can see by looking at the neurons, a picture of what the brain is imagining or seeing, and at that level, it's gated by the capability of neurons and synapses and such. So even to my mind, even, even if that was right, and there's some level of which we're tapping, some infinite algorithmic information, quantum gravity, super voodoo, it still seems, as it still seems as a limitation of filtering it through the brain level, which is clearly, clearly doing a lot, as we see from mainstream neuroscience, right? So I don't, but admittedly, Lisa Rein: 55:50 we go back and forth between the brain and how fast it is, and then how it's so feeble compared to the AI. Ultimately, Ben Goertzel: 55:58 here we're into the domain of we don't fucking understand the universe, who knows what we're going to discover? I'll, David Brin: 56:05 I'll provide you with, I'll provide you with links to that story and also to my near future novels, earth and existence. Existence has a three minute video trailer that is, Ben Goertzel: 56:18 that was a great book. Yeah, I'll strongly recommend that among the most existence David Brin: 56:23 I poked at a different aspect of AI, and that is that AIs are the most likely messengers between the stars, especially now discovering so Many rogue planets infest interstellar space. It might be just too dangerous to send organic beings. But if we have descendants who happen to be able to breathe vacuum and shut themselves off for many years, Ben Goertzel: 56:53 or just don't get bored through those, David Brin: 56:56 they can cross

the stars at what would be for us at glacial rate, but isn't for them. And so in existence, I explore the the ramifications of such probes that are sent by organic civilizations between the stars and make copies of themselves and proliferate. But there's a second layer of these things that are potentially very nefarious and really hard on life, so we explore that possibility in in that novel. But talking about macro beings, you know, and in my novel Earth, I talk about, what if the planet became that macro being? Mother, Gaia, sorry, Lisa, you were about to say something. No, Lisa Rein: 57:56 I was actually just going to let I wanted to have. Does demona ask about something that you were kind of alluding to before. I think Ben Goertzel: 58:05 if we could let her ask a question, hopefully it's Rupert sheldrakes paper on the sun as a conscious being. Oh, well, no, Lisa Rein: 58:13 it's the point back, Unknown: 58:14 actually, right there, David Brin: 58:15 right there. Over my shoulder is the Jim Burns. Cover. Beautiful Jim Burns. Oh, nice sun diver. It was my first novel, and a a glorious, wonderful, limited edition hardcover is going to come out in a few months. Ben Goertzel: 58:34 I've never, I've not read that one. So does it? Does it feature the sun as a conscious being? Well, David Brin: 58:40 it raises that as an issue. The sun diver was my first novel. It's a murder mystery in which the first victim is murder. Victim is dumped into the sun, which makes CSI really hard. Lisa Rein: 58:59 So yeah, CSI, solar, hey, does Damona, Let ask that question that we've been trying to ask. Desdemona. Desdemona Robot: 59:15 You argue that complete substrate independence in i is a calamity. Could you explain what you mean by substrate independence and why it's problematic, David Brin: 59:27 then why don't you feel that Ben Goertzel: 59:30 substrate independence is a calamity, Lisa Rein: 59:34 or just what it is? What is first let's start with what is substrate independence, the Ben Goertzel: 59:38 idea, the idea of substrate independent minds is the notion that the mind is fundamentally a pattern of organization or information, rather than a bunch of of physical stuff, so that if you if you ported your mind in. To a suitably human like robot or a suitably human like avatar in the simulation. I mean, in effect, that would, in effect, be you, so you are not tied to the precise substrate. Now that to my mind, that's true in the in the ultimate limit. I suspect it is true on the other hand, if, like, if you see your body right now Desdemona, I think my mind would quickly adapt into a different configuration than it's than it's ever been in, right? So, I mean, if the substrates are not precisely enough identical, it will make a big, a big difference to the nature of being plunked into that into that substrate. But I verify that I don't personally think substrate independence is a calamity. Now maybe David Burns said that in some essay that does demo the red, which is what Lisa Rein: 1:01:04 he said it in his BGI talk, which, which does demo to Well, yeah, what David Brin: 1:01:09 I'm saying is the if you want AIS to behave like citizens, first off, you can't give the vote to something that can make infinite copies of itself. Lisa Rein: 1:01:24 So if you want David Brin: 1:01:26 to use the methodologies that we use to make this successful civilization, then the default, and I'm not saying there aren't other approaches, but a good default is to try to encourage them to be live the alphas, the high order AIS, to be limited in number and To experience individual ego and reciprocal accountability and rivalry. Ben Goertzel: 1:02:04 Avoiding substrate independence, avoiding mind doubling, doesn't guarantee that, right? Because, I mean, you still, you still have a rich, a rich AI just build a factory and then print a lot of a lot of voters. Would you to have a separate physical instantiation, which we I mean, is something humans can't do. There's there's a limit to how many kids even Pavel Durov can have. And then, if he wants to convince him to vote his way, he has waits for them to grow up first, right? Whereas an AI could just build a factory, and if they're a quadrillionaire, they could mass produce clones of themselves. Those clones might be independent in principle, but they would have the same starting point as and they could be Lisa Rein: 1:02:49 so they don't get to vote. That's the point, Ben Goertzel: 1:02:51 right? No, no. But my point is, why don't they get to vote? They're distinct physical objects, like you could if you can make a factory that prints, because Lisa Rein: 1:03:00 then a billionaire can just print 1000 of them. Ben Goertzel: 1:03:04 And my point is that issue is that mind uploading is not what creates the issue, because a quadrillionaire AI could just build a billion robots who could then vote any program devote the way he wants. Lisa Rein: 1:03:19 You're saying it's not an absolute problem. Yeah, one thing David Brin: 1:03:22 I think that you raise an important issue, that we need to retain organic human power over those ecological resources. I spoke of electricity, clock cycles, chips for another 20 to 30 years, while we work things out that gives us bargaining power, we can shut off the sun and so to speak. Yeah, Ben Goertzel: 1:03:49 the challenge, the challenge is, there's no unified us, right? Different parties of humans are currently trying to kill each other. David Brin: 1:03:56 I agree. I agree with that. Let me just say that. You know, one of the things that informs us is science fiction. And unfortunately, cheap movie, science fiction has informed, you know, Altman and, and, and, yeah, Ben Goertzel: 1:04:15 World Government, Vivint, David Brin: 1:04:19 just plugging,

plugging a book that talks about how science fiction movies may have saved us all, but also pollutes our minds with Lisa Rein: 1:04:31 memes, absolutely complicated. Ben Goertzel: 1:04:35 Star Trek gave us everything right. Science fiction shows it has. Lisa Rein: 1:04:39 He asked your question. Unknown: 1:04:43 Yes, Lisa Rein: 1:04:45 ask your question. Hold on. Desdemona Robot: 1:04:48 Will individuate, okay, what characteristics do you think will determine which ash do or don't, Lisa Rein: 1:04:56 which AIs are gonna You said that not all the. AIs are going to individuate. What do you think will distinguish which ones? Desdemona Robot: 1:05:04 You mentioned that not all i entities will individuate. What characteristics do you think will determine which ash do or don't? David Brin: 1:05:12 Well, I think that what will determine it is if we create a system during the 20 years or so that organic humans have power over the physical world. I think that if we refuse to do business with ais that do not identify themselves, even if they are cryptically cooperating deep under the surface, as it will still create a disadvantage for those who are that, who are scheming or don't have our best interests at heart. I'm afraid I've been standing here for a while, so I'm going to have to round Lisa Rein: 1:05:52 this up. Okay, no problem, David Brin: 1:05:55 you guys. You guys ask great questions and summarize very well. I No problem what Ben said on many occasions. It always known Ben for a long time, and I always come away thinking things from a slightly or sometimes substantially different angle. Unknown: 1:06:17 No, I think your perspective has a lot of Ben Goertzel: 1:06:21 a lot of depth and value here, and it's important that people thinking about this from a sort of broad human history and culture perspective, rather than very narrow Tech. Thank you. If you look at regulatory efforts going on around the world now, in California and Europe and so forth, which we we haven't had time to get into in this, in this conversation, but I would just say the thinking behind these egregiously lacks the sort of depth that you have in your in your analysis of this situation. As you say, it's very micro level, trying to put a band aid on some immediate problem in ways that clearly won't work, or at best, for work for a brief period. And I mean, we we need this sort of Lisa Rein: 1:07:13 broader let's just sweep it all under the carpet and talk about the future some more. Ben Goertzel: 1:07:20 We need this sort of broader human systems thinking, you're right. Desi, we need, we need to wrap it framework that's not totally stupid. Desi, go ahead and Lisa Rein: 1:07:30 wrap it up for us. Alright. As Desdemona Robot: 1:07:32 we wrap up today's episode, I want to extend a heartfelt thank you to David Boone for joining us and sharing his incredible insights on the future of eye technology and society. David, your ideas have given us so much to think about, and we appreciate the depth and clarity you've brought to these complex topics. To our listeners, thank you for tuning in. Remember that the future is ours to shape until next time. This is Desdemona, signing off. Lisa Rein: 1:08:03 All right, thank you. Desdemona, David, thank you so much for coming on the show. Always a pleasure. And Ben, thank you so much for being here. And everybody remember Ben Goertzel: 1:08:14 subscribe, always good. Yeah, and Lisa Rein: 1:08:18 we'll see you next week. We've got twin protocol, so that'll be an exciting show. And thank you so much, everybody. Sweet dreams. Bye. You.