Jeremy Gilbertson (00:00)
would it look like if AGI or AI gets so good, we trust it to manage something like
the world economy. What does that look like?
Anders Sandberg (00:10)
Well, hopefully it looks like a very well functioning world economy, but that can also be a bit of a
why do we have legal systems and states and this? Well, that is the way we outsource our cognition. And we handle it by having these artificial systems made out of people and pieces of paper. But now we're of course also getting software that can take over more and more of that. Why do we have law? Well, we want peace and prosperity and of course power to some people. They are very good at latching onto that. And with AI, we might get even more peace and prosperity, even more efficiently.
and maybe power to some old man or other people who happen to get grasp it in the right way. But that's a separate issue. But the really interesting part is we might really want better coordination because the world is big and somewhat dangerous. We have climate change. We have controlling powerful AI. We have biotechnology getting more and more powerful. We probably need to coordinate about some global problems way more powerfully than we can do right now.
And maybe we could do that with a United Nations plus plus, but that seems a bit unlikely. It might be that we actually want to live more and more to super smart distributed AI systems. But then you end up with a funny situation. We might be swimming in peace and prosperity. That's great. But previously we needed human autonomy. We needed to make decisions to run the legal system, to run the market. Now it runs itself and probably runs itself much better without us messing with it.
It's a little bit like chess playing. Once humans were better than computers that played chess, then gradually computers got better and better and started beating humans. But a team of humans and computers could actually still do better. But after a while, the computer was so good that actually the humans were just slowing things down. So actually you should just listen to the computer. So if we end up in that world, we have this weird situation that we might be like pampered aristocrats. Really good material standard, but
In some important respects, our lives are not our own. And I think that sounds bad to many people. And I think the reason is we want to feel like we're in control because all our ancestors, the ones that we kind of find giving up control to others, they were just exploited by the ruthless people. So we have this tendency that we want to have autonomy. And then the funny thing is once we have autonomy, we're very happy. We are very happy to give it away in various ways, but we want to be the ones that do it.
It might be that you give away your autonomy by getting married. It might be that you join a company and in the exchange for getting a paycheck, you get some limitation of what you can do, but that might be worth it. I do. Yep. Kids are amazing example. People, every person I know that has gotten kids say, yes, it's super stressful and there's a lot of hard work, but it's the most important meaningful thing I ever done.
Mark Fielding (02:50)
Have kids.
Anders Sandberg (03:05)
It's amazing. It's almost like mind control as somebody who haven't got a kid, but it's kind of a happy uncle. It's kind of, whoa, those babies,
really got some psionic power here, but it's very effective and it's kind of good for our species that we have it. And it's good that we can give away autonomy in a reasonable way, but we should also in some sense be able to withdraw it. Although of course, when you're a parent, you realize, okay, I cannot even withdraw that thing. Now I'm in for it because it's what it is.
But this is where a world economy run by AI might be tricky because it might be so effective that yeah, running your own part might be outcompeted and might be so inefficient. It's a bit like living your life without electricity in a little log cabin in the woods. Sounds absolutely lovely and probably totally lovely the first day. After a few weeks of carrying water, a bit less lovely. And after a month, most people are back and writing on Facebook about the experience.
Mark Fielding (04:04)
We can have two economies like we'll have two of me and two of you and two of everybody else. You have two economies, one that's run efficiently and well by the AGI and then another smaller one that's run and managed by humans and they just run in conjunction with each other.
Anders Sandberg (04:18)
Yeah, and in some sense, I think that this might be what we're developing. We have the backbone of our civilization, the big infrastructure, the really important things that we must get right. When the electricity goes, well, nothing else works. So you really need to get that working. But once you're wealth enough, you can spend that excess wealth on all sorts of weird stuff. There is a reason why you have so many consultants as big companies. They can afford it.
And they're probably more involved with status games and politics than actually doing something useful. I think it's a good thing that we have enough wealth and safety that we can try out different lifestyles, even if many of them turn out to be slightly embarrassing once you're middle-aged.
Jeremy Gilbertson (05:00)
There are things as humans, power dynamics and control and me versus you, like if we were to design or not design, but like kind of enable this world economy through AI,
How do we make sure as humans we don't bring that baggage that always wrecks stuff?
Mark Fielding (05:18)
Remove the
human.
Anders Sandberg (05:21)
even that is a problem. When you try to remove the human, well, what have you trained the large language models on? They're all trained on all the human literature. They have read every single thread we've written about betrayal and power dynamics and corporate shenanigans. One of the big debates right now in the AI safety community is when you test your AI models and they try to do sneaky stuff like escape the system if they think it's going to get shut down. Is that because they
"want" in some sense to escape or just because they know that's expected of me because of all the science fiction I ever read says that that's what an AI does in this situation. and maybe it doesn't even matter. So I do think that it's very hard now to take away the imprint of the human world from the AI world. On the other hand I think there's also going to be evolutionary pressures pushing in certain directions. There are various forms of competition that means that certain
software, pieces of software are going to be copied more widely than others. And that is going to act anyway, regardless of what human values are. And of course, my more doomy friends say, yeah, that will inevitably be pushed in a very scary direction that's going to be very different from what we humans are. So I think one important thing we might want to be involved in, we might want to be an essential spice in this mix, to make sure that there is at least some of us in that. But then we want to filter out the really bad stuff.
of cruelty, for example, it's kind of useful to stabilize altruistic punishment. Taking delight in the misfortune of somebody else is a really good way of kind of giving you extra points when you're trying to punish somebody breaking norms. Because otherwise, I would be taking a risk in policing these norms. And actually, it would be easier if I just stayed at home and let the norm breaker get away with it. Except that today we can do this with laws. We can use police.
And suddenly cruelty is just a bad thing that generally makes us nasty. That's an emotion I totally think we should kind of try to get rid of and at the very least make sure our AI programs never experience.
would it look like if AGI or AI gets so good, we trust it to manage something like
the world economy. What does that look like?
Anders Sandberg (00:10)
Well, hopefully it looks like a very well functioning world economy, but that can also be a bit of a
why do we have legal systems and states and this? Well, that is the way we outsource our cognition. And we handle it by having these artificial systems made out of people and pieces of paper. But now we're of course also getting software that can take over more and more of that. Why do we have law? Well, we want peace and prosperity and of course power to some people. They are very good at latching onto that. And with AI, we might get even more peace and prosperity, even more efficiently.
and maybe power to some old man or other people who happen to get grasp it in the right way. But that's a separate issue. But the really interesting part is we might really want better coordination because the world is big and somewhat dangerous. We have climate change. We have controlling powerful AI. We have biotechnology getting more and more powerful. We probably need to coordinate about some global problems way more powerfully than we can do right now.
And maybe we could do that with a United Nations plus plus, but that seems a bit unlikely. It might be that we actually want to live more and more to super smart distributed AI systems. But then you end up with a funny situation. We might be swimming in peace and prosperity. That's great. But previously we needed human autonomy. We needed to make decisions to run the legal system, to run the market. Now it runs itself and probably runs itself much better without us messing with it.
It's a little bit like chess playing. Once humans were better than computers that played chess, then gradually computers got better and better and started beating humans. But a team of humans and computers could actually still do better. But after a while, the computer was so good that actually the humans were just slowing things down. So actually you should just listen to the computer. So if we end up in that world, we have this weird situation that we might be like pampered aristocrats. Really good material standard, but
In some important respects, our lives are not our own. And I think that sounds bad to many people. And I think the reason is we want to feel like we're in control because all our ancestors, the ones that we kind of find giving up control to others, they were just exploited by the ruthless people. So we have this tendency that we want to have autonomy. And then the funny thing is once we have autonomy, we're very happy. We are very happy to give it away in various ways, but we want to be the ones that do it.
It might be that you give away your autonomy by getting married. It might be that you join a company and in the exchange for getting a paycheck, you get some limitation of what you can do, but that might be worth it. I do. Yep. Kids are amazing example. People, every person I know that has gotten kids say, yes, it's super stressful and there's a lot of hard work, but it's the most important meaningful thing I ever done.
Mark Fielding (02:50)
Have kids.
Anders Sandberg (03:05)
It's amazing. It's almost like mind control as somebody who haven't got a kid, but it's kind of a happy uncle. It's kind of, whoa, those babies,
really got some psionic power here, but it's very effective and it's kind of good for our species that we have it. And it's good that we can give away autonomy in a reasonable way, but we should also in some sense be able to withdraw it. Although of course, when you're a parent, you realize, okay, I cannot even withdraw that thing. Now I'm in for it because it's what it is.
But this is where a world economy run by AI might be tricky because it might be so effective that yeah, running your own part might be outcompeted and might be so inefficient. It's a bit like living your life without electricity in a little log cabin in the woods. Sounds absolutely lovely and probably totally lovely the first day. After a few weeks of carrying water, a bit less lovely. And after a month, most people are back and writing on Facebook about the experience.
Mark Fielding (04:04)
We can have two economies like we'll have two of me and two of you and two of everybody else. You have two economies, one that's run efficiently and well by the AGI and then another smaller one that's run and managed by humans and they just run in conjunction with each other.
Anders Sandberg (04:18)
Yeah, and in some sense, I think that this might be what we're developing. We have the backbone of our civilization, the big infrastructure, the really important things that we must get right. When the electricity goes, well, nothing else works. So you really need to get that working. But once you're wealth enough, you can spend that excess wealth on all sorts of weird stuff. There is a reason why you have so many consultants as big companies. They can afford it.
And they're probably more involved with status games and politics than actually doing something useful. I think it's a good thing that we have enough wealth and safety that we can try out different lifestyles, even if many of them turn out to be slightly embarrassing once you're middle-aged.
Jeremy Gilbertson (05:00)
There are things as humans, power dynamics and control and me versus you, like if we were to design or not design, but like kind of enable this world economy through AI,
How do we make sure as humans we don't bring that baggage that always wrecks stuff?
Mark Fielding (05:18)
Remove the
human.
Anders Sandberg (05:21)
even that is a problem. When you try to remove the human, well, what have you trained the large language models on? They're all trained on all the human literature. They have read every single thread we've written about betrayal and power dynamics and corporate shenanigans. One of the big debates right now in the AI safety community is when you test your AI models and they try to do sneaky stuff like escape the system if they think it's going to get shut down. Is that because they
"want" in some sense to escape or just because they know that's expected of me because of all the science fiction I ever read says that that's what an AI does in this situation. and maybe it doesn't even matter. So I do think that it's very hard now to take away the imprint of the human world from the AI world. On the other hand I think there's also going to be evolutionary pressures pushing in certain directions. There are various forms of competition that means that certain
software, pieces of software are going to be copied more widely than others. And that is going to act anyway, regardless of what human values are. And of course, my more doomy friends say, yeah, that will inevitably be pushed in a very scary direction that's going to be very different from what we humans are. So I think one important thing we might want to be involved in, we might want to be an essential spice in this mix, to make sure that there is at least some of us in that. But then we want to filter out the really bad stuff.
of cruelty, for example, it's kind of useful to stabilize altruistic punishment. Taking delight in the misfortune of somebody else is a really good way of kind of giving you extra points when you're trying to punish somebody breaking norms. Because otherwise, I would be taking a risk in policing these norms. And actually, it would be easier if I just stayed at home and let the norm breaker get away with it. Except that today we can do this with laws. We can use police.
And suddenly cruelty is just a bad thing that generally makes us nasty. That's an emotion I totally think we should kind of try to get rid of and at the very least make sure our AI programs never experience.
