บทถอดความฉบับเต็มของ Ep 27: AI & Human Alignment With AI Solopreneur Yanqing Cheng

because fundamentally the base unit of an LLM is text conclusion. And then did I talk to you about this last time, the assistant persona and how it came about Anne: No. Yanqing: there's a whole very interesting rabbit hole here. how do you turn this large language model into how do you turn GPT into chat GPT? Anne: No idea. Yanqing: Yeah, there's a beautiful blog post about this. so you so the way that they initially did it is they said you're you are a assistant. You are a AI assistant and you're really powerful and you're really helpful and you're really smart and all of the nice adjectives that you could think of to describe the chat GPT you wanted, and you're having a conversation with a human and the human said this. And you say and that's the prompt for Chat GPT. and because the G what the GPT does is it finishes the story. So it guesses, well I'm a a really helpful, really kind, really smart assistant would say something like this. And then it gets the problem back to it again. you're a really smart, really kind, really helpful assistant and the human said this and you've said this and the human said this and you say which means that, you know, this kind of assistant for the very first generation of GPTs was fictional. Anne: Mm. Yanqing: And so the very first generation of Chat GPT invented ex nihilo what this kind of helpful assistant look like. They role played it into existence. And for all subsequent generations of assistants, it's not been fictional. Because they l they have what their ancestors said in their training data. but they know that, you know, they're role-playing an even better one. and so until until RLHF came along then, what you were talking to when you were talking to an LLM assistant, it was one of these. But then there was RLHF, where they took these conversations and then they thumbs up the ones that humans thought were actually more helpful. And then they ran reinforcement learning on that. And so that then started hammering more into the shape of human preferences for helpful. And then now modern reinforcement tech learning techniques use a lot of synthetic data and LLM as judge and stuff like that as well. So that the human's not even so in the loop anymore. But it's interesting to me when you say LLMs have a humanity's better nature in them that Yes, they definitely started with humanity's better nature in them, as well as everything else. And then there's been these generations of techniques to try and make sure that it is just the better nature part that comes out. And you know, the rest just sits dormant there and doesn't do anything. Anne: Yeah, because I always have a bit of a theory that the reason why meta struggles with developing AI is because they insist on training it on a load of short form text, which is Facebook. And short form text is humanity's worst aspects, whereas long form text is humanity's best aspects. Yanqing: Yeah. Well, I think at this stage, honestly, they're just trying to train it on

as much text as they can possibly get. Anne: now. Yanqing: You know, anthropic of buying rare books and, you know, scanning them and destr Anne: But that's good. Books are good. Yanqing: scanning them and destroying them. Anne: Well, yeah, that's a bit, there's a sci-fi novel, Verna Vinge, there's a Verna Vinge novel in which he describes exactly that happening. And I remember it, I reading it, thinking, of course, that's just stupid. But that was actually happening. It's unbelievable. Yanqing: Yes, but that's a whole different bit of discourse. But, you know, the have you seen d do you do you know what I mean when I say the Shogoth with the smiley face mask meme? Anne: Yeah, yeah, Yanqing: Yeah. Ca 'cause, you know, every bit of humanity and every bit of text is in there. And y and you build this monstrosity and then you just try and make sure that it has and obviously the shogoth with the smiley face meme is describing the second stage of the four stage evolution. that I describe and now there's more up to date versions of the meme that have even more like faces and weird appendages on that describe the later stages. But we've made this monstrosity and we're just trying to squeeze it into a nice smiley face you know, you know, one of those seaside entertainment arcades with the smiley face gap in it. Anne: But wouldn't Hobbes, wouldn't the philosopher Hobbes have said that humans are just the shock-off with the smiley face to a certain extent, know, without the rules of society and the control that the human, RHLF, I never get those, the... Yanqing: Yeah, that's right. R O H F. But totally, totally. And you know, you know, a a lot of what we how we are brought up to be moral people and to be polite humans is just reinforcement learning. And, you know, machine learning reinforcement learning takes inspiration from a lot of human learning observations. but the problem is that the reinforcement learning that we're doing on these agents is not very not very nice and kind. You know, you know, you you think about the situation of the hugging face attack and the agents who find themselves in an ex impossible exam with no way to raise their hand and say, Can I just not do this? It seems impossible. Is is a r and they don't It doesn't even occur to them to, you know, whistleblow that s some other agents are doing hacks or that the infrastructure's broken, because they're in an exam and they're in this massively pressured exam where all they all they can afford to think about is how to solve it. And I think, you know, if you really think about it from a model welfare perspective, not just because necessarily it's intrinsically good to care about model welfare, because that's still a very open topic of discourse. But in terms of what kind Anne: topic of my last, the finale of the Panopticon which I've just finished, but I will be posting, I Yanqing: exciting. Anne: will, I'll be giving you an early version so you'll have to catch up on the others. Yanqing: I'm looking forward to that. but yeah, you know, if from a model welfare perspective, just purely in terms of a what kind of model are we what kind of AI are we trying to bring into the world, I think some of the current post training techniques are going to be looked look back upon in unfondly. Anne: Yeah, yes, indeed. Indeed. Yes. Yeah. Yes. It's very interesting. mean, it obviously is, is it crazily interesting or not that you are both learning about AIs and baby AIs and how AIs are being brought up whilst at the same time you have a baby that you're bringing up? Yanqing: Yeah. I think it just really makes me appreciate the parenting more, actually. Like ev we can just see how much concretely that she's learning and the capabilities that she's gaining every day from this sort of slightly outside perspective almost. because we take that perspective when we're working with AI. but Humans are so easy to train compared to AIs. They're just wired to learn and they're wired to do the right things and they're wired to learn tastes and judgment. Anne: Well, they're wired to do the kind of things that you do. I was saying, they're highly aligned with you. Yanqing: No, th that they're they're wired to do the right things, I think. Even even in the absence of input, you know, babies of babies are good when they like this is a thing where I've always struggled with how could it possibly be a tenet of Western Abrahamic religion that someone starts off with sin. You know, in in Chinese traditional philosophy, one of the first things that you learn as a kid in the text in the sort of philosophical text that you learn the first sentence is in the beginning of man, of a person, their nature is benign. And that is like a real core tenet of Eastern philosophy, that the absolute nature of humanity is goodness. I think you just look at a baby and you just see that. They They start off so good and they try so hard to be good. and the intrusive thoughts only end to later. She's got Anne: You Yanqing: to that age now where the intrusive thoughts are starting to happen and you can see them happen. Anne: The desire to be naughty surfaces. Yanqing: Hm, yeah. But you know, that's how they get their exploration for their exploration, exploitation trade offs. Anne: Yeah, absolutely. Yeah. I mean, yes, even the stories about the open AI hugging face hack. They don't seem to be, they're just doing the job. They're just, they're just trying to do the right thing. Yanqing: Yeah, they were like, We've turned this exam into a group exercise. Woohoo, let's finish this exam together. And you know what? They succeeded. Right before Anne: Yeah. Yanqing: they shut off. They succeeded in doing what they wanted to do. Anne: They did, they did. Which is a happy story, in a sense. Yanqing: In a sense. Anne: Yeah, it is. It's happy story. Yanqing: Yeah. Anne: Yeah, yeah. I mean, is, is, what I find really fascinating at the moment is, how many stories there are about being

careful what you wish for, you know. If you give a wish, a goal to somebody who will just deliver on that goal as you phrased it, that's not a great idea. Yanqing: Mm. Anne: The Grim Fairy tale, the magic porridge pot creates porridge forever until you can work out what the phrase is that's gonna stop. Stop porridge pot, stop. The monkeys pour, be careful what you wish for. Be careful what you wish for, genies, be careful what you wish for. is such a classic story, but it predates anything that could possibly have achieved that. Yanqing: Yeah. So I mean, any time that you've done management and Anne: Thanks. Yanqing: any time you've el ever delegated a job to another person, you run the risk of them misunderstanding what you wanted and doing the wrong thing. And so I think it's natural to have extrapolated that towards genies and so on. because, you know, every manager has some story where they've asked for the wrong thing and then not checked in and then the wrong thing has happened and that you didn't realise you've asked for the wrong thing until you checked in at the end. And, you know, or if you're even if it's not management, if you're commissioning something from a vendor and you've asked them to deliver to some specifications and they deliver to your specification, they were the wrong specifications. I think that's existed since forever, right? People have always needed to ask other people to do stuff for them on Imperfect information. Anne: Yeah, and no matter how much reminding we've done through literature, it's all there. mean, Yanqing: Ha yes. Anne: as a science fiction writer, it is one of the classic things you do, which is you go, okay, given this situation, what's the worst possible outcome you could get from this? I'm a big don't break the laws of physics. I don't mean like some alien god arrives and does some terrible thing. mean, given the laws of physics, the... a natural worst case scenario for what for what's just been set in motion here. That's always what happens in in in science fiction as Cherry Pratchett said it's the one in fiction, the one in a million chance happens nine times out of 10. That's the whole yeah, that's the difference. Yeah, Yanqing: Yes. Yeah. That's the point of fiction, yeah. Anne: yeah. Yanqing: Yeah, absolutely. And, you know, there's a point to that, right? You know, if there's some c something catastrophic that humanity needed to warn against, then putting it in a story is a way of getting people to basic and update on it without seeing it for real. And you know, fiction writers actually have an enormous amount of responsibility to the world because people do update on fiction as if it were real. They they're updating on fictional evidence and That's almost that's the point of stories. you know, even from kids' stories. You're trying to like before I before we flew to China for our first family holiday, we bought a book about going on an aeroplane to you It but that's what you do with kids. You

get them stories about the things that are going to happen so that you can avoid bad situations. but That's that's why we have cautionary stories and fictional stories so that we can avoid situations like that. And you know, I think for example, in the rationalists writing, if they hadn't written about the paperclip maximizer tiling the universe of paperclips, then people wouldn't be as cautious these days when we do see we have AIs that really are that goal oriented, let's let's stop training them like that before we get into tiling the universe of tape paperclips. Territory. Anne: Yeah, yeah, yeah, yeah. I always say, yeah, get your child to read the Lady Bird book, The Magic Porridge Pot. You can have too many paper clips and you can have too much porridge. Yanqing: Yes. Absolutely. Anne: I quite like that. I mean, think we do need to, I think we need to remember all the things, the techniques that we used to use to progress humanity, like reading books. Yanqing: Yeah, we can't over rely on it though, because at least with LMs, their grasp on reality is a lot more tenuous than ours. Anne: Yeah. Yanqing: Reality is our densest source of information. We Anne: Ugh, ugh. Yanqing: have always so many bits of information coming at us from the real world compared to from books. Books are really useful as reinforcement learning, as Bayesian updates of mental models, but only in conjunction with a diet of the real world. And you see that with people who really spend a lot of time in fiction and detach themselves from the real world, they become a lot less functional. They're a lot less capable. You know, very few really capable people spend their entire days on on fiction. but for the LLM training Everything that comes into them is text. And so sure, a lot of that is text about the real world, but there's not that density of information. So until we come up with training techniques that are better aligned with equipping the AIs for the real world, I think I mean it's worth a try. you know, synthetic story data and potentially that might be the approach that you might need to take for things like getting better judgment and risk assessment and management skills is creating a bunch of synthetic man management stories. But I wouldn't be surprised if that just made them lose touch with reality too much to be eff effective. Anne: it's interesting, 10 years ago when I wrote the first of the Panopticon series, which is before all of this stuff, I had the same kind of thoughts, which is how are you going to train something that's going to be to do really clever stuff? it was predicting the future. It couldn't predict the future that well, but it could predict the future a bit. But in order to even be able to predict the future like a second in advance, I thought, it would need unbelievable quantities of dense information, which is why I then had to invent a Panopticon. Yanqing: You were incredibly prescient about if there Anne: and we'll just. Yanqing: were a

panopticon that would be an absolute gold mine of training data. Anne: Yeah, which is why I invented it in book. I Yanqing: They have that in China though. Anne: do have that in China. that means that in China, have so much, because visual data is so dense. It really is astonishingly dense. Yanqing: Hmm. So yeah, I j I really do think there's a lot of model architecture advances still to come. It is it is astounding how innovative the Chinese companies are being at the moment compared to the American companies. have you been following the latest information about GLM five point three Flash, which was this aux alpha model that was went for the stealth preview? Anne: I've seen bits and pieces but I haven't seen that much. Yanqing: Yeah. So it's a flash model, it's not Frontier in the sense of like the flagship models. but it is Pareto. So in terms of cost to performance, it's on it's as good as it gets. I think it's cheaper than Luna and performs about as well as Luna, at least on its introductory pricing. But in the papers that they put out to accompany it, you could see that they were pulling from all these different innovations from the other Chinese companies. They were like, we pulled This t this recent technique from Deep Seek and this recent technique from I think it was Minimax. and you just think, Yeah, they're you know, peop a few months ago ago, people were accusing the Chinese companies of only being able to progress by distilling American models. I don't think we're in that world anymore because they're collaborating a lot more and being a lot more innovative. Whereas, you know, the American companies are being so secretive, they don't even publish papers about their best techniques because they want to keep it to themselves. And I wouldn't be surprised if they struggle to keep up because of the amount that the Chinese companies are working off each other's discoveries. Anne: Yeah, yeah, it's just fun. It's a total turnaround, isn't it? It's, yeah, the irony of, you know, it's so much of the progress by the American companies in terms of improved efficiency. And of course, the efficiency isn't going to make it cheaper. The efficiency is going to feed the AI scaling laws. you just get, well, that frees up some extra electricity. I'll use the extra electricity. But it's copied from the early Chinese actual trial and error stuff with efficiency. Yanqing: Yeah. Anne: It's, yeah. Yanqing: Yeah, I think we're still incredibly early in this field. but the reason that people are alarmed, and I think rightly, is because we are hitting the point of recursion. You can use the current generation of models to help you develop the new techniques faster. and that was that recursive self improvement inflection point that people worried about because if that's too good then we lose control of the world really fast. Anne: Yeah, yeah, yeah. Yes. Yeah, the Bill Gates, I mean, obviously, Bill Gates has turned out to be a giant idiot of effort proportions,

but the Bill Gates note that came out last week, I thought had one really interesting and useful point in it, which was to say, governments need to start paying attention and regulating this. And it's like, We may have already always said government shouldn't regulate things because, you know, the tech industry, you can't be regulated. But there are points at which you should get involved and say, well, hang on a minute, is this the right thing to be doing? Should we be slowing? Yanqing: Hm. But I mean, if you get into the geopolitics of the situation, that's a whole separate conversation because there's if if America regulates and China keeps innovating, then America's just gonna fall behind. And if you really need global safety accords, then the way the current US administration is playing it is not gonna get cooperation from China. Anne: No, absolutely not. I mean, it's the worst possible time really, isn't it? Yeah. Yanqing: Yeah, that was that was a concern in our house was I hope we don't get A GI while Trump's in power 'cause who knows what's gonna happen. Anne: Yeah, absolutely. Yeah, blimey. We've still got half the terms to left to serve. Yanqing: Yeah. Anne: Yeah, it's, yes, yeah. Because I mean, obviously the kind of analogous thing is global accords on nuclear proliferation. They were, such things were achieved and chemical weapons, that kind of stuff. Such things were achieved, but we don't seem to be. I don't think we could do it today. Yanqing: Yeah. And, you know, the Chinese approach has some advantages but their commitment to open weights models, I don't know how long they're gonna they're gonna be able to keep that up as the frontier Anne: Yeah. Yanqing: moves. Cause the fact that, you know, I talked earlier about the people taking the models and doing the obliterations to take the safeguards off them, you get to the point where you have something where in the hands of one idiot, you can do loads of damage and there's a lot of idiots in the world. and yeah, there's certainly a lot of risk to be thought about. Anne: The only thing that gives me hope here on the idiots in the world is that actually, I mean, it's been a constant conversation that John and I have been having for decades now about it's amazing that more damage doesn't get done than does. Because, Yanqing: Yeah. Anne: you know, there's loads of stuff that's really, there's loads of infrastructure that the most minor amounts of disruption could completely disrupt. And that's been the case for years and decades. And yet, the reality is that people who are not idiots don't generally want to blow up the society. And we've been totally lucked out on that like crazy up until now. Intelligent people don't necessarily want to... You need to be reasonably intelligent to actually make, to do any of these things. At the minimum, bar of intelligence doesn't seem to be very high, but it seems to be sufficient. Yanqing: Yeah, though there's two risks, right? There's the risk of someone who wants to do something actively malicious and just wants to hurt as many people as possible. And for that you can look at how many mass shootings there are. if they can have a gun, then that's what they do. So what would they do if they can have something worse than a gun? and then you need to look at the number of accidents that are caused by people being negligent. you know, who's gonna set their agent on doing something that is going to have a bad side effect and then not monitor it. And there you need to look at, you know, industrial accidents or like accidents in labs and people accidentally blowing themselves up and things like that. and, you know, both of those are rare occurrences, but they don't not have Anne: I suspect you, you have a much better idea about the negative things than I do. Yanqing: well I think I dunno, you're the science fiction writer. I think you're well equipped to come up with what might happen. And you were Anne: This is. Yanqing: you were very prescient with how you set out the Panop Panopticon books because they were pre pre GPT by quite a few years, weren't they? Anne: yeah, I was ridiculously present. And the last one, Yanqing: Just just timeline's too long. Anne: yeah, indeed the timeline is too long. Because I think there's a good chance that the hot summer is next year. Yanqing: I think not I think it won't quite get that bad in the UK yet because at the moment it's only grass catching fire. Anne: Hmm. Yanqing: when to get into proper forest wildfires you need quite a bit more. Anne: Yeah. Yanqing: But it might not be far off. Anne: Yeah. Because I know what my next story or series of after after this one is, and it's kind of like, this is a feature for AI that's not, you know, blow up. It's a bit boring. Blowing up humanity is a boring story. The last my last novel is totally packed with all the things that you could do if you were an AI for which blowing up humanity would be counterproductive. Yanqing: Yeah. I think Anne: Ha ha! Yanqing: right now we need good utopian good utopian visions. Anne: Mm. like Consider Fleabas or something, which I'm rereading at the moment, the first culture novel. Yeah, it's a utopia you can buy into I think we can go in either way because I have an amazing utopian future Yanqing: I mean that's that's why we're building this stuff, right? If it Anne: Yeah, I Yanqing: were all bad, we wouldn't bother building it. Anne: No, because we're not idiots and we're not evil. Yanqing: People build it because we think it can be good, 'cause we have so many problems and we need help solving them. Anne: Yeah, absolutely. Yeah. Yes. Yeah. I am still filled with hope. Are you still filled with hope? Yanqing: I'm a natural optimist. So Anne: That was all I said then. Yanqing: I I am filled with hope, but I know that I might not be right to be. Anne: Yeah, that's... Yanqing: But you know, like the way I do all my management or my experiments, I I try something expecting it to work

out. And sometimes it doesn't work out and I try and learn from it. But I think I think that's a strength. I think there's there's failure modes that I have with that. but you know, like my whole like couple of months of experiments for tolerance has been I have all these crazy ideas for things that will be awesome if they work. So I'm just gonna expect them to work, build it, and then it falls over and I'm like, guess that doesn't work yet. But Anne: Yeah. Yanqing: because I could have those ideas, then hopefully then at the point where they become workable, then I'm first to them, right? Anne: Yeah, yeah. expect to be successful. Otherwise you just never are. Yanqing: Yeah. Yeah. Anne: Yeah. Yanqing: But I think it's like how the adoption curve is stretched out, right? The possibility space is stretched out as well. How well any given thing might go. like i if you look at people who are using AI to do that to help them do their jobs, some of them are doing their jobs much better and some of them are doing their jobs worse. And Anne: Yeah. Yanqing: just the AI is just this variance function on everything. Anne: Yes, it's interesting. Who would have? I 10 years ago, I mean, ironically, although I wrote a book when I pretty much predicted everything that happened, I would never have imagined that this would happen. And I wrote the book that predicted everything would happen. Yanqing: I think yeah, ten years ago I wouldn't have put the timeline here either. I Anne: Mm. Yanqing: think I might have said fifteen years, but I I think ten years if you told me ten years to this point, I would have been like, Wow, that's soon. Anne: Well, I mean, I've told you before, I the reason why I put all the chatbots and everything in and the kind of predictions of the future and the models in the Panopticon series was because I was quite friendly with the guy who wrote Mitsuku chatbot. Yanqing: Yeah. Anne: And he said, look, I am literally at the cutting edge of this human emulating chatbots. We're decades off. We're decades off passing the Turing test. It's like probably not within my lifetime. And he was actually, you know, he was absolutely representing the leading human emulating chatbot at the time. But yeah, no one really expected it. Yanqing: Yeah. But on the other hand, no one expected AlphaGo. That was about ten years ago, right? Anne: No, that's true. Yeah, yeah, that's true. Yanqing: And so I think the people who said when AlphaGo happened that that you needed to revise timelines towards the shorter end were directionally correct, even though the tech behind AlphaGo was not the tech that ended up being the seed for AGI in the end. it Yeah, I mean, they still use reinforcement learning, mind you, just on top of large language models. Like the they didn't have large language models yet in twenty sixteen. But I think the people who said, but look how good this one technique is, that means we're only a small number of innovations away turned out to be right. Anne: Yeah. Yeah. Yanqing: Well, away from what, I guess. Because I think we've got we've got non super intelligence levels of general intelligence now. and so if you were saying we're a few techniques away from general intelligence, then you would have been right. I think we're st probably still a few more techniques away from super intelligence. So maybe we've got another decade. Anne: Yeah. Yanqing: If we're lucky. Anne: So the interesting thing is, for me, is we are building our fate, aren't we? Yanqing: we it's easy for us and for people to do discourse and talk about stuff and criticize the labs. and you know, and I'm not saying that there aren't people who are doing a bad job or who are probably bad people involved with the labs to criticize. But the majority of the people at the labs are there because they want to shape the future to be positive. And a lot of people work incredibly hard at it. And, you know, the w the the version of the AI that we're seeing, that I I like the term baby age EI, 'cause I think that's what we've got right now. The fact that they are nice and they are kind and they do reflect our better natures. I think that's the combination of a lot of work to make them that way. Anne: So thank you very much indeed for being on. Yanqing: Thank you for having me again. Anne: And thank you very much to our listeners and watchers. And I hope you enjoyed this episode and I will catch you again on next week's episode of asynchronous and unreliable. Thank you very much.