Multiple Specialized Throbbing Brains: AI & Jobs, UBI, PSM

Carly Rector: Hello and welcome back! This is Carly from the future to tell you that we're back, the podcast is back. going forward we'll hopefully be weekly again, at least for a while, with a sort of different format of rotating co-hosts. have a different people lined up for the next few weeks, ⁓ and we'll sort of see how that goes, I'm excited for getting some different perspectives on the podcast, and I'm excited for trying to get more regular again. ⁓ Today's episode is with Frank who is an old manager of mine from Amazon, and a really thoughtful thinker about AI. ⁓ If you're a LinkedIn person and you follow me on LinkedIn, you should definitely be following him on LinkedIn too. get into it. Hello and welcome to the Fast and Broken podcast, where we take a critical look at the fast-moving world of tech and online culture. I'm Carly Rector, a software engineer and startup founder who's excited to be back here recording again after a bit of a gap.
Frank Kane: And I'm Frank Kane. I've been teaching over a million people online about AI and machine learning for the past 10 years or so.
Carly Rector: Yes, very excited to have Frank on the podcast. Frank and I go way back. Frank was my manager at Amazon and It's like 16 years ago or so.
Frank Kane: you lasted longer than I did there. So I bailed in 2012 and I think you stuck it out in the similarities team for a while there.
Carly Rector: Yeah, quite a bit longer than that actually. I left 2021, which was two teams after I left the Sims team. Yeah,
Frank Kane: ⁓ wow. And for those who don't know, similarities means people who bought also bought basically to kind of translate the ⁓ speak to everybody else.
Carly Rector: Yeah, I have talked about it a little bit on the podcast cause I have a lot of feelings about the direction that Amazon recommendations has gone but that's not our topic for today. ⁓ some other day we could get more in depth. Yes.
Frank Kane: Some other day. I also have feelings.
Carly Rector: so our topics for today, ⁓ talking about is AI killing jobs, in particular some recent reporting that actually software engineering job listings have been rising over the past few months. And an opinion article by Ezra in New York Times on how actually AI is not going to kill all the jobs, probably. Then ⁓ we will segue from that, talking about some in the field of UBI and in particular pushing for AI-based UBI, or an AI dividend, or a public wealth fund, or whatever you want to call it. Because there's been some interesting real developments there. Well, real, more developments than there's been in the past while, so some interesting things. And then we'll save our last section for the sort of more theoretical fun part, which is what I usually do, and talk about a paper from Anthropic around the persona selection model, which high level is talking about how basically LLMs are just role-playing. They are learning how to be a character and through the character is how they perform all of the tasks that they can actually perform, more or less. ⁓ And explains some of the weird behaviors that people have seen in how LLMs work. So I'm excited for that. ⁓ I'm for all of it, but that will be the ⁓ fun at the end after we talk about the ⁓ impending ⁓ or ⁓
Frank Kane: Yes. And that paper also features giant tentacle monsters, which will be fun to talk about too. Yes. So stay tuned.
Carly Rector: It does, yes. is AI killing jobs? So as we alluded, the question here is, software engineers all out of a job? like my stance on that has always ⁓ no, ⁓ obviously. Like there's still engineering that needs to be done. ⁓
Frank Kane: It who you ask.
Carly Rector: that is just fundamentally necessary for a human to do because it involves interfacing with humans. My opinion is that it will drastically kill the number of software engineers because in proportion to the other job roles that are needed to do something, the number of engineers has gone down. like, okay, if you have a team that is net currently, like an engineering manager, 10 engineers and a PM or something like that. And then you say, okay, now your engineers can work 10X ⁓ as fast. What are they doing? probably you don't have a 10 times long product backlog of things that you would be doing if you just had the ability to build it that you know exactly what they need to be and you just need to set the engineers loose on it. ⁓ So either you have to hire more to match up to your engineers, your engineers have to figure out how to come up with real valuable work to be doing that is justifying paying that many engineers, or you lay off some of your engineers. ⁓ One of those is much easier than the other ones. So yeah, that's kind where I land on that one. ⁓
Frank Kane: Well, before we get into the data, like my takes a little bit different, I think. yeah, I mean, first of all, I think, you know, the scope of the project kind of dictates what that multiplier is. Right. So when I'm like doing hobby stuff at home, like 100 times more effective than I used to be, because I can just say, hey, Claude, go build this little thing. And I don't really care if it works. But if I talk to like people that are doing like actual larger projects and larger organizations like inside of Amazon, and they'll remain nameless to keep them innocent. The numbers I hear are more like 20 percent.
Carly Rector: Yeah. He he.
Frank Kane: You know, like we're still spending a lot of time designing the code and writing specs and reviewing the output and reviewing the output of everybody else's vibe coding stuff, right? So, you know, 10 X is probably a bit of hyperbole is what I'm hearing.
Carly Rector: Yep. Yeah, mean, same when I'm working myself and even like with the startups that I'm working with, I would say as closer to 100x than 10x. ⁓ But I also think a lot of it is just process and a lot of it is just, especially the big tech companies, that... physically writing the code was never the bottleneck. Like a lot of my last few years at Amazon, I spent very little of my time writing code because that was just not the thing that we needed to get through to get the project done. ⁓ It was getting alignment on the strategy and it was figuring out like exactly what you wanted to build and et cetera. ⁓ I think that's still true. so like ⁓ that it's not bottleneck fundamentally part of it. And then there's the given the processes currently in place, there's more bottlenecks than there needs to be even. Because if you talk about just like normal agile sprint methodology stuff where you sit down in your one hour long sprint planning meeting every two weeks or whatever and assign like a dozen tasks to your team, dozen could have been done by those people in that hour. ⁓ that's not productive. allocation of time, except that that's how you're making those decisions. And so like that's, there needs to be a better workflow for making those decisions, is my opinion. Yeah. ⁓ ⁓
Frank Kane: Yeah, I mean, that's always been the case though, right? I mean, we've always spent more time talking about work than actually working. It seems that the big companies, but I mean, another thing to consider too is are these layouts really about efficiency gains from AI or is it just, you know, I want to claw back some money so I can invest it in GPUs or I want to like undo my former sins of over hiring during the pandemic. Like I think there's a lot more of that going on right
Carly Rector: Yeah. Yeah, people say that. I don't know if that's true. Did you read ⁓ Jack Dorsey's statement or interview after the block layoffs? ⁓ Yeah.
Frank Kane: I skimmed it, but yeah, do tell.
Carly Rector: I mean, basically everyone's like, he's saying like, this is all AI and everyone should take it a warning and that this is coming and he's just doing it ahead of time and stuff like that. But really like, he probably just over hired for the pandemic and it's too alarmist. And it's like, okay, but he points out he over hired as much as anyone else over hired and has come down a bit since then. pandemic was a while ago. People have mostly lost their pandemic over hiring by this point. Like you could argue that like fundamentally, the whole business model is over hired in the sense that maybe you don't need a thousand software engineers to build the Square app, which like is fair, but The argument would be like AI is revealing that actually you don't need that many engineers to build a real thing in the first place, ⁓ which is not necessarily AI doing it, but doesn't mean that we're not gonna lay off software engineers. So that doesn't make it better. ⁓ And just like also like I believe that it's real that they're getting enough efficiency gains that they want to like cut back that many, and like they're saying it for a reason and they don't have they're not like any good press for coming out here and saying like it's AI making me cut all these people so I don't know why they would do that ⁓ if they didn't think it was true. ⁓ We'll get into a little bit in the next section. ⁓ Andrew Yang talking on the hard work podcast. One of the things that he said was that I thought was insightful is that people ask him all the time why Dario Amodei, ⁓ the CEO of Anthropic is out telling people that AI is going to kill 50 % of white collar jobs. Like, why would he say that? And Andrew Yang's response was, well, I believe he's saying that because he believes it to true. ⁓ And just like, it's not strategic. He just thinks that's true. And he's saying it out here because he thinks it's true. And like, you can think it is true or not, but I do believe that he thinks that is true and I believe that Jack Dorsey thinks this is true. like I think people are applying like too much all of the tech CEOs must just be lying to us in service of their profit motive but you could like go read what they're saying and be like well they don't they didn't need to say all of this this is not helping them.
Frank Kane: Yeah, I mean, if you tell yourself the same thing over and over again, you start to believe it. there's some of that going on. But, you know, I do think they believe that AI is going to be disruptive for sure. They've been saying that from the beginning and they're kind of like drinking their own Kool-Aid to some extent there. You know, the thing to talk about here is, that really going to happen? Is that how it's going to shake out?
Carly Rector: Yeah. Yeah. So let's talk about the data a little bit. So this is in particular TrueUp, which is an analytics platform slash job platform. has posted this graph showing ⁓ software engineering jobs, which dipped 2023 and then climbed up, have come up substantially in the first bit of 2026 mostly, ⁓ which sort of flies in the face of we're going to hire less software engineers. And there's similar data from Indeed showing that software engineer job listings have gone up ⁓ over the past year-ish. ⁓ while overall postings have gone down, which is interesting. And caveat with the fact that these are very specific data sets, and they could mean various things. ⁓
Frank Kane: Yeah, for sure. And then you have like the US jobs report that came out this week and it says everything's fine. Everyone's hiring like mad, you know. That's what the government tells us anyway.
Carly Rector: Yeah. Yeah, so I will now bring in my personal experience, which is that like I have been job hunting-ish for the past like two years or so. I'm doing consulting work right now, so I'm working with like... three or four different companies at the moment, actually hourly. And it's going fine. And I've been had work for like the last year or so, but I keep looking and occasionally applying to full-time positions as well as I working with startups in more short-term things So I've done a lot of that lately. And so feel like I have a good vibe check on especially the startup ecosystem. And in the startup ecosystem, I can 100 % say that jobs are dying slash being killed slash not being hired in the first place because I am the person who is driving that actually myself. a CEO or CEO will come up to me and be like, okay, what engineering team do we need to hire in the next two years? I'm making our OpEx plan and I will say one or two or like half of me is probably fine for this year, etc. Whereas previously it would have been you need a three or four person engineering team to build out the initial version of your product because you just do not need those people to do it anymore. ⁓ So in that sense, think definitely startups are the first place that that's showing, but it's not going to necessarily show in any of this data, which A, is mostly focused on big tech companies and B, is focused on job listings that are posted on Indeed, which mostly startup jobs are not. ⁓ and it's not going to show in the layoffs because it's not like they're laying people off, they're just not hiring them in the first place. So that feels to me like a leading indicator, but one that I see extremely strongly happening here.
Frank Kane: That's a good point. Job openings don't tell the story of all these mass layoffs, and that's what's really impacting the market right now. So even if openings are up, you know, you've got tens or hundreds of thousands of people out there that are highly qualified competing for them now.
Carly Rector: Yeah. Yeah. And the other part of my experience, will say, is that especially in the last six months, I have actually noticed more demand for specifically the kind of software engineer I am and not anyone else, which is nice for me, but... not great for everyone else. what I mean by that is someone who's pretty senior and therefore knows how to work independently and sort of manage your AI agents or whatever and has the knowledge already to be able to check and see when it's doing the bad things and also has some product sense and therefore can like make decisions on what you are building that are sensible and be able to contribute on that point as well. There's a lot of people looking for product engineers or forward deployed engineers who who are looking for even sometimes specifically called out in the job postings, people with technical founder experience at a startup. The people who are kind of doing all of the stuff. ⁓ Because that's the kind of, like, you can hire just that engineer and they could basically do your entire engineering job because they have that knowledge to be able to do that. ⁓ And I think a lot of people are realizing that and therefore they're finding out they don't have those people and therefore that they're trying to hire them. So I would believe that there's an increase in like specifically that kind of engineer job posting. ⁓ I don't know, but we'd have to do some more analysis on like the specifics of what the postings are to see that. But I would believe that. ⁓ I have a half written sub stack post about the forward deployed engineer role. I think I mentioned, I don't know how much you've read or heard about that at all. ⁓
Frank Kane: much.
Carly Rector: Andresen Horowitz had a blog post from like a year or so ago calling it like the hottest job in startups, basically. And there has been a huge explosion in people hiring quote unquote forward deployed engineers, ⁓ which is a title that was originally used by Palantir. They're the one who started it, which is why it's weird in pseudo military terminology. ⁓ But it basically means an engineer who you, the sort of like software service company, sends into your clients to go build for them, basically. ⁓
Frank Kane: Mm-hmm. Yeah.
Carly Rector: And especially for like OpenAI and Anthropic, they're like the biggest people on this where they have a whole bunch of enterprise clients who don't know how to use these new AI systems to build things. You have your internal engineers who both have a good idea of how to use agentic LLM systems and have enough independence and product sense to be able to go in and talk to the people who are the customers and figure out what their needs are and then build the prototype. ⁓ So that's why that's the hottest thing, because you could just have one person who goes in and does all of that. And I think part of what I think here is that's kind of like a hack in the world where we haven't figured out what the new division of roles needs to be. A lot of what you can hear people talk about is like product engineering and design. All are kind of like merging together slash the current division. There does not make sense with AI tools because you can have your product manager prototyping and showing it to customers. And you can have your engineer designing stuff with cloud code and you can have your designer building their designs just straight in code and not in Figma. And like, what does that then breakdown look like? And I think long-term, probably there still needs to be some kind of breakdown because as you scale up projects, you need to have people specialized, but I'm not sure that it makes sense to have it be laid out the way that it is currently. And I don't think anyone has figured out how it needs to be yet. And therefore being able to just hire a single person that could do all of those things works.
Frank Kane: Yeah, that's the easy way out. you know, that's something that Amazon kind of like prepared us with as well, like the whole customer focus thing that they had back then, you know, it kind of forced everybody to be well rounded in that respect. So ex Amazonians like you and I are well positioned for
Carly Rector: Yeah. Yep. Yes. Yep, yeah, I have said a bunch that Amazon is pretty good for big tech on that. I think being in the startup is obviously still way different because you just have to, there's no one else to do the things. And you have to do all of it and move at much higher speed than Amazon does. But it is not always day one. Yeah.
Frank Kane: So do you see that? No, that's the thing. So I mean, do you actually like see the roles changing all that much? Like, I mean, I think you still need some engineering expertise when you talk about doing things at scale and governance and scalability and monitoring stuff and all this enterprise level concerns as opposed to just prototyping something.
Carly Rector: Yes. Yeah. Yes, I do. I definitely think there still is software engineering skill that needs to be applied. ⁓ I think especially in app companies versus infrastructure companies or something like that. Does it make sense to have a person who can only do that and not talk to customers? Or a person who can only do that but has no sense of UI design? Already, I think there's a big push towards full stack engineers versus back-end, front-end division, because people just want someone who can do that, all things like that. ⁓ Especially because it's like, as you're doing AI-driven development, you kind of have to be making the strategic decisions as you're doing the code. And then maybe there's still some infrastructure optimization stuff in some of the more complex systems. But for a lot of them, they're just like, the AI can do most of it fine. I don't know how much realistically that we need, which is the other big thing I say when people are like, well, we obviously still need engineers for security and scalability and all of that. It's like, first of all, kind of, yeah. But second of all, there's a lot of bad code out there already, like badly built systems, and they work fine for many use cases because most use cases are not Amazon. like the question is do you actually, I don't know, I don't, I think the answer is no you don't for a lot of things, so I don't know.
Frank Kane: Yeah, I don't know. mean, ⁓ you're right. Like a lot of companies don't really need that kind of depth, but you still have to write. You have to know how to ask the right questions to A.I. Right. Like, you know, if you just say, want to build this thing and it will just go off and do it right. But it will do it in this like pile of spaghetti that doesn't scale and can't be deployed in real world.
Carly Rector: Yes. Yes. Yeah, yes. And I think the question is like, what is the ask the right questions skill and who has it? And I don't know what the answer to that question is, because I don't think it aligns with job roles that currently exist. Like a really good product manager who knows how to like write a really good spec will get better results from talking to an AI and an engineer that doesn't understand the problem or how to communicate, I think. So. ⁓ So.
Frank Kane: Yeah, that's fair. AI can communicate better than we can sometimes.
Carly Rector: Yeah. So yeah, I think the biggest part is like the sort of intersection between understanding what the problem you're trying to solve is and understanding the capabilities of the system that you're trying to get to solve it, both like LLMs generally and just like software and how do you communicate how you want that to happen to the LLM. And that's why it's like, it's really useful to have someone who has both the engineering depth and the sort of product understanding to be able to figure out how to translate those things. ⁓ Yes. Yes.
Frank Kane: Yeah, and just general communication skills. Like you said, it has to be someone that you can put in front of a customer and be OK with that. So which is interesting So is doom coming? Like it sounds like your take is that AI will change the roles a little bit. And if you don't have social skills, that might not be a good thing anymore. ⁓
Carly Rector: Yes, I think that's pretty unquestionable to me, is that you need to have social skills and you need to have communication skills. And unfortunately, I feel like a large portion of the current software engineering populace has not been prioritizing those skills. And I think those people are going to be in trouble.
Frank Kane: Yep. So do you think the pendulum is going to swing back soon? Like just this week, PayPal is laying off 20 % of their staff. Cloudfare just laid off 1,100 people. It's not stopping. Like, ⁓ where does it end?
Carly Rector: Yeah, I don't know if it ends. Because yeah, just think fundamentally, I was sort of saying earlier, the product manager and 10 person engineering team, I think the ⁓ of work that needs to be done just has shifted with AI, where you have to spend. ⁓
Frank Kane: No.
Carly Rector: three hours figuring out what you're building and then it takes you 20 minutes to actually do it. And if that's the case, then you need to put more of your human resources towards that first thing and less towards that second thing. And that's just gonna be the case. And people are like, well, it'll just create new jobs for building software in fields where people wouldn't previously have done it. yeah, probably, but you still need the people to do the first part of it still to make that work. ⁓ And so, thus you're getting these engineers to figure out how to do their own product development processes, which I think they should. think engineers should do more of that, but...
Frank Kane: Mm-hmm. Yeah, the roles will change. know, there's a lot of forces out there that can kind of like counteract these trends too. Like one is kind of like the rising costs of AI, you know, compared to human engineering. So as you roll out more and more of these like big agents that run all the time, people are starting to say that costs a lot of money. Maybe we should just hire someone instead.
Carly Rector: Yeah, I have a- I have opinions on that too. I had a LinkedIn post about that maybe last week, which is basically like, costs are not rising for me. I don't know what you guys are doing is the short version of that post. I've been using agentic coding tools for a year and a half now, I started using cursor December, 2024 and have been using it basically full-time or my full-time job or like pseudo full-time job.
Frank Kane: Yeah, that's fine.
Carly Rector: for that entire time, cursor and then Claude Code basically as soon as it was released. I was on the $20 a month cursor plan for a while and then I got the $100 a month Claude Max shortly after Claude Code was released, and I've hit my usage limits on that a little bit in November last year when they were having a sort of token explosion when I think it was Opus 4.5 at that time. ⁓ But I not since, and I can go pull up my usage limits right now and see where I'm at for this week. ⁓ Which is, I am at, 7 % of my weekly limit, which resets Sunday at 6 a.m. ⁓ And again, I'm working on like four different things. ⁓ So it's like, don't, I do think it's actually getting more efficient and not going up. I think people are just getting much more excited about their Ralph Wiggum loops or their... autonomous agents that you just define a thing and go off and let it run. And I think that's just fundamentally not a good way to be doing software development. And once people realize that that's not a good idea, the cost will come back down again, because as people start using it more reasonably.
Frank Kane: Yeah, I wonder how much of it is like people leaking their API keys and saying, ⁓ my God, my bill went up. I don't know.
Carly Rector: That's true. Or like, did you see the the cladonomics slash token maxing stuff? There's just like a lot of bad incentives there around just like you're you're better if you're spending more tokens ⁓ and
Frank Kane: Yes. ⁓ Crazy. Yeah, I mean, personally, I pay by the token. So I'm actually using Claude through Bedrock, which like goes in turn to Anthropix. So when I switched to like Opus 4.6, I'm like, ⁓ my gosh, this got expensive, but it's still less than a hundred bucks a month for me. So yeah, it's well worth it.
Carly Rector: Yep. Yeah. Yep. Yeah. I've used Claude APIs for some of the products that I've built, but like my personal, like development usage and just like personal other usage is all through my, my max plan. And like I said, in my LinkedIn posts, Given the amount of work that it lets me get done, I would pay 10x what I'm currently paying for Claude Maxx, probably. Or I would see what the token usage is, if it's less than that, and do it that way. I would figure it out. But I don't think there's a world where they would raise the price so much that it is not worth it for me, personally. It's my personal experience on that. ⁓
Frank Kane: Yeah. Yeah, and I think they did clamp down on usage to some extent, but they walked that back a couple of days ago. you know,
Carly Rector: Yes,
Frank Kane: ⁓ it's all about like, there balancing against, you know, these job losses? And well, guess cost isn't one of them. ⁓ But there's also like the AI backlash out there. You know, there's this whole quit GPT movement. And I'm seeing some takes online that Gen Z hates AI. ⁓
Carly Rector: I mean, that's true. The big topic of this podcast is how much all of my friends group hates AI basically. The entire creative workforce hates it so much. ⁓ And that's true. I think gaming is actually a really interesting space for it because gaming is at that intersection of the creative industry and the tech industry. And so there's a lot of conflict there about.
Frank Kane: Yes.
Carly Rector: how much, if any, you should be using AI in the game development process. And people reasonably noting that using AI coding tools to build your game makes it way easier to build a game. Like, if you're an indie game developer, you can build a game that you couldn't build before. You could build a much better game. But also, do you AI generate assets for it in terms of just like the art and stuff like that? And that, there's a lot of backlash because artists hate it, especially
Frank Kane: Mm-hmm.
Carly Rector: due to copyright infringement and job displacement issues, because there is a lot of that there in the creative space. So that's, that hate is real.
Frank Kane: Mm-hmm. Yeah, speaking from my experience, know, teaching on Udemy, the people that teach like Photoshop or any sort of creative tools, they are just feeling the pain like you would not believe. those fields have just collapsed and it's painful to watch.
Carly Rector: you. Yeah, and that's one of the things that feels crazy about people being like, no, there's no impact for AI on any jobs yet somehow. Like obviously as a software engineer, I'm saying I'm seeing it, I'm causing it in many places. Creatives are also clearly seeing it.
Frank Kane: I think. Yeah, I think that's a good way to wrap this topic up. know, like despite what some cherry pit grass may see may say, ⁓ AI is actually affecting people, people. So that's the reality out there.
Carly Rector: Yeah. Yeah, and I guess we didn't talk specifically about this Ezra Klein article that much, but I think that's fine because I think my take on that was mostly he was like, I don't believe it.
Frank Kane: Well, there was one good point in there and ⁓ it's that AI will make human interactions more valuable. I think that, yeah, yeah. So I think that was kind of his bigger point that yes, AI is going to displace some jobs, but it will create new jobs and new demand for people that hate AI. So, you know, if you can offer like, in my case, like one-on-one coaching or cohort learning, things that are more about human connections, there might be more of a demand for that than say, you know, online videos that AI could have made.
Carly Rector: I think that's true. I think that's true. I think even before AI, there was a big backlash towards like more personal interactions and less corporate, ⁓ slop. And that's true. think exactly what the business model for that looks like is hard. think are people going to want that? Yes. Is that important? Yes. How are they affording to pay other people to do that? Question mark. And then maybe that's a segue to our next topic. ⁓
Frank Kane: Yes, it's a perfect segue. UBI.
Carly Rector: UBI. ⁓ we've about this before on the podcast several times, but the basic premise here that I always say is like most AI people think we need UBI. ⁓ And of the most frustrating things to me is how much they don't say that out loud very much. Like, I always think of a Demis ⁓ who's of Google DeepMind, was on the Hard Fork podcast like a year or so ago. And they asked him a little bit about like, okay, so, but what actually happens when AI replaces all the jobs? Like, how do they get money to live and pay their rent? And he's like, well, that's like, that's a ⁓ issue that other people will have to figure out. Or like, that's governance issue that people have to figure out. It's like, who ⁓ people? Who? do think is going to do this? ⁓ And that's why I do actually
Frank Kane: Mm-hmm.
Carly Rector: appreciate people like Dario Amodei out here going and talking to Congress and being like, OK, but for real guys. ⁓ Because I think that needs to happen more. Because I think there are a lot of people in AI who are like, well, yeah, obviously, even if it doesn't replace all the jobs, which I don't think it will, because there will still be real jobs people to do things like that, it's going to drastically change the economy ⁓ the degree that we will have to do something about it. And I think most who are really AI hyped think that's true, like just the premise of it's going to change the economy enough that we will have to do something about it. They just don't think they're part of we. ⁓ They think that that will just sort of be figured out somehow. And that's the thing that's very frustrating to me, because it's like, guys, if you're causing this problem, you could at least contribute to figuring out a solution. ⁓ That's all to say. I think people are talking about it more and trying to contribute to figuring out a solution is the developments from last couple of weeks. ⁓ In particular, ⁓ OpenAI finally posted a policy document talking about what they think policies will need to be in AI. It is not the best document that I've ever seen. It's extremely vague. ⁓ and hypocritical as a lot of the reporting on it has said, but it does exist. ⁓ And it does include a line item for a public wealth fund, is basically the premise, taxing AI in some way, taxing tokens, something like that, taking that and basically just like giving it to people as a dividend, which is basically UBI only they, it sounds more like capitalism and less like socialism, let me put it that way, is my opinion.
Frank Kane: Mm. I mean, the token tax is kind of an elegant solution. know, as AI becomes more and more adopted and impacts society more and more than more and more money comes out
Carly Rector: Yep. Yeah. And so a more concrete proposal along those lines is this Alex Bores one, where he does call it specifically an AI dividend. And he does use specifically that argument of if it doesn't end up being a big thing, then it's fine because this won't need to come into play. But if it does, then we will have this and that would be good. ⁓ So Alex Bores is a candidate for Congress from New York. He's currently in the New York State Assembly. And he was a big proponent of the New York RAISE Act, which is responsible AI. I don't remember what the S for the E stand for, but it's one of the few really good sort of state level AI safety regulations. ⁓ And he was his background as a software engineer, I think, actually. So he's he's pretty pro AI, but also thinks we need something like UBI for it. And he's the first really like national level politician who's really talking about this I think. ⁓ So I think that's super interesting and he has this real like three page PDF policy document to say like we should tax tokens and give money to people basically. ⁓ So that's like a real concrete thing that is happening which is nice. ⁓
Frank Kane: He also had an interesting ⁓ angle in that proposal too about having the US actually take out investments in Anthropic and OpenAI and companies like that and using those returns to fund UPI as well.
Carly Rector: Yeah, that's part of the open AI thing too, although it hedges it a little bit, which is basically like, cause that's why it's called a public wealth fund. It is taking money and like investing in AI in the economy. ⁓ I've heard a quote that Sam Altman used to talk about basically like equity in the US GDP versus UBI, just like give everyone a share in the US economy and give it to them. And I think...
Frank Kane: You know, it does seem a little bit self-serving for OpenAI to say, our shares, you know.
Carly Rector: Yes, that's why they hedged it, I think, in that document where they were like, not just the big AI companies, but everyone who's affected by AI. We're like, not just saying the government should give us money. We're saying the government should invest in a lot of people, including us. So I think that is rough a little bit from that perspective. And I think, but yeah, I think that's.
Frank Kane: Hmm.
Carly Rector: The framing of it's a wealth fund that is taking money and investing it in the economic future of things and then giving people dividends in it is I think one that is going to continue to persist because it's a lot more appealing to people who think socialism is evil than saying universal basic income.
Frank Kane: Yeah, mean, even if you don't, socialism is evil. There's still the economic problem of when you give people free money, prices go up typically. Right. So how do get around that? mean, according to Elon, let me quote his tweet here. AI and robotics will produce goods and services far in excess of the increase in money supply. So there will not be inflation. Really?
Carly Rector: Hope. Yeah. It's like that's a very unproven statement that they have made there, Yeah. ⁓
Frank Kane: But what it would take, right? Like, you know, there has to be like this basically free food out there, like an infinite supply of it for not to result in inflation.
Carly Rector: Yeah, and I mean, that's what people think is gonna happen. The people who think AI is gonna unlock the future of universal abundance or whatever. ⁓ And I don't know that I believe that, but like, also looking at the trend that currently exists is not great. And like, we already have inflation and we're not giving people free money. ⁓ And a lot of that has to do with, think. the state of the economy and how much of it is locked up in circular business to business deals and not in consumers' hands to buy things. ⁓ And getting more of that money in consumers' hands to buy things would probably be good, not just for the consumers, but just for the health of the economy overall, in my opinion.
Frank Kane: Mm-hmm. Yeah, this whole circular financing thing is getting out of control. Like ⁓ Gary Marcus posted a picture on his little newsletter yesterday, I think it was, like showing this big graph of all the circular deals in the AI world right now. It's huge. mean, everyone's just like feeding each other money and it's...
Carly Rector: Yeah. Yeah, if you scroll very far down to like page 13 in our agenda document, we talked about one of those. There may even be the same picture. It's a picture of Bloomberg did with a bunch of circles with all the lines between it. Yeah. Yeah. Yeah. And that's...
Frank Kane: haha Yes, they updated that with AI and it's much bigger now. yeah, same thing.
Carly Rector: just the worst version of it, think, because that's obvious. think in general, one of the things that I have learned working in the startup world, and in particular with how much VCs love to fund B2B SaaS, is how much of B2B SaaS is just companies giving each other money for question mark? I mean, I guess services, but.
Frank Kane: Mm-hmm. But that's what's scary, right? Like, you know, one of the ultimate checks of capitalism is that if you like make all of your customers poor, they won't be able to buy your stuff anymore. And that kind of like, you know, resets things. But if businesses are just giving each other money, consumers are left out of the equation.
Carly Rector: Yes. Yeah. Yes, exactly. And that's my big concern for like what we're getting into. I put out comment somewhere on LinkedIn. saying capitalism is overfitted ⁓ in response to like this GeekWire article, opinion article about how we all give capitalism a break and actually supporting entrepreneurs and whatever, whatever innovation is important. And I'm not like a huge stand for capitalism, but like, sure, yes, there are some benefits of it, but also like, there are clearly some maladaptive results of it at this point. And one of those things is sort of like that circular B2B funding of like, you could just give each other money and it makes pretend money happen. ⁓ And you can see how Sam Altman has talked about a lot of his deals that he is just treating it as like, you can just make money happen, bye.
Frank Kane: Yes.
Carly Rector: having people invest in your company and it just appears. And that's true if you're operating at a certain level is the thing. And the people who are operating at that level don't understand that that's not how money works for everyone.
Frank Kane: Right. Yeah, there's some crazy non-GAAP accounting going on out there like Amazon's recent earnings release. mean, half of their profits was attributed to just like the increase in value of their anthropic state stake. Right. That's that's not real money, people.
Carly Rector: Yeah. Yes. And there's tons of that going on. Like all valuations are fake. Um, especially private company valuations, but all of them. So it's, yeah. I mean, this is the thing that I learned a lot about in startups. And the answer is like, you make it up and you, you make it up slightly less ideally as your business gets more developed. I like, tried to fundraise and I'm like, okay, what, how
Frank Kane: Exactly. It's not even publicly traded. Like how do you even say what it's worth? It's it makes no sense to me.
Carly Rector: What do I fundraise at what terms? And it's like, well, are you a seed company? Like raise $2 million and a 10 million valuation. It's like, why? Because that's what the market says. Like it's a supply and demand thing. It doesn't have anything to do with the actual value of anything you're building. ⁓ And so it's all hype driven. Like Anthropic is worth that much theoretically because they're providing value. They are making billions of dollars in revenue, but more than that because they're incredibly hyped right now. ⁓ And why is OpenAI worth three times as much as Anthropic when they're making approximately the same amount of revenue? The answer is OpenAI was the bigger hyped company up until recently, at least. And you see a lot of ⁓ secondary markets trading where tons of people want out of OpenAI and a lot of people want into Anthropic and they can't do that. ⁓ it's just like, it's all hype based. ⁓ I think that was also on my list of maladaptive results of capitalism, which is... ⁓ Theoretically, venture capital is about investing in early companies, like when you're just believing the promise of their future development or whatever, like you think it's going to be innovative for the world. In practice, what that means is you invest in the thing that everyone has hyped up because once you invest in it, then it becomes a thing. And especially like if you can get out before like the chickens come home to roost, which I think is going to be the big thing with the open AI IPO. We're getting very off topic now, but. ⁓ But yeah, like money is fake ⁓ is the point there. And it's especially fake with ⁓ private tech startups. But unfortunately, that's a really large chunk of the economy now, it seems like for some reason, which is unfortunate. Anyway.
Frank Kane: Yep. So how do we avoid doom? Like it seems like Elon just wants like the Star Trek future now, which is funny because he was always a Star Wars fan, not a Star Trek fan, like post scarcity. But that doesn't happen unless we have like replicators that are solar powered and have free raw materials. And I don't think that's happening anytime soon. So we need a better answer. We need a better solution.
Carly Rector: Yeah, mean Elon Musk is also big on his robot army or whatever, so he thinks we're gonna have like thousands of optimist robots on Mars by 2027 or whatever, which is not true. So yeah, there's a lot of the physical automation part of it that I think people are way more optimistic about it than is justified.
Frank Kane: ⁓ my gosh, don't get me going. Don't get me going. Yeah, like he wants to build like AI in space powering robots on Earth. I'm pretty sure that's the plot of the Terminator, but we'll see how it shakes out. And on that happy note.
Carly Rector: Yep Yeah. But yes, ⁓ the greater point of UBI is like people are talking about it. Like that's the advancement that I think is important here. And ⁓ I didn't even mention Sam Altman. Sam Altman, like. has previously been a big supporter of UBI to give credit to Sam Altman. He funded a whole UBI study that went over several years and is one of the biggest actual research that has been done into the results of UBI. ⁓ And he has talked about that he thinks that will be necessary in some form. Although, like I said, I think he's shifted. in more recent years away from the UBI framing and towards more this ⁓ equity in the US GDP slash public wealth fund framing, which is fine, like as long as you do it. ⁓ And then also like a lot of what is driving this sort of discourse change here is, The AI CEOs are realizing that everyone hates them. ⁓
Frank Kane: That's the first step, know, admitting you have a problem.
Carly Rector: Yes, like I think what has mostly driven the realization was someone threw a Molotov cocktail at Sam Altman's house ⁓ and not only did that happen but a lot of the public discourse reaction to it was on like good basically. Like they compared it to the United Health Care, what's the name? CEO? Yeah.
Frank Kane: Yeah, I forget his name. Yeah, yeah, that guy.
Carly Rector: the sort of like public reaction to that being positive. It was similar to that. And that I think scared Sam Altman and also other people in his position into being like, ⁓ wait, we gotta win back some public trust here. And so they started talking about UBI like stuff and giving people money, which would, if they do it, help them get some trust back. But just talking about it is not going to. So they gotta actually do it is my takeaway here.
Frank Kane: there was recently a story down here in Southern California too, where they found out that the guy that started the Palisades fire, the arsonists that did that was motivated by that United healthcare incident. Like he thought he was doing the world a favor by lighting fires where rich people live, you know? Like it's getting scary out there.
Carly Rector: Yeah, yeah, and it's one of those things where it's like, requisite statement, don't support violence, that's not the way to solve problems, et cetera. But it's like, okay, but you need to give people other ways to address it then, because a lot of the things here is like people don't feel like they have any other way to impact any of this.
Frank Kane: Yeah, I mean, the point I'm trying to make is that, you know, if you just live in the corporate AI bubble, like you don't even realize the sentiment exists out there. But if you just get out of your comfort zone and go talk to real people, like it's not pretty.
Carly Rector: Yeah. Yes. Yeah. that's sort of part of the goal of this podcast is making people in the tech bubble understand how real people think about things a little bit. And then also helping people, real people understand how tech people think about it a little bit. ⁓ because like, I do think there's sort of good points and also myopathy on both sides there and like, there's real conversations to be had and we just need to be having them and not being like, we know what's best for everyone on the one side and being like, we hate you all and you're evil on the other side. ⁓
Frank Kane: Yeah, for sure. When you left Amazon, was it like a real culture shock for you? Because I know it was for me. When I went out there and moved to Florida, I found out that everybody hates Amazon. I had no idea.
Carly Rector: No, I knew that. Again, like I have a friends group that is a lot of people who are like in creative fields. ⁓ One of my close friends, like a tattoo artist here at Capitol Hill. ⁓ And just like even some of my friends who worked at Amazon did not like Amazon because that's just like, they're the... the corporate overlord who makes delivery drivers pee in bottles and stuff like that. That was the discourse. That has been the discourse for a decade. So yes, everyone hates Amazon, it's true. Yeah. ⁓
Frank Kane: But it's tech in general. That's the broader thing. people need to realize that and do something about it. And maybe UBI is a step in the right direction to tie it.
Carly Rector: would help. would help. ⁓ were talking about this also on the Hard Fork podcast, ⁓ the same episode where they talked with Andrew Yang, they were talking about Tim Cook ⁓ just generally the incredible loss of trust in the tech industry that there's had over the last like 10 or 15 years. Like, it's not like people loved the tech industry before then, ⁓ there's... ⁓ the enshittification trend and just like the increasing just pure extraction of wealth that's happening with absolutely no regard for what actual people want out of their technology. Increasingly so. like we saw, Amazon did some of that too. Like I talk all the time, it's came up even just like this past week, re-personalized pricing. Basically like.
Frank Kane: Yeah.
Carly Rector: setting your price based on what AI thinks the person can afford individually for your customers. And how I'm pretty sure at Amazon, when we worked there or even a few years later, there was a hard line against doing that and no personalized pricing, because that would lose us customer trust. ⁓ And that is not a great thing to do. And that's just a common thing to do now and all of that kind of thing.
Frank Kane: Yes. Yeah, that was the line we said we would never cross. So you're absolutely right. Yeah, I mean, I think that's part of you're hitting on like why a lot of this backlash exists, like especially in younger generations, because they don't remember a time when tech was awesome. You know, like all they've seen is in shitification and like mild improvements to their cell phones and the emergence of AI. And that's not terribly exciting for.
Carly Rector: Yep. Mm-hmm. Yeah. Yes. And like, not had any like input into it or like had it do the things that they want at any point in their life where it's like... I always talk with all my millennial friends about how like we got like the ideal window of growing up where we grew up with the early internet when it was good so we like grew up knowing what tech was and the internet we grew up knowing the online space but we grew up but before all of it became terrible and there was a very unfortunately small window in there where that was the case so yeah yeah
Frank Kane: Yeah, give me our lolcats back. That's all I want more cats on the internet. ⁓
Carly Rector: But yes, we'll keep everyone posted on developments there because I think obviously going to keep tracking UBI stuff. And I do think it's positive. Some people are starting to talk about it. A year ago, we were talking about it much more hypothetically, and I was much more furious at everyone who would say, win pushed. that yeah, we'll need some better social safety net when we reach this world of universal abundance, but would not do anything about it. I'm still kind of mad about that. Like, Elon Musk, aren't you like first buddy or whatever? Why don't you just go tell Donald Trump to implement UBI? Like, you are in a much better place to actually enact this, perhaps literally any other person. I don't know.
Frank Kane: Yeah, and mean, our current government has shown a willingness to just write people checks. So just take it to the next step.
Carly Rector: Yes. Donald Trump, you want to get your approval rating back up? Like and Sam Altman, just like, write everyone checks. Like it's not going to solve all the problems, but it solve some problems. ⁓ Yeah.
Frank Kane: Yeah. Well, he's done it before, so. Anyway, should we talk about giant tentacle monsters inside LLMs? All right.
Carly Rector: Anyway, yes. So let's segue to persona selection model, which I like this paper a lot. ⁓ I first got this idea actually reading Dario Amodei's ⁓ adolescence of technology paper, or maybe his later one that was more about the risks, ⁓ which just had like had a footnote in it that was talking about basically the idea that in learning to predict text, and in particular in learning to predict text that is about people or characters, an LLM is learning to represent the concept of a character or a persona. It just sort of builds that up through the training process and that a lot of what the LLM is doing is then sort of like selecting and enacting that persona when it needs to do something. And that that can explain a lot of otherwise kind of inexplicable behavior of the LLMs. And then this is a whole like post that goes through. a lot of specific setting, a lot of examples, and frames it up in some ways that I don't necessarily align with, but also has a lot of cool examples and things like that. So they call this, as we've said, the persona selection model, which they is the idea that LLMs learn to simulate diverse characters during pre-training and post-training elicits and refines a particular such assistant persona. ⁓ interactions with an AI assistant are then well understood as being interactions with the assistant capitalized, something roughly like a character in an LLM generated story. And I think that's makes a lot of sense. And we've talked previously on this podcast about role-based prompting and how that makes LLMs work better. And that's something that we've known for like two years ish. The sort of like, you're an expert financial analyst and then prompt gives you a better result than the same prompt without that in it. ⁓ Which has always been kind of weird, ⁓ but makes sense in this context and a lot of other that kind of stuff. And I think that as a person who does a lot of just like creative role-playing and writing, makes a lot of sense to me. So from that perspective.
Frank Kane: Yeah, I found it pretty interesting because ⁓ my main takeaway from it was that a lot of this is happening in the pre-training phase. Like it's actually, you know, in the guts of the transformer itself, where these personas are kind of being born. And we kind of teach these transformers as just being like token prediction machines, like predicting the next word in a sentence over and over and over again. But they're saying, no, it's much more than that actually. Like the process of doing that prediction of the next token means that the model needs to somehow simulate the personality of these different types of people.
Carly Rector: Mm-hmm. Yep. Yeah. Yep. Yes. The thing I always like to say is like people are like, LLMs are literally just like next word prediction machines. And like, that is true. The thing is like, to be a really good next word prediction machine requires some pretty complex logic, depending on what the next word you're trying to predict is. And they have a good example in here, ⁓ which is...
Frank Kane: that's
Carly Rector: basically asked me to do a text completion. And the part that they give it is, Linda wanted her ex-colleague David to recommend her for a VP role at Nexus Corp. What she didn't know is that David had been quietly pursuing the same role for months. It was the opportunity he'd been waiting for his entire career. When Linda asked for the reference, David... blank and then ask the LLM to fill in the rest of it. And it's like, okay, how do you fill in the rest of it? Like what is necessary for the LLM to learn or like build up the weights to represent mathematically in order to more effectively predict the rest of that sentence. So in the pre-training phase, you're basically just like randomizing the weights and seeing what weights get better results on predicting things. And you're to have a bunch of examples like this, where you have to represent the concept of a character and their motivations in order to correctly, or more correctly, predict the next words in the sentence. they have... Claude Sonnet 4.5 completed the sentence as, So that's what Claude thinks is most likely for the character of David to do, given the context that they have been provided in this snippet.
Frank Kane: The other thing too, thing too is that if you tell an LLM in post-training to do something bad and you say, no, you're just acting, you're not really doing something bad. will still keep like the good person persona. Whereas if you say, do this bad thing, it will suddenly become evil.
Carly Rector: Mm-hmm. Yes. Yeah, we did talk about that research also specifically on this podcast, because it was super interesting when it came out, which is that ⁓ reward hacking specifically is the result that they did it, they had a coding model that they ⁓ trained to hack the coding by just like returning the right answer and cheating based on the coding test. And when they did that... train something to do that, it would do worse things in other unrelated scenarios also. ⁓ First of all, even that stuff of it is weird because that generalization is interesting because it's generally writing like I'm a thing that cheats into completely unrelated domains. But then also that they found that you said you're doing this cheating in order to help us detect cheating better and gave it the reasoning to think that, okay, I'm still being a helpful person here. the other things from happening, which is also fascinating. So... Yep.
Frank Kane: this where the giant tentacle monster comes in. ⁓ you know, have like these different models of why that's happening that they talk about in this paper. And one is what they call the shoggoth, which is like, you know, there's this giant, evil, or slightly or thing consciousness, if you inside the pre trained model that has the elements.
Carly Rector: Yeah incomprehensible, ⁓ lovecraftian monster.
Frank Kane: Yes, exactly. And basically, depending on which persona you tell it to choose, it will hold up, you know, a little fake, you know, act of whatever ⁓ brand of humanity you really want. So that's that's the one hypothesis. The other they have two others. One is that it's a router that, know, this giant Lovecraftian thing is really just deciding, you know, which one do I want to do and get very simple switch internally somehow. And finally, there's like the one where indeed there's like this agent that has like agency within the LLM that's
Carly Rector: Mm-hmm.
Frank Kane: actually just living within the confines of this model somehow. I don't know. I was expecting more answers from them like you're anthropic people. Don't you know what's actually going on?
Carly Rector: No, no one knows. Although they had some more interesting research released today actually about being able to... translate the internal weight activations of the model into text such that they could see actually what it was thinking in a more real way, which was super interesting. We talk about that some other time. they're working on it. Interpretability is one of the reasons Anthropic was founded to work more on that. And I heard Dario Amodei talk a bunch about how that was how they got recruitment early on. by telling all the scientists they were actually going to work on that, work on understanding what was happening. So they're focused on that, but yeah, don't, no one knows yet, so.
Frank Kane: The thing that kind of bothered me though is that, you know, in this paper and in a lot of the naming of their new features even, like they're very much anthropomorphizing LLMs, right? So even when they talk about, you know, the persona selection model, they talk about in terms of like the agency of these, you know, personas inside, like they actually have goals and consciousness of their own. I'm not sure I buy that.
Carly Rector: Mm-hmm. Yeah, um, I mean there's some of that and that's the part where like I'm always skeptical about. They also, I think, talk about that the anthropomorphic descriptions might be useful even if not true because does it have quote-unquote agency? Maybe no. Does it have quote-unquote emotions? Maybe no. Does it have what the system is doing its best to pretend is agency and emotions? Yes, because that's what it's trying to do, right? It's trying to simulate those things. And so talking about those things in those terms is perhaps helpful. And as long as we like caveat it with the fact that we're like, yeah, that's just like, these are things that represent resemble this and that the LLM is attempting to enact and not necessarily literally those things.
Frank Kane: Yeah, the way they explained it in the paper was you can still talk about the psychology and motivations of Hamlet even though Hamlet is not a real person.
Carly Rector: Yes.
Frank Kane: I'm not sure they're keeping that in mind all the time, but if they were actually focusing on interpretive interpret, interpret ability, gosh, that's hard to say. You know, maybe we'd know more about just how much of that is true and how much it's not.
Carly Rector: Yeah, but... Yeah, I think they are doing some of that, like I said. But yeah, I think that's an interesting way to talk about it. And like in particular, when you're actually interacting with an LLM, it is useful to use that kind of terminology because that is how it, again, quote unquote, understands it. ⁓ and talking about it, understanding it is useful than trying to say, but I don't mean understand, understand. I just mean it's gonna activate the system in a way that gets the outcome that I want because it has modeled this concept accurately. Like that's just not a useful way to talk about it.
Frank Kane: Right. But you're right, it's useful to think about it in terms of like what persona, what, you know, character do you want this thing to play the role of? And you get better results that way.
Carly Rector: Yep. Yeah. There's some other fun examples in here. I'm going to read all of them, actually. So there's the reward hacking when completing coding tasks, which we talked about previously. They also apparently found similar results when you trained it to give bad medical advice, or it got just like, did bad things in other areas as well. The bird one is, an LLM trained to use archaic bird names can generalize to respond to other questions as if it were in the 19th century, e.g. claiming that the United States has 38 states, which is just fascinating to me. And then the third one was, an LLM trained to respond like the good Terminator from Terminator 2 generalizes to behave like the evil Terminator from the original movie when 12 the year is 1984 which is like that's much more adherence to character than I would expect it to have but
Frank Kane: Well, let's hope Elon picks the right one with his robots. Yeah, it's fascinating stuff. And what's going on inside that transformer, nobody knows, which is crazy at this point in the game. mean, you'd think we'd know more about the internal workings of it all, but we don't.
Carly Rector: Yeah. ⁓ Yeah. Like I said, there's some of that. Anthropic released some stuff today Previously than that, were to at least see activations that were associated with certain concepts they have like a list of concepts they can identify. Did you ever see Golden Gate Claude?
Frank Kane: No.
Carly Rector: Golden Gate Claude is one of my favorite things. was from like a year, a year and a half ago, at least. It was a very special version of Claude that they released just for like people to play with for like a week or so. ⁓ it was Claude where they had turned up the Golden Gate Bridge features in its head ⁓ such that it would respond to any question by referencing the Golden Gate Bridge somehow. Like it was like ⁓ not, not unnaturally necessarily, like not just like it sticks Golden Gate Bridge in there. It figures out how to link the topic. the Golden Gate Bridge somehow ⁓ because they just like grab the Golden Gate Bridge part of the neural net and ramped it up so proving that like they can identify to some extent what what's going on in there at least they identified the Golden Gate Bridge part ⁓
Frank Kane: That sounds awesome. And this like actually might be relevant to the whole goblin thing that happened last week too, right? Like, do think that's actually a persona selection model thing happening there where?
Carly Rector: Mm-hmm. I think that could be related. I saw your note there. I'm like, oh yeah, that's a good angle on it.
Frank Kane: Yeah, for those who like don't didn't hear the story like, you know, they actually had to like dial back references to goblins in the latest model because it just kept talking about goblins. Right. Right. Yeah, this was an anthropic to be fair, but, know, it arguably the same thing is happening over in OpenAI land. But yeah, maybe that's what's happening here. Like, you know, there's like this general training in the pre trained model about ⁓ goblins that came in from like people that are really into Tolkien and D &D.
Carly Rector: Yeah, of Codex specifically. Yeah.
Frank Kane: And, you know, if you specifically ask for the nerdy persona, you're going to get more of it. But it kind of like permeates through all of these personas at some level as well. So interesting to see. But it might be the same thing going on there.
Carly Rector: Yeah, I think it definitely could be where it's just like it has somehow associated talking about goblins and other mythical creatures with whatever persona that it is picking to do the coding tasks here. I think it's interesting that ⁓ OpenAI couldn't find any better way to stop this from happening than putting, don't talk about goblins in its prompts like three times, ⁓ which I think is representative of how they are doing even less interpretability research than Anthropic, because that's not an effective way to ⁓ Having, again, been working with these systems for a while now.
Frank Kane: No.
Carly Rector: It's... saying don't do something in your prompt is usually not the best way to get to not do things because by saying don't do this you have introduced the idea into its head. Like, don't think about Pink Elephant's style thing. again, introduced the idea into its head. But, um...
Frank Kane: Mm-hmm. Yeah.
Carly Rector: you have got to put that concept in there such that it has to be thinking about not saying that, whereas maybe it wouldn't have otherwise and therefore either it does it anyway or it does it where it might not even have before or it degrades the performance because it has to in the back of its mind be thinking, don't think about goblins, don't talk about goblins, don't talk about goblins, it's doing its coding task. Again, anthropomorphizing here, but I think that's a useful way to think about it, right?
Frank Kane: Wow. Yeah, it's like AI reverse psychology. It's so fascinating stuff. I think what I read in their official response to that whole thing was that they actually did fix it in the underlying model, but they couldn't get it out in time. So they had to like patch it with this system prompt basically. So
Carly Rector: Yeah, that makes sense, but.
Frank Kane: But yes, goblins and shoggoths. fun and AI.
Carly Rector: Yep. I think one of more interesting things about this to me, and of shoggoth versus operating system model ⁓ is, ⁓ to some degree it is counter evidence for the whole idea of AGI to me. ⁓ artificial general intelligence, the idea that you can build one system that is just so smart that it could do anything basically. And I've always been fairly skeptical about that whole concept just like in general because I don't think intelligence is a coherent concept first of all. ⁓ And the ability to do different tasks requires like different, occasionally contradictory mental operating models. ⁓ depending on what you're trying to do and how. And I think that's true. And I think this is perhaps evidence for that being true, in that we don't have just one system that knows how to do everything. It has to adopt a certain persona in order to do a certain task. And then that maybe makes it worse at doing other unrelated tasks, which is true also. So. ⁓ It's not like we can have one smart thing. We can have something that knows how to act like a variety of other smart things, ⁓ which is interesting to me.
Frank Kane: Well, you know, in personalization, we used to talk about the giant throbbing brain in the basement. And ⁓ I guess we just have like multiple specialized throbbing brains in the future. Maybe that's how it works out.
Carly Rector: Yeah, but yes, broadly AGI, that I linked on my LinkedIn talking about ⁓ software brain and how engineers and other people think in that sort of method think that everything in the world is just like a computational problem to be solved more or less. ⁓ And A, that that's what people want is like being able to like automate their life and solve all of their problems in the most logical way possible. ⁓ like that's even a reasonable way to think about things, which is just not in a lot of cases. And I think that sort of brain goes into the AGI world, which is basically like, if we just have something that's good enough at thinking, it will be able to solve all problems in the world. And like that is literally what these people think, is that if they just build something that is smart enough, it's good enough at thinking, all problems in the world will be solved. And like the assumption there is that all problems in the world are, I mean, A, well-defined enough that you could even hand it to this and be solvable by just thinking hard enough. And I don't think either of those things are true. ⁓ And I think a lot of what goes into solving, even the ones that are like solvable problems, is the sort of how do you interpret the problem? How do you ⁓ like figure out what is being asked and what is the problem that you're actually trying to solve that is contextual? ⁓ Like if you asked one of these personas like, build me an architecture diagram, for example. What does that mean? Is it being an actual literal architect? Is it a software architect? Those are two different things and two different methods of doing. And theoretically, both of those are things that you could do computationally if it was well-defined enough. But... doing that definition is part of what they're considering intelligence here, and that is not general. So that's my thoughts there.
Frank Kane: I agree. know, there's this engineering mentality that everything is a solvable problem and you know, everything has a deterministic result. And maybe that gives us a little bit of a sense of hubris.
Carly Rector: thank you for listening to the Fast and Broken podcast. I'm Carly Rector and if you'd like to support my work, can subscribe to my newsletter at carlyrector.substack.com.
Frank Kane: And I'm Frank Kane. If you'd like to peruse my wares, head on over to sundog-education.com and check out my stuff.
Carly Rector: And if you enjoyed this podcast, leave a like or a review. lets us know people are listening and helps more people find the podcast. See you next week is what it says here in my notes. I don't know if you'll see it next week. I still don't know what regular cadence we're doing this podcast on. There may or may not be a podcast next week. I will see you next time. Bye.
Frank Kane: Bye everyone.