00:00:02
Speaker 1: Bloomberg Audio Studios.
00:00:04
Speaker 2: Podcasts.
00:00:05
Speaker 3: Radio.
00:00:06
Speaker 2: News.
00:00:08
Speaker 1: I'm Stephen Carroll and this is Here's Why, where we take one news story and explain it in just a few minutes with our experts here at Bloomberg.
00:00:19
Speaker 2: I'm worried about misuse of AI models. Instead of talking about the probabilities, let's talk about what we can do.
00:00:27
Speaker 1: It's not like we're just rolling the dice. I think there are genuine safety concerns, but I also think there's probably a bit of self-interest involved.
00:00:36
Speaker 3: We need agreement from Republicans and Democrats in Congress to put aside our partisan pre-election bickering and put in place some real guardrails before AI goes off the rails.
00:00:50
Speaker 1: All of a sudden, a lot of people are really worried about AI. Donald Trump is a notable exception.
00:00:57
Speaker 3: They're just playing right into the hands of a lot of people that don't want to see it happen.
00:01:01
Speaker 1: And that could be political people.
00:01:03
Speaker 2: It could also be China.
00:01:05
Speaker 1: And we're not going to let that happen.
00:01:07
Speaker 2: It's a hoax.
00:01:09
Speaker 1: This debate was blown open by a warning from a former anthropic and OpenAI researcher about the risks posed by artificial intelligence. It came after the startling cyber attack by OpenAI's rogue agents on fellow tech firm Hugging Face. The result has been widespread alarm, not just about what AI can do, but also the ability of top developers to control their creations. Here's why and how AI could kill us all. Bloomberg Tech Europe anchor Tom McKenzie joins us now for more. Tom, the researcher Jacob Coxon had warned his former employers, Anthropic and OpenAI were gambling with our lives. in that social media post. What exactly is he warning of, is he worried about? How could this technology actually kill people?
00:01:58
Speaker 2: Well, he and other researchers from other frontier labs, current researchers and those who have recently left, point primarily to the developments around something called recursive self-improvement or RSI, which is essentially the idea that agents, AI agents are getting so good that they will soon be able to do their own AI research and build their own agents. And those agents will be better than them. And then those agents will build other agents. And that will continue. That research has started. And a lot of researchers say that the results are more extreme than many maybe had expected. There hasn't been confirmation that it's exactly going to play out the way I described, but that is theoretically how it could play out. And in that world, there are deep questions then for these researchers about agency and how humans control those agents if they are continuing to build their own agents, and how you understand what is going on under the hood, essentially, something called interpretability. It's still not clear to a lot at the frontier of AI how these models work, and there's still a lot of work being done to try and understand that. And it's the pace of all of this innovation on top of those developments that is leading the a number of prominent researchers to highlight these concerns and very, very real concerns that they are pointing to potentially the very existence of humanity itself.
Speaker 1: Bloomberg Audio Studios.
00:00:04
Speaker 2: Podcasts.
00:00:05
Speaker 3: Radio.
00:00:06
Speaker 2: News.
00:00:08
Speaker 1: I'm Stephen Carroll and this is Here's Why, where we take one news story and explain it in just a few minutes with our experts here at Bloomberg.
00:00:19
Speaker 2: I'm worried about misuse of AI models. Instead of talking about the probabilities, let's talk about what we can do.
00:00:27
Speaker 1: It's not like we're just rolling the dice. I think there are genuine safety concerns, but I also think there's probably a bit of self-interest involved.
00:00:36
Speaker 3: We need agreement from Republicans and Democrats in Congress to put aside our partisan pre-election bickering and put in place some real guardrails before AI goes off the rails.
00:00:50
Speaker 1: All of a sudden, a lot of people are really worried about AI. Donald Trump is a notable exception.
00:00:57
Speaker 3: They're just playing right into the hands of a lot of people that don't want to see it happen.
00:01:01
Speaker 1: And that could be political people.
00:01:03
Speaker 2: It could also be China.
00:01:05
Speaker 1: And we're not going to let that happen.
00:01:07
Speaker 2: It's a hoax.
00:01:09
Speaker 1: This debate was blown open by a warning from a former anthropic and OpenAI researcher about the risks posed by artificial intelligence. It came after the startling cyber attack by OpenAI's rogue agents on fellow tech firm Hugging Face. The result has been widespread alarm, not just about what AI can do, but also the ability of top developers to control their creations. Here's why and how AI could kill us all. Bloomberg Tech Europe anchor Tom McKenzie joins us now for more. Tom, the researcher Jacob Coxon had warned his former employers, Anthropic and OpenAI were gambling with our lives. in that social media post. What exactly is he warning of, is he worried about? How could this technology actually kill people?
00:01:58
Speaker 2: Well, he and other researchers from other frontier labs, current researchers and those who have recently left, point primarily to the developments around something called recursive self-improvement or RSI, which is essentially the idea that agents, AI agents are getting so good that they will soon be able to do their own AI research and build their own agents. And those agents will be better than them. And then those agents will build other agents. And that will continue. That research has started. And a lot of researchers say that the results are more extreme than many maybe had expected. There hasn't been confirmation that it's exactly going to play out the way I described, but that is theoretically how it could play out. And in that world, there are deep questions then for these researchers about agency and how humans control those agents if they are continuing to build their own agents, and how you understand what is going on under the hood, essentially, something called interpretability. It's still not clear to a lot at the frontier of AI how these models work, and there's still a lot of work being done to try and understand that. And it's the pace of all of this innovation on top of those developments that is leading the a number of prominent researchers to highlight these concerns and very, very real concerns that they are pointing to potentially the very existence of humanity itself.
00:03:20
Speaker 1: To think through the causality of this, the thought is that if AI is out of human control, that it's, you know, creating its own, developing its own agents, that it could take control of critical systems or what exactly is the concrete risk if you follow that chain? And as you say, it is a theoretical chain.
00:03:39
Speaker 2: So you highlighted the Hugging Face open AI incident, hacking incident, which stretched from May to July of this year. And what that showed was that agents that are currently active, that they have currently built at these frontier labs, are already capable of breaking out of so-called sandboxes where they're cut off for the internet. They conspired, about 1,200 of these agents conspired with each other to break out of these so-called sandboxes to hack into Hugging Face, which is a depository of AI code. and other things that go into how we use AI, but largely a depository of AI code. They managed to break in. That showed agents coordinating at a level that many had said we've never seen before. And if you put on top of that, this recursive self-improvement where the agents are getting better and better and better, then all sorts of potential scenarios could map out. You could, for example, as you said, get to the point where these agents are running industries and taking those over and that we lose control as they took over the energy system.
00:04:37
Speaker 3: Or.
00:04:38
Speaker 2: you could get to a point where these agents are being used by nefarious actors to, for example, build biological weapons. That's a risk that's been flagged by researchers in AI for a long time. And then there's the question of the nuclear deterrent. And if AI gets embedded or gets into our nuclear systems, what that potentially could mean. So there's all sorts of different scenarios under which agents, training agents, we don't understand to what extent they are aligned with our own aims and human aligned, which means they are working on our best interest. We don't know if they are, because that's still an area of research that is yet to be fully resolved and fully understood. And we don't exactly know what's going on under the hood of these models. That's the interpretability part of it. Those things combined pose this potential very, very significant risk, is what these researchers are suggesting.
00:05:31
Speaker 1: So how seriously are these risks being taken? We've heard a little bit about the alarm that's been raised, but within the industry, outside the industry, how much are these risks being taken seriously?
00:05:42
Speaker 2: Well, they are being taken very seriously to the point where Dario Amadei, the CEO and the founder of Anthropic, one of the leading frontier labs, of course, posted a more than 3,000-word essay over the weekend calling for a three-step approach to pacing, as he described it, pacing the frontier, essentially slowing some of the development of making sure that they have independent auditors essentially inside their own business. They say they're committed to that already. He laid out a number of measures where he wants to see leading frontier and AI labs working together, coordinating, but he also wants a regulatory framework imposed by the government as well to ensure there are those checks and balances. And he wants work at the international level to try to mitigate the risks as he sees them. That essay was supported by Sam Altman of OpenAI.com. But also, Elon Musk seemed to give it his endorsement by saying, Dario is right, and posting that on X. We've had people like Demis Hassabis from Google DeepMind, who has previously put together a plan that he says could help Swage and address some of those risks, a regulatory framework along the lines of FINRA, which is used in the United States, which is a self-regulating body for the world of finance. He says something like that could be appropriate. And shortly after that Hugging Face OpenAI hacking event, that happened and ended in July, about 1,200 researchers from across different frontier labs came out together and wrote an open letter calling for regulation and better safeguards. So it is being taken very seriously, though there is a divide within the AI community and indeed with the Trump administration itself about what the risks really are, how dangerous this technology really could be.
00:07:19
Speaker 1: You mentioned the FINRA model there, but in terms of putting guardrails on technology, it's a debate we've seen play out, for example, in social media as well Is there a model that actually tells us how we can regulate a technology like AI, given its nature and the risks of it going rogue?
00:07:35
Speaker 2: Someone like Mark Zuckerberg from Meta has come out just in the last couple of days and said, what you need is independent evaluators inside the organization to evaluate these models before they're released. He doesn't seem to be making the case for a new regulatory framework. So he seems to be taking a slightly different approach to Dario Modi and Anthropic and others. You could look to different industries, the airline industry, for example. You can't put out Airbus and Boeing can't put aircraft out before they've been rigorously checked and before they have appropriate licenses and credentials and the sign-off. And the same for the car makers as well. And both of those industries started off putting out products that were very dangerous. And obviously, we've evolved and come a long way since then. And there are now regulatory frameworks around both of those industries to mitigate the risks and very effectively. So, some in the AI world and some in this space point to regulatory regimes like that. But again, there is debate because One of the contentions from those within the Trump administration is that this is essentially Anthropic and OpenAI, a duopoly calling for regulations so that they can lock in their market advantage and stop some of their competitors catching up because their competitors will be wrapped up in this regulatory burden. And Anthropic and OpenAI will be able to, because they've got such market share, work through that regulatory burden and have a clear advantage that will be locked in. That is one of the counters to the calls for a government regulatory body or an international regulatory body for this technology.
00:09:06
Speaker 1: You've mentioned some of the previous challenges that we've seen with technology and that we've overcome in something like the car industry where there have been questions of safety. But do we need to rely on the companies here to move us towards a model that's more protected or to have the safeguards in place? Or does this need some sort of overarching government or otherwise regulations?
00:09:28
Speaker 2: Well, that is a question that is getting a lot of traction in the US and some bipartisan traction as well. Both Democrats, independents like Bernie Sanders and Republicans working together to call for greater scrutiny and a regulatory framework. And laws in place, regulatory framework in place to make sure that the risks around this technology are fully accounted for and that this can be safely put out into the world. Others say that the current laws current laws in place, company liability, for example, mean that should an anthropic or an open AI put a model out that proves dangerous and ends up harming people, then they will face litigation. And that in itself will be a curve, that existing laws are enough. And that's something that the NVIDIA boss and CEO, Jensen Huang, that's the case that he has made. There are examples of the industry itself slowing the pace of this innovation and doing it in a way that they would describe as mindful and thoughtful. Anthropic, for example, with its Project Glasswing and one of its most powerful models, only putting that out to a select handful of companies, Mythos model, so that they could check their cyber resilience to this model. And that has still not been widely distributed. So that's an example of one of the frontier labs taking what they would say is a responsible approach to this technology. Mark Zuckerberg says that they have also taken similar measures and slowed the release of some of their models for similar reasons. Sam Altman has also said that they have put in place similar checks and balances to make sure that the models they put out don't pose a risk and don't pose a danger. And certainly there are big teams in all of these frontier labs that are working on that alignment question, making sure the models are do align with our own human interests, though there are still grey areas around that, and putting and baking in guardrails into these models so that, for example, the bad actor can't take them and use them to develop biological weapons. But those measures are by no means conclusive.
00:11:23
Speaker 1: Okay. Well, giving us plenty to watch in this space going forward. Tom McKenzie, anchor of Bloomberg Tech Europe on Bloomberg Television. Thank you very much for joining us. For more explanations like this from our team of 3,000 journalists and analysts around the world, go to bloomberg.com slash explainers. I'm Stephen Carroll. This is Here's Why. I'll be back next week with more. Thanks for listening.