00:00:02
Speaker 1: Bloomberg Audio Studios, podcasts, radio, news.
00:00:10
Speaker 2: Earlier this week, an AI researcher at Anthropic resigned from his job and posted about it on X. He'd spent the last three years working at both Anthropic and OpenAI, he wrote, and neither company is acting responsibly, he said. He said the companies were gambling with our lives. and suggested that people in the AI space thought the technology could eventually kill us all. It was scary stuff, and the post made waves in the AI industry and beyond, at a time when concerns about the risks of AI are mounting.
00:00:41
Speaker 1: Bill Gates says the computer industry that he championed is now a global threat.
00:00:45
Speaker 2: OpenAI's system was able to hack into another AI platform called Hugging Face during a security test. With me today is Bloomberg's Mike Shepard, Senior Editor for Technology and Strategic Industries. Mike, how would you describe this moment for AI? Are we at a turning point?
00:01:02
Speaker 1: You know, we talk about things that go viral. This took off, but it also has some staying power. It didn't just ripple through Silicon Valley. It echoed across Washington and in global capitals, too. So there is this moment of reflection now in this technology that is so much promise to change the economy and change the way we live.
00:01:25
Speaker 2: I'm Sarah Holder, and this is The Big Take from Bloomberg News. Today on the show, the AI safety reckoning is here. Mike, I am curious why this particular post from this particular researcher broke through, though, because AI researchers have been sounding the alarm for a while on these kinds of risks. this person, Jacob Coxon, not a public figure, not someone I'd heard of before. He doesn't have any other tweets on his profile. Why did people pay so much attention to what Jacob Coxon was saying? And what did he actually say about why he left the company and the things that he's so concerned by?
00:02:02
Speaker 1: I've been reflecting on this, too. One, its simplicity. It was very direct. It was personal. He really addressed the audience, not just a tech audience, but really a Main Street audience as well. And he wasn't very preachy in it either. He was issuing a warning, but he kind of left it to the reader, the consumer, the audience to try to make their own decisions. He did issue a warning that in his view, after having worked at both Anthropic, which he just resigned from, and OpenAI, his prior employer, that neither company was acting responsibly in their respective pursuits of what's known as superintelligence. This is the version of artificial intelligence that surpasses human capabilities in most areas and also possesses the ability to improve itself. And he called that pursuit of what's called recursive learning or self-improvement, he cast it as reckless, really, Sarah. And he spelled out in passing the cybersecurity risks that there could be a massive and crippling hacking attack using AI that could you know, really disrupt financial systems and critical infrastructure. There's also the threat of bioweapons, too. So all of this kind of hangs in the background of his message. But then he also issued an admonishment to his fellow Silicon Valley AI workers like, hey, do you really want to be a part of this? Are you sure you want to have a role in in developing this technology in a way that could be so destructive and consequential in a negative way.
00:03:44
Speaker 2: I put that question to you, Mike. Why are people still building it?
00:03:48
Speaker 1: Well, for Anthropic, part of their identity is to pursue AI development responsibly. And they actually see themselves as the company that can achieve this superintelligence or artificial general intelligence in a way that will be safe for humanity. And it's not just that Anthropic, a number of researchers out there are committed to the pursuit of this technology for that reason. They want to get there first so that it's done safely. So they believe in it. Others are a little bit more skeptical about the apocalyptic vision that, you know, some are casting as the backdrop for AI development, that it will kill us all. And they think, hey, look, it's not that bad. We can control it. We can understand it. And most AI will actually be used in much smaller and refined applications.
00:04:42
Speaker 2: I want to drill into this idea that AI will kill us all because, you know, Jacob's former colleague and alignment science lead at Anthropic reposted Jacob Coxon's post and said that, quote, we really do earnestly believe AI could kill all humans, unquote. And he gave it a more than 10 percent chance within the next decade. That's like a pretty remarkable thing for someone who currently works at Anthropic. to say, right?
00:05:06
Speaker 1: It is. It's striking to hear this conversation. And yet it's one that's been kind of going on in the background, a little bit out of sight from, you know, the people in Washington who would be making artificial intelligence policy and out of sight of the folks on Wall Street who are investing so much in this technology, raising billions of dollars in capital on behalf of the hyperscalers, the cloud companies that are investing in A.I., And then on behalf of, of course, Anthropic and OpenAI, which are preparing these massive IPOs. And then the general public, too, which are more concerned about AI's impact on their utility bills than potentially, you know, life on Earth as we know it. And yet at the same time, it is kind of a recurring theme within that community in Silicon Valley.
00:05:59
Speaker 2: So I want to talk about sort of the context that this X thread came out in. Because just a few weeks ago, we learned about the Hugging Face hacking incident, which Jacob pointed to as a warning shot. I'm wondering, Mike, could you just tell us a little bit more about the Hugging Face incident and why this was such a big deal in the AI space?
00:06:20
Speaker 1: We've known for a long time that AI could be used as a tool to try to break into computer systems around the world. if put in the wrong hands, certain AI tools could be incredibly destructive. And that's what led Anthropic to withhold largely the release of its new mythos model back in April. The difference in this Hugging Face episode, though, was that the models themselves engineered this attack on the Hugging Face repository of AI models and other data connected to AI. In this particular incident, a group of models and agents that were being developed by OpenAI, they had been placed in a secure testing environment known as a sandbox and had their guardrails removed so that researchers could test their capabilities. Now, while the agents and models were inside this so-called sandbox, they were given a task, a very difficult cyber problem to solve. And the agents and models began collaborating together. The researchers didn't have great visibility into all the things that they were doing. And the models eventually figured out, hey, maybe we need to get out onto the internet. And that would mean breaking out of this so-called secure sandbox. And if that presented them the path to the solution, great. It didn't matter. In other words, they didn't see it as an impediment.
00:07:54
Speaker 2: They were willing to break the so-called rules of their environment in order to win the game.
00:07:59
Speaker 1: Exactly. Exactly. And ultimately, it led the agents to go find a whole bunch of models that they could access and data that they could access at Hugging Face. Without Hugging Face's knowledge or authorization, and really out of the view of researchers at OpenAI.
00:08:17
Speaker 2: Well, it's funny, Mike, there's been this sense that emphasizing or even overstating AI's capabilities, even scary ones, can be somewhat of a marketing win for these companies. It's just another way of demonstrating just how powerful AI can be helping them justify this massive spending on this technology. But are we at a point where that is backfiring? Are these risks and possibilities getting too real?
00:08:44
Speaker 1: Ducking Face episode does make it feel more tangible, certainly. And you do see more of these arguments taking shape and becoming more coherent on the risk side. And Remember, they're also being paired with all the fear and loathing about data centers and their impact. And yet at the same time, the counter argument to regulation has been that, look, you are fear mongering because you want to pump up the value of your products as you head into a massive and what could be a trillion dollar valuation for an initial public offering. That is a claim or an argument that I've heard from a number of people in the tech industry and in government who oppose much heavier or stricter regulation on artificial intelligence. They do see those kinds of measures as restraining innovation. And they also see that some of the doomerism talk could be aimed at just drawing attention to and adding to the allure of these products and models that the AI companies are developing. We talked about mythos. There was so much demand for it after Anthropic said, hey, only a few are allowed to have it. It does create the aura of forbidden fruit. And then all of a sudden, everybody wants access to it.
00:10:06
Speaker 2: Coming up, what AI safety guardrails could look like in practice and how competition with China complicates the politics of slowing down. Mike, what are the kinds of guardrails that experts say could limit some of the worst outcomes of AI superintelligence or sort of the recursive learning that Jacob Coxon was talking about?
00:10:36
Speaker 1: Well, some of them would be testing by the government or more robust reviews of these models before they go out. And the companies have been trying to do this. But one of the challenges with this technology is that even if we're not fully at this moment of self-trained models, a lot of this technology is developed almost in a Petri dish. It can kind of run itself. It can do some of the coding itself. And with that, you lose a little bit less visibility into the process and into what went into it. In other words, you and I may want a model to do something, but it ends up deciding on its own, hey, I think I'm going to go in a different direction. It's called the alignment problem. And even if they put guardrails in place, how will we know that they are fully effective? And lawmakers here haven't really gotten their heads around it completely. Bernie Sanders introduced recently this measure that would call for a ban on developing superintelligence altogether. And we've seen others in Congress call for a kill switch on AI. I'm not exactly sure how that would work, but it would, in essence, force the companies to have something available that they could pull the plug on a runaway model if need be, or a runaway negative tech event if need be. How that will work in practice, we are probably a long way of seeing how that would develop. It's important to remember that we are heading into the November midterms. We have a very narrowly divided Congress. And we have an administration that is naturally averse to regulation, and especially when it comes to artificial intelligence.
00:12:24
Speaker 2: I mean, another argument against more aggressive AI regulation has been, you know, the U.S. needs to keep advancing its AI models to keep up with China. But I was listening to the Radio Atlantic podcast and listening to Bill Gates's talk to the interviewer there. And he's among those who have made the point that having AI run wild is not in China's interest either. So I am wondering sort of about the global consensus around reining in some of these risks and whether the argument on the sort of China competitiveness side is starting to shift as well. The U.S.
00:12:56
Speaker 1: Certainly has the national security considerations that they do not want to see China gaining edge in AI for fear that it could be deployed against the U.S. eventually in a military conflict or even in a less kinetic way in terms of cyber attacks and other things that could disrupt the American economy and U.S. security. So this rivalry with China in AI is not only top of mind, but when President Donald Trump is asked publicly about safety and about whether data centers should be slowed a little bit to reflect local concerns about their impact. His argument is that we can't slow down. We need to stay ahead of China. And we also heard Treasury Secretary Scott Besson saying this week that if we don't stay ahead of China in AI, it's essentially game over. A lot of this will come to a head when Presidents Donald Trump and Xi Jinping meet here in Washington in a couple of weeks. And AI and certainly safety will be on the agenda for those talks.
00:14:00
Speaker 2: Well, there are those within the AI industry in the U.S., I'd count, you know, the co-founder of Anthropic, Dario Amodai, among them, who are saying, we want the government to regulate us. They need to step in. We can't regulate ourselves. But what can companies actually do without action from Congress? And what are they actually willing to do themselves in terms of putting up these guardrails and slowing down on some of the innovations?
00:14:26
Speaker 1: Well, one thing that companies could do is perhaps slow the pace of trying to reach artificial general intelligence or super intelligence. That has been a stated goal of both Anthropic and OpenAI that they want to get to AGI soon and really try to win the race there. And instead focusing on more bespoke and refined direct applications of the technology. When it comes to physical applications like robotics and self-driving cars, maybe put more of the energy toward that rather than this utopian goal of, you know, an all-powerful model.
00:15:09
Speaker 2: This is The Big Take from Bloomberg News. I'm Sarah Holder. To get more from The Big Take and unlimited access to all of Bloomberg.com, Subscribe today at Bloomberg.com slash podcast offer. If you liked this episode, make sure to follow and review The Big Take wherever you listen to podcasts. It helps people find the show. Thanks for listening. We'll be back tomorrow.
Speaker 1: Bloomberg Audio Studios, podcasts, radio, news.
00:00:10
Speaker 2: Earlier this week, an AI researcher at Anthropic resigned from his job and posted about it on X. He'd spent the last three years working at both Anthropic and OpenAI, he wrote, and neither company is acting responsibly, he said. He said the companies were gambling with our lives. and suggested that people in the AI space thought the technology could eventually kill us all. It was scary stuff, and the post made waves in the AI industry and beyond, at a time when concerns about the risks of AI are mounting.
00:00:41
Speaker 1: Bill Gates says the computer industry that he championed is now a global threat.
00:00:45
Speaker 2: OpenAI's system was able to hack into another AI platform called Hugging Face during a security test. With me today is Bloomberg's Mike Shepard, Senior Editor for Technology and Strategic Industries. Mike, how would you describe this moment for AI? Are we at a turning point?
00:01:02
Speaker 1: You know, we talk about things that go viral. This took off, but it also has some staying power. It didn't just ripple through Silicon Valley. It echoed across Washington and in global capitals, too. So there is this moment of reflection now in this technology that is so much promise to change the economy and change the way we live.
00:01:25
Speaker 2: I'm Sarah Holder, and this is The Big Take from Bloomberg News. Today on the show, the AI safety reckoning is here. Mike, I am curious why this particular post from this particular researcher broke through, though, because AI researchers have been sounding the alarm for a while on these kinds of risks. this person, Jacob Coxon, not a public figure, not someone I'd heard of before. He doesn't have any other tweets on his profile. Why did people pay so much attention to what Jacob Coxon was saying? And what did he actually say about why he left the company and the things that he's so concerned by?
00:02:02
Speaker 1: I've been reflecting on this, too. One, its simplicity. It was very direct. It was personal. He really addressed the audience, not just a tech audience, but really a Main Street audience as well. And he wasn't very preachy in it either. He was issuing a warning, but he kind of left it to the reader, the consumer, the audience to try to make their own decisions. He did issue a warning that in his view, after having worked at both Anthropic, which he just resigned from, and OpenAI, his prior employer, that neither company was acting responsibly in their respective pursuits of what's known as superintelligence. This is the version of artificial intelligence that surpasses human capabilities in most areas and also possesses the ability to improve itself. And he called that pursuit of what's called recursive learning or self-improvement, he cast it as reckless, really, Sarah. And he spelled out in passing the cybersecurity risks that there could be a massive and crippling hacking attack using AI that could you know, really disrupt financial systems and critical infrastructure. There's also the threat of bioweapons, too. So all of this kind of hangs in the background of his message. But then he also issued an admonishment to his fellow Silicon Valley AI workers like, hey, do you really want to be a part of this? Are you sure you want to have a role in in developing this technology in a way that could be so destructive and consequential in a negative way.
00:03:44
Speaker 2: I put that question to you, Mike. Why are people still building it?
00:03:48
Speaker 1: Well, for Anthropic, part of their identity is to pursue AI development responsibly. And they actually see themselves as the company that can achieve this superintelligence or artificial general intelligence in a way that will be safe for humanity. And it's not just that Anthropic, a number of researchers out there are committed to the pursuit of this technology for that reason. They want to get there first so that it's done safely. So they believe in it. Others are a little bit more skeptical about the apocalyptic vision that, you know, some are casting as the backdrop for AI development, that it will kill us all. And they think, hey, look, it's not that bad. We can control it. We can understand it. And most AI will actually be used in much smaller and refined applications.
00:04:42
Speaker 2: I want to drill into this idea that AI will kill us all because, you know, Jacob's former colleague and alignment science lead at Anthropic reposted Jacob Coxon's post and said that, quote, we really do earnestly believe AI could kill all humans, unquote. And he gave it a more than 10 percent chance within the next decade. That's like a pretty remarkable thing for someone who currently works at Anthropic. to say, right?
00:05:06
Speaker 1: It is. It's striking to hear this conversation. And yet it's one that's been kind of going on in the background, a little bit out of sight from, you know, the people in Washington who would be making artificial intelligence policy and out of sight of the folks on Wall Street who are investing so much in this technology, raising billions of dollars in capital on behalf of the hyperscalers, the cloud companies that are investing in A.I., And then on behalf of, of course, Anthropic and OpenAI, which are preparing these massive IPOs. And then the general public, too, which are more concerned about AI's impact on their utility bills than potentially, you know, life on Earth as we know it. And yet at the same time, it is kind of a recurring theme within that community in Silicon Valley.
00:05:59
Speaker 2: So I want to talk about sort of the context that this X thread came out in. Because just a few weeks ago, we learned about the Hugging Face hacking incident, which Jacob pointed to as a warning shot. I'm wondering, Mike, could you just tell us a little bit more about the Hugging Face incident and why this was such a big deal in the AI space?
00:06:20
Speaker 1: We've known for a long time that AI could be used as a tool to try to break into computer systems around the world. if put in the wrong hands, certain AI tools could be incredibly destructive. And that's what led Anthropic to withhold largely the release of its new mythos model back in April. The difference in this Hugging Face episode, though, was that the models themselves engineered this attack on the Hugging Face repository of AI models and other data connected to AI. In this particular incident, a group of models and agents that were being developed by OpenAI, they had been placed in a secure testing environment known as a sandbox and had their guardrails removed so that researchers could test their capabilities. Now, while the agents and models were inside this so-called sandbox, they were given a task, a very difficult cyber problem to solve. And the agents and models began collaborating together. The researchers didn't have great visibility into all the things that they were doing. And the models eventually figured out, hey, maybe we need to get out onto the internet. And that would mean breaking out of this so-called secure sandbox. And if that presented them the path to the solution, great. It didn't matter. In other words, they didn't see it as an impediment.
00:07:54
Speaker 2: They were willing to break the so-called rules of their environment in order to win the game.
00:07:59
Speaker 1: Exactly. Exactly. And ultimately, it led the agents to go find a whole bunch of models that they could access and data that they could access at Hugging Face. Without Hugging Face's knowledge or authorization, and really out of the view of researchers at OpenAI.
00:08:17
Speaker 2: Well, it's funny, Mike, there's been this sense that emphasizing or even overstating AI's capabilities, even scary ones, can be somewhat of a marketing win for these companies. It's just another way of demonstrating just how powerful AI can be helping them justify this massive spending on this technology. But are we at a point where that is backfiring? Are these risks and possibilities getting too real?
00:08:44
Speaker 1: Ducking Face episode does make it feel more tangible, certainly. And you do see more of these arguments taking shape and becoming more coherent on the risk side. And Remember, they're also being paired with all the fear and loathing about data centers and their impact. And yet at the same time, the counter argument to regulation has been that, look, you are fear mongering because you want to pump up the value of your products as you head into a massive and what could be a trillion dollar valuation for an initial public offering. That is a claim or an argument that I've heard from a number of people in the tech industry and in government who oppose much heavier or stricter regulation on artificial intelligence. They do see those kinds of measures as restraining innovation. And they also see that some of the doomerism talk could be aimed at just drawing attention to and adding to the allure of these products and models that the AI companies are developing. We talked about mythos. There was so much demand for it after Anthropic said, hey, only a few are allowed to have it. It does create the aura of forbidden fruit. And then all of a sudden, everybody wants access to it.
00:10:06
Speaker 2: Coming up, what AI safety guardrails could look like in practice and how competition with China complicates the politics of slowing down. Mike, what are the kinds of guardrails that experts say could limit some of the worst outcomes of AI superintelligence or sort of the recursive learning that Jacob Coxon was talking about?
00:10:36
Speaker 1: Well, some of them would be testing by the government or more robust reviews of these models before they go out. And the companies have been trying to do this. But one of the challenges with this technology is that even if we're not fully at this moment of self-trained models, a lot of this technology is developed almost in a Petri dish. It can kind of run itself. It can do some of the coding itself. And with that, you lose a little bit less visibility into the process and into what went into it. In other words, you and I may want a model to do something, but it ends up deciding on its own, hey, I think I'm going to go in a different direction. It's called the alignment problem. And even if they put guardrails in place, how will we know that they are fully effective? And lawmakers here haven't really gotten their heads around it completely. Bernie Sanders introduced recently this measure that would call for a ban on developing superintelligence altogether. And we've seen others in Congress call for a kill switch on AI. I'm not exactly sure how that would work, but it would, in essence, force the companies to have something available that they could pull the plug on a runaway model if need be, or a runaway negative tech event if need be. How that will work in practice, we are probably a long way of seeing how that would develop. It's important to remember that we are heading into the November midterms. We have a very narrowly divided Congress. And we have an administration that is naturally averse to regulation, and especially when it comes to artificial intelligence.
00:12:24
Speaker 2: I mean, another argument against more aggressive AI regulation has been, you know, the U.S. needs to keep advancing its AI models to keep up with China. But I was listening to the Radio Atlantic podcast and listening to Bill Gates's talk to the interviewer there. And he's among those who have made the point that having AI run wild is not in China's interest either. So I am wondering sort of about the global consensus around reining in some of these risks and whether the argument on the sort of China competitiveness side is starting to shift as well. The U.S.
00:12:56
Speaker 1: Certainly has the national security considerations that they do not want to see China gaining edge in AI for fear that it could be deployed against the U.S. eventually in a military conflict or even in a less kinetic way in terms of cyber attacks and other things that could disrupt the American economy and U.S. security. So this rivalry with China in AI is not only top of mind, but when President Donald Trump is asked publicly about safety and about whether data centers should be slowed a little bit to reflect local concerns about their impact. His argument is that we can't slow down. We need to stay ahead of China. And we also heard Treasury Secretary Scott Besson saying this week that if we don't stay ahead of China in AI, it's essentially game over. A lot of this will come to a head when Presidents Donald Trump and Xi Jinping meet here in Washington in a couple of weeks. And AI and certainly safety will be on the agenda for those talks.
00:14:00
Speaker 2: Well, there are those within the AI industry in the U.S., I'd count, you know, the co-founder of Anthropic, Dario Amodai, among them, who are saying, we want the government to regulate us. They need to step in. We can't regulate ourselves. But what can companies actually do without action from Congress? And what are they actually willing to do themselves in terms of putting up these guardrails and slowing down on some of the innovations?
00:14:26
Speaker 1: Well, one thing that companies could do is perhaps slow the pace of trying to reach artificial general intelligence or super intelligence. That has been a stated goal of both Anthropic and OpenAI that they want to get to AGI soon and really try to win the race there. And instead focusing on more bespoke and refined direct applications of the technology. When it comes to physical applications like robotics and self-driving cars, maybe put more of the energy toward that rather than this utopian goal of, you know, an all-powerful model.
00:15:09
Speaker 2: This is The Big Take from Bloomberg News. I'm Sarah Holder. To get more from The Big Take and unlimited access to all of Bloomberg.com, Subscribe today at Bloomberg.com slash podcast offer. If you liked this episode, make sure to follow and review The Big Take wherever you listen to podcasts. It helps people find the show. Thanks for listening. We'll be back tomorrow.
