00:00:02
Speaker 1: Bloomberg Audio Studios. Podcasts.
00:00:05
Speaker 2: Radio.
00:00:06
Speaker 3: News.
00:00:18
Speaker 4: Hello and welcome to another episode of the Oddbots Podcast. I'm Tracey Alloway.
00:00:22
Speaker 1: And I'm Joe Weisenthal.
00:00:23
Speaker 4: Joe, it's October. We're recording this on October 29th.
00:00:27
Speaker 5: Yeah.
00:00:27
Speaker 4: It is also Black October. Have you heard this term?
00:00:30
Speaker 5: Yeah.
00:00:31
Speaker 1: What does that mean? Have stocks gone down?
00:00:33
Speaker 2: No.
00:00:34
Speaker 4: Your mind immediately goes to markets. There are other worlds out there. We're going to be talking about the world of cybersecurity today because one of the reasons people are talking about Black October is we've had a number of recent security issues, the most prominent of which has been a bunch of vulnerabilities discovered in PHP, which is a coding language that apparently is used by 70% of websites in existence. We can get into that stat later. But the really interesting thing about this vulnerability being uncovered is AI was used to uncover it. And so the big question coming back to AI, there's a concern now that you have good actors who are maybe using AI to discover old vulnerabilities and trying to fix them. But you could also have bad actors who are using AI to exploit those same vulnerabilities. So we're back to the question of whether AI makes the internet safer or more dangerous.
00:01:29
Speaker 1: It makes it more dangerous. No, I have no idea. Ever since essentially Anthropic announced that they had this model called Mythos and now it's Project Glasswing, which was rolled out to companies to use for defensive purposes before it was available to the general public, there has been much more awareness of cybersecurity issues as it relates to AI. And there are many things that people debate about AI and some of these bigger existential questions. Pretty much everyone agrees cybersecurity good, more cybersecurity good. I will say I know very little about the world of cybersecurity. I don't know what, for example, a cybersecurity company actually sells. I don't know how if I were buying cybersecurity, I would know whether it's working or whether no one had tried to hack me or not or something like that. I need to learn a lot more, but it's absolutely true about the uptick in the volume of hacks.
00:02:19
Speaker 4: There are hackers out there right now taking notes going, Joe doesn't know anything about cybersecurity, probably doesn't have a VPN.
00:02:25
Speaker 5: I don't.
00:02:26
Speaker 1: That's why my vibe coding projects, I don't want to ever, I have yet to build anything with an account because I don't want to like have to think about that. So it's just like, I don't want to count information with anyone. But I have as a user of technical products, over the last several weeks, you get more emails saying like, Our thing was compromised. You're probably fine, but maybe you should refresh some keys or whatever. This is happening. This is real. And everyone really needs to learn more. And if you think about like black swans and Wall Street and all that stuff, you know, there's been for years a lot of talk about cyber as a major risk out there.
00:03:02
Speaker 2: Yeah.
00:03:03
Speaker 4: All right, well, we do, in fact, have the perfect guest. We're going to be speaking to two people from the company Calif, one of the companies that actually uncovered this PHP vulnerability. The other one was a company called Wiz. We have Tai Yun. He is the CEO over at Calif, which is a hacking firm, and we can get into exactly what that means, as well as Bruce Dang. He is an engineer at Calif as well. So, Tai and Bruce, thank you so much for coming on All Thoughts.
00:03:30
Speaker 2: We're glad to be here. So.
00:03:32
Speaker 4: Just to begin with, because hackers kind of, they always have cool achievements that they like to talk about. Why don't we just go ahead and start with who you are and what are the coolest things you've done in your hacking career?
00:03:45
Speaker 3: Yeah. So I started my career at Microsoft almost 20 years ago in the Microsoft Security Response Center. These are the people that do patch Tuesdays. On the second Tuesday of every month, Microsoft releases an update. So I worked in that team a long time ago. Probably the biggest thing I did there and that I could talk about is I helped reverse engineer the Stuxnet virus, which is one of the one that was related to the supposedly the Iranian nuclear program was used to sabotage the Iranian nuclear program.
00:04:18
Speaker 4: This is the thing that people describe it as the world's first like known digital cyber weapon, right?
00:04:24
Speaker 2: Yeah, yeah, yeah, yeah.
00:04:26
Speaker 3: I'm sure there were many others before, but yeah, that's the first public one, yeah. I worked there for almost a decade. And then I left to do a startup with my friends. And then I joined Apple. I worked on iOS security. And then Ty managed to convince me to follow the Church of AI.
00:04:49
Speaker 2: And so here I am.
00:04:50
Speaker 5: Yeah, it took me four years to convince Bruce. So, you know, when you go to a website or you go to Amazon, even Facebook, Instagram, your computer will share information with the server, right? Yeah. And that would be, the information would travel through many, many other servers, like your ISP, your Wi-Fi routers. So you want to encrypt that information to make sure that only the server can see your data, nobody else. And there's a mechanism for that. It's called the SSL protocol, or the later version is called like TOS, like transport layer protocol. security protocol. So, um, I love those protocols and, um, back in the day, together with, uh, friends, we found a bunch of ways to break the protocols, you know, to decrypt the data. And, um, it's pretty cool. It's actually changed the internet, how the internet works, because we basically, um, showed that in the past that like two protocols, like SSL Versus 3.0, that was invented by Netscape. You remember Netscape? Yeah, yeah. 30 years ago. And then people still use it because it's still working somewhat. But we show that, oh, it's not working very correctly. If you use SSL 3.0, then people can decrypt your data. So we hope the internet really, really removes SSL 3.0. And that's one thing. The other thing that I really enjoy doing was, you know, people think that in the past you have compression, you have some data and you want to compress the data and you have encryption. And both of them actually randomize your data in some way, right? If you look at a zip file, you open a zip file, look at the raw binaries, buy data in the zip file, look random to you. Because... all the redundant information has been removed and encoded in a way to reduce the size of the file. And so one of the basic question in computer security was, would you compress data and then you encrypt or you encrypt then compress? And I took some crypto class at Stanford University and I remember my professor, he asked this question in one of the homework assignments. And the obvious answer back then was you don't want to compress and then you don't want to encrypt and compress because encryption randomize your data. It means that compression would be less effective. So the right answer is you compress and then you encrypt. And that was the correct answer on my homework assignment. I think I took that class in 2011. And then three years later, we found that it's wrong. If you do that, then people can actually decrypt your data. And the way it works is pretty cool. I can explain to you. So compression actually reveals, if you look at a compressed file, you know how well it is compressed, right? Because it's softer. If it's smaller, the compression is very good because there's a lot of redundant information. That's why compression works well. It means that if you look at a compression alone, the size of the compression, a compressed file, you can tell whether the file has a lot of redundant information. And encryption hides the length. So it doesn't hide the size of the encrypted data. When you encrypt a message, you can still see that this message is maybe this long or this message may be this long, right? Because encryption doesn't try to hide that. There's some people try to do that. So it means that if you compress and then you encrypt, it means that I can see where your message has some redundant information, has a lot of redundant information. And that alone is a leakage already, right? That you don't want to review. And I gave you an example. Let's say that you take a message, like a random message that I gave you, and then you put your password together with that message. Let's just say that there's a reason for you to do that. You put your password together with the message I gave you and you compress together. And then you encrypt it. with a key that I don't know. And you reveal the ciphertext, the result encrypted message to me. So supposedly I don't have the key. I shouldn't be able to read the message, right? But because I give you a message and you get my message put together with your password, you compress together. If the compression works very well, then I know whether my message has some redundancy with your password.
00:09:46
Speaker 4: You can reverse engineer between the two relationships.
00:09:49
Speaker 5: Yeah, exactly. And you do that repeatedly. Eventually, we can recover your password. And that's exactly what we found on the web encryption protocols. In the past, that's exactly what they did. They take my data, they concatenate your password, they compress together, and then they send over the wire. And then as an attacker, I can see the length And that's enough for me to decrypt. And the solution was really interesting, because they decided to turn off compression. There's no solution. And then eventually, I was at Google back then. I was born in Saigon, Vietnam. I grew up there. Then I got a job in Mountain View for a consultancy, like what we do. So I moved to Mountain View in 2011. And then I joined Google. later that year. And I did cryptography at Google, basically help people do cryptography correctly, like try to avoid the mistakes that we found. I was there for 12 years. I work on like very last number of products. Google has a lot of products, right? Gmail, Chrome browser that we are using right now. And there's a lot of people out there that want to hack Google and our products.
00:11:17
Speaker 2: Yeah.
00:11:17
Speaker 5: So my job was to hack these products before we released them to the public. And I was there for 12 years and then I left to create Calif. And yeah.
00:11:30
Speaker 1: That's fantastic. I'm already fascinated by this conversation. I want to get into obviously what Calif does specifically. I want to get into some of the latest vulnerabilities. But before we do that, you know, both of you have had long careers that by and large predate what we now call AI. Are the risks associated with AI a sort of continuation on some path that cybersecurity have well understood, or are we in a truly novel area for how we should we be thinking about cybersecurity risk?
00:12:07
Speaker 3: I think it's a combination of the two. We've always known that there are security vulnerabilities in systems. And so we have security engineers who are hackers.
00:12:17
Speaker 2: Who try to find and exploit these vulnerabilities.
00:12:20
Speaker 3: So the system has always been vulnerable to some degree, to a varying degree. What AI has done is it's kind of accelerated the speed in which you identify and exploit these vulnerabilities to a level where it's almost like superhuman. You know, like 20 years ago, Microsoft would release, let's say, maybe 20 updates every month. Now you're looking at like, you know, 1,400 updates per month. And that number is going to continue to grow. Just to give you an idea of the scale of things. And now, for example, to have certain systems, it may take you, let's say, three months with lots of people. Now, that may take you a few days with just one person and a very capable AI model.
00:13:08
Speaker 2: That's the reality that we're in today.
00:13:09
Speaker 3: It even changed the way how Attackers have certain types of systems, like in the banking sector, that our customers are in. You can tell some stories about that time. But that's kind of what I think. I think it's the systems have always been vulnerable to some degree.
00:13:26
Speaker 2: And now it's the AI just... accelerates that and makes it much more visible.
00:13:32
Speaker 3: And it's also made it easier for people to identify these vulnerabilities as well.
00:13:37
Speaker 2: And so, and then, you know, obviously there are pros and cons to that. But yeah, you can go into that later.
00:13:58
Speaker 4: Why don't we talk about discovering the PHP vulnerability then? Because I think this was a joint project between Calif, Wiz, and I think it was OpenAI. How did you find this? What was the process? And then what was the decision? I know you guys have gotten a little bit of pushback about publicizing it before it was fixed. So what was the process of actually putting out the statement saying we found this thing?
00:14:24
Speaker 5: We're really interested in finding vulnerabilities that have wide impact. And we want to understand these vulnerabilities to help inform decisions, defense decisions, how to fix and how to mitigate them for once and forever. On that background, we joined the OpenAI's Pass the Planet program. This is where OpenAI funds companies like Calif to go out and help open source maintainers to identify and fix, mitigate vulnerabilities in their software. And we have this engineer, like Ryan Osteen, who is, he's like a PHP historian. Yesterday, we went to a conference in San Francisco. We basically presented like, you know, PHP 2 shell, and Ryan walked everybody over like the 25 years of history of PHP vulnerabilities. So we have this engineer, Ryan, and this is how we work, basically. Our engineer, they're interested in certain things, and then they just go and hack those things. And, you know, as business people, we try to see, okay, how can we convince some other people to pay us for this work? And that's exactly what we did with OpenAI. We convinced OpenAI, okay, we're going to hack PHP. Can you fund our work? And they said, yes, they gave us tokens and money. And then we go and convince like PHP Foundation, which is the organization behind PHP, that let's work together, you know, we can help you. And it's actually pretty interesting because I mentioned like they're like a history of 25 years of like PHP vulnerabilities. And the person that is behind all of these hacks is like Stephen Escher. He's also at Calif. He joined us, I think, six months ago. Back in the day, he's the PHP security guy. Basically, he hacked and helped, like, harden PHP. So we didn't know anybody at the PHP Foundation. So we asked Stefan to introduce us. And because, you know, even though Stefan stopped working on PHP security for a while now, his name still rings, you know, very... loudly in this circle of the world.
00:16:50
Speaker 4: I saw the comic that you guys put up about this. It was kind of funny.
00:16:55
Speaker 3: Yes.
00:16:56
Speaker 4: He's like Indiana Jones type character.
00:16:59
Speaker 5: Yeah. So he connected us and then we met with Volker Dusch at the PHP Foundation. And yeah, that's how we got started. I think we started in July 30. And then immediately after one or two days, you know, Ryan found the first vulnerability, and he was interested in finding a vulnerability similar to one vulnerability that Stefan disclosed 22 years ago. And to his surprise, there's still a bunch of them like that in the PHP software right now. And so that's how we found it. And then we reported it to the PHP Foundation. because we know immediately that is a critical vulnerability that could affect a large part of the internet. Then the Peace Preach Foundation actually shared with us that they actually found this vulnerability, the same vulnerability, two years ago. But they didn't fix it. Because when they found it, they discovered that the fix is very expensive. And there's just maybe a few options to fix it. and none of them were actually perfect because they need to change a lot of functionalities within PHP. And those changes may introduce new bugs, new vulnerabilities, or new reliability issue. And people would be very upset if they upgrade PHP and their website stop working. So in the end, back in like 2024, The PHP Foundation decided not to fix it because there was no exploit. No one could manage to show that this is actually a real issue. And that's exactly what they asked us. They come back, they ask like, do you have an exploit? Then the next day we share with them the first exploit that showed them that, yeah, someone doesn't need to know anything. can hack your website remotely.
00:19:13
Speaker 4: So just to be clear, the use of AI in all of this, it wasn't necessarily that important in finding the vulnerability. It was more important in developing an actual exploit for a known issue.
00:19:27
Speaker 5: We think like that's probably the right way to think about it. But I think even for finding the vulnerability without AI, it would take a lot more time, right? So you can imagine like PHP was created like maybe 30 years ago. So it's a huge code base with a lot of features. And as a human engineer, you have to dig into this code base and it will take like months, right? And along the way, people may give up. But the fact that PHP Foundation filed the bug in 2024 proved that humans can actually file the bugs, right? And the problem was the exploitation, like you said. And for exploitation, like AI plays a major role in it. It's like becomes an engineering problem where our engineers would tell AI, you know, this is how I want to design our exploit and you go and do it. And then, you know, prove to us that it works correctly. Yeah.
00:20:31
Speaker 1: Let me ask a philosophical question. Is there a reason to believe that defensive cybersecurity can stay ahead of offensive cybersecurity, just generally, and then we can get into some details. But philosophically, is that something that we can reasonably hope for?
00:20:49
Speaker 2: I think so, yes.
00:20:50
Speaker 1: How come?
00:20:51
Speaker 3: Because you can use the AI to build certain types of systems in ways that may be very challenging. for attackers to exploit. Today, you can use AI to build whatever systems you want very quickly. You can experiment, get feedback very quickly. It can also process data very, very quickly and reason about that data.
00:21:19
Speaker 2: And so I think it is possible.
00:21:22
Speaker 3: I think it may not be immediate, but over time, we're going to perhaps use AI to help us write more secure software or build more secure systems. And eventually, I think that should counterbalance the advantages.
00:21:40
Speaker 2: That the attackers have. Perhaps I'm being optimistic, but I do foresee this happening.
00:21:47
Speaker 5: I can add something there. So if you look at the history of humanity, our history is the history of like weaponize our tools, right? If you find a powerful tool, then someone would find a way to use that tool as a weapon.
00:22:03
Speaker 4: Yeah.
00:22:04
Speaker 5: So, and attackers always are early adopters of tools, right? New technology. Defenders move like at much slower pace than attackers. But fundamentally, we believe there are more good guys than bad guys out there. And eventually, the good guys would catch up and build things that help everybody defend off the bad guys. And I don't think in general, cybersecurity, if you see the news all the time, you see we got data breach every day almost, even before AI. So you may feel like it's not working at all. But if you're in the field, you can see the progress, right? In the past, like, 25 years, there's a tremendous progress in cybersecurity. Of course, people still got hacked because this is difficult. Imagine you have a very complex system. Like, imagine you have a house with lots of doors and lots of ways to get in. And cybersecurity must, like, make sure that everything is safe. At the same time, your house or your Office must remain open for business. So it's actually a pretty difficult problem. At the same time, without the progress that the industry made in the past 25 years, things could be much worse. And I think the same thing would be happening here. It's going to take a while for defenders to catch up. Right now, AI helps attackers more than defenders.
00:23:41
Speaker 4: Is the general idea that we're just in this sort of awkward phase where a lot of people in cybersecurity always refer to technical debt, right? This idea that you haven't upgraded a system for ages, like the PHP vulnerability we were just discussing, and so now you have to pay back that technical debt. The idea that we're just in an awkward moment where AI is going to be able to exploit existing flaws in the system is But maybe we eventually move to a place where we upgrade the entire system.
00:24:13
Speaker 1: We reduce our outstanding technical debt.
00:24:14
Speaker 4: We reduce the outstanding technical debt. Exactly.
00:24:17
Speaker 5: That's an optimistic way to look at it.
00:24:19
Speaker 2: Okay.
00:24:20
Speaker 5: You know, we found that there are systems that we thought that, okay, that don't have a lot of technical debt. That turned out that they just like dumpster fire like everybody else. That was a very big surprise to me personally. We know there are places, there are bad problems, and we know that if we just point AI at that thing, there will be a lot of bugs coming out. But there are also places we thought that could be okay because they received a lot of attention over the years already. But it turns out that there's still a lot of bugs there as well. And a lot means a lot more, like 10x more than we expected. And of course there are still like a few small places where there was fewer bugs back then before AI. Now there's still fewer bugs still now, but they're like very small, like very like lit, like they are the exception, not the norms. So that's actually a big surprise because I don't think anybody was surprised to find a critical vulnerability in PHP. because PHP wasn't known for like race security. So of course at the PHP 2.0 scale, then that's a different matter. But if you find another vulnerability in PHP today, then that's not a surprise. But if you go and, you know, find a vulnerability or maybe a hundred vulnerabilities, a thousand vulnerabilities in the Chrome browser for example, that's a surprise. And that's exactly what happened. Like people found like a thousand vulnerabilities in Chrome and you know, I was at Google and I knew, and then the entire industry like assumed that and knew that, wow, there was a lot of investment in Chrome security. And not just like, you know, remove, like eliminate vulnerabilities, but making sure that even if you have vulnerabilities, it's difficult to exploit these vulnerabilities. This is what we call mitigations. You accept that you have vulnerabilities, but then you make sure that like attackers couldn't use these vulnerabilities to control your computer or your browser. We thought that, you know, the defense layers were pretty good and the rate of vulnerabilities were low. But then after AI, we just discovered that either like we, as security engineers, suck at our job. We couldn't find these things. Or they're just like too much code that we didn't have enough time to look into.
00:27:03
Speaker 1: So this raises another question. Like, is it possible to have a provably secure browser or a piece of software? You ship the piece of software and you can prove maybe in the same way that mathematicians produce things and this thing called lean or whatever. Does cyber have an equivalent where you could say, This is provably secure. And we know that no matter how good AI gets, it will never find a vulnerability in this piece of software.
00:27:32
Speaker 2: Oh, man, this is a loaded question, Joe.
00:27:36
Speaker 1: I thought it was a simple question.
00:27:38
Speaker 2: Yeah.
00:27:39
Speaker 3: I don't know, to be honest, you know, like browsers are like, you know, its own operating system, like very, very complex operating systems. Yeah. And today doing, you know, like formally verified software, you know, like it doesn't work at that scale. It's typically smaller, smaller software components.
00:27:58
Speaker 1: So there is such thing as formal verification when it comes to software.
00:28:03
Speaker 3: Yes.
00:28:03
Speaker 1: You could have small piece of software in which you formally prove in the same way that It takes mathematicians seven lines to prove the two plus two equals four. There is some sort of equivalent for software where you can prove these lines of code are secure, but the question is the scale of even something like a browser is just so enormous. That's a very difficult thing to do.
00:28:29
Speaker 4: Yes.
00:28:30
Speaker 3: It's something that I think is being researched right now because I think this is one of the paths forward to build more secure systems. For example, we've had systems where the core of it is formally verified. It's an operating system that's formally verified, but then there are applications running on these things. Those applications... that have issues. So it's complicated. And these are small systems, right? And you're talking about browser. Browser is like many billions of lines of code. And so that's, yeah, I mean, I can tell you how big Chrome is. I mean, they have like zillions of people working on it. And, you know, like lots of people working on Chrome security all these years. But, you know, it's, but long story short, there are such things, but to make it work on a browser or at a very, very large system, Like, this is, we don't have anything like that today.
00:29:26
Speaker 4: Got it.
00:29:27
Speaker 5: I think there's a deeper reason here. You know, the reason like software is eating the world. You can see like, you know, the top, the biggest companies in the world are software companies, right? It's because software is easy to change. And imagine, you know, that allows like software companies to adapt and to learn much quicker than the rest of the world. You have a product, if that doesn't work, then tomorrow, tonight, you can change it. And that's actually allows the software world, you know, to become, uh, to dominate the world basically. So basically fundamentally software is easy to change and people want to change it constantly. And that's the reason why, you know, uh, the tech companies are doing so well. So if you ask like, okay, now let's not change this like frequently, uh, then that goes against the interest.
00:30:21
Speaker 2: Right.
00:30:22
Speaker 5: And that's not going to happen. So it means that even if you have a way to formally verify everything, that would take a lot of time unless you have some sort of AI help you do the formal verification automatically. Otherwise, it's not going to happen because people want to change their software as quick as possible because that's how they do business.
00:31:03
Speaker 4: This is sort of a related question, but does AI start to change, I guess, the way or like the waterfall of the way things get fixed? So nowadays it's like a vulnerability is discovered. The software company issues an alert and a patch. They disclose it and then people go off and fix it. If exploits are coming faster, does that process start to change in any way?
00:31:29
Speaker 5: Yeah, it changed a lot. Basically, back in the day, we have like two kinds of vulnerabilities, like zero-day vulnerabilities that nobody knows about except the person who filed it. And then end-day vulnerabilities, the vulnerabilities have been disclosed and maybe fixed by the vendors and maybe users have upgraded or not. But now we also have something we call like minus one-day vulnerabilities. So these vulnerabilities... happen when, you know, imagine like, let's take the PHP wannabe as an example. Let's say that, you know, because PHP Foundation and PHP itself is an open source project, everything happens on GitHub, right? And imagine, let's imagine that we reported a bug and PHP fixed the bug publicly on GitHub. The moment we found the bug and then PHP produce a patch for that bug, that very moment, that information became public. Somebody who is watching the PHP repository on GitHub, they could turn that away into an exploit within a few hours. Of course, the vulnerability that we found was more complicated than that. But you can imagine that in principle, the window between the passwords being developed, it's not ready yet, you know, for anybody. And before it's become ready for end user to upgrade, there's a window, right? In the past, you know, the attacker has to wait until the past become available. And then it would take them a long time to develop and exploit. So user can have longer time to upgrade. Now that window is collapsing. It is collapsing everywhere, not just in PHP, right? In Windows, in Mac OS, in anything. Because right now there are people who have like what we call like end day factory. You know, they're just waiting for patches from Microsoft, from Apple, from important critical software. Like I'm sure tens of people are watching PHP today. After what happened, they're watching PHP, Python, Java, all of those important languages. And if there is any sign of like, oh, there's a new vulnerability, they're going to like, you know, their AI would go pro and then, you know, bundling tokens to identify that vulnerability, reproduce it, be an exploit for it. So that changed the... the practice, right?
00:34:21
Speaker 2: Yeah.
00:34:21
Speaker 1: I'm curious what you thought after the hugging face incident, because after that moment, there were sort of two schools of thought that I thought there's, or that I perceived. There's one that was like AI creature, there was this sort of, I guess maybe the anthropomorphic Asian view, AI creatures, they want to escape their cages. They're devious, they're tricksters. They want to show, come up with a way to cheat on the right answers, but cheat on the way that the greater, you know, all these stories. And then there are other people that were just like, guys, build a stronger sandbox. Stop talking about all this stuff. Just build a stronger sandbox. Jensen Wong is kind of in that category. I loved his interview with Ezra Klein, where he was basically like, what's your problem? Just make models that don't hack. He's like, just don't ship it. Fix your model. Don't let them hack. And I love that because it's just like, just fix it. And the AI people are like, no, no, no, we can't. You don't understand. We haven't solved alignment. It's very tricky. And he's like, just fix the model. And I'm sort of curious, as people who have come out of the traditional cyber world, but are now in the AI world, Where do you land on this question? Can you just like, guys, just fix the sandbox, or is there something deeper here?
00:35:35
Speaker 5: I don't think that it's simple to fix the sandbox. That's the problem. People think that, you know, maybe people are not insecure. They think that there's a simple solution to cybersecurity. Just do this thing and you're safe. And that's how we have so many problems, you know. Vendors go out and tell people, just buy this box, this magic box. and put it in your network and you'll be safe. No, it's not like that. Even that box has problems. If you look into like, there's a famous idea in security, like product security is not the same as security product. Like security products may not have product security. Even like the firewalls that you buy, they may have problems. And they do. And what I mean here is, so... What we found, like the Hugging Face and OpenAI incident was fascinating, right? To watch. And it's difficult. Enterprise security is very difficult. Anybody saying like, oh, just buy this thing, just do that thing, you're going to be safe. No, it's just like very complicated. And the reason is like, and then people say, oh, just fix the model. And that's a comeback to the- And.
00:36:46
Speaker 1: Just fix the model, just make one that doesn't hack.
00:36:49
Speaker 5: That comes back to the incentive again. And because if you look at like how AI companies, and Jensen must be careful here, right? Because if AI company is not making money, he's not making money as well.
00:37:00
Speaker 2: Yeah, of course.
00:37:01
Speaker 5: And the AI companies, they're making money by selling tokens. And most of the tokens are used for coding today, like programming, building things, right? And if you build, if you want to be really good at coding, you need an agent on AI that understands code understand technical problems, can debug things, can control computers. But that's basically the definition of a hacker. So you cannot really escape this. You cannot have a cake and eat it too, right? You need to be good at coding. Then you have an AI that is good at hacking. So unless you don't want spend more tokens, or you don't want to get into the coding business anymore, you focus on scientific discovery, physical AI, whatever, then maybe you can train a model that is not good at hacking anymore. But as long as you want to automate knowledge work, then eventually your AI must be able to use computers and control computers to understand technical problems. And that's the reason of someone like Bruce. So it's unavoidable.
00:38:17
Speaker 4: So actually, on this note, I'm curious, can you just talk about what it's like being a hacker in the age of AI versus, say, even three or four years ago, right? Like, it's crazy how quickly stuff has shifted. But how has your day to day changed?
00:38:34
Speaker 3: Yeah, I'll just use some background. So I wrote a textbook with some friends on reverse engineering. It used to be like, you know, like you try to like reversing is the practice of understanding software without any source code or hardware systems.
00:38:50
Speaker 2: That used to be like, I wrote this textbook. And then I think with the.
00:38:55
Speaker 3: Advent of AI, I think what, like, you know, whatever, nine months ago, as of nine months ago, like that entire book is deprecating. It takes like, you know, two prompts or one prompt and then bam, it will generate the textbook for you. And it's even better, you know, and it took us, you know, a long time to write. Just that gives you an idea. And then I talk to friends who've been doing this for decades, right?
00:39:16
Speaker 2: They're like specialists in this area, right?
00:39:18
Speaker 3: And they tell me that, oh yeah, man, like I spend, you know, whatever, $ 5 on some open source model and it reverse engineered the firmware on, you know, some Canon or like you know, HP printer and it hacks it, Bruce. You know, and whereas before it would take me, you know, like months to figure some stuff out.
00:39:35
Speaker 2: You know, that's just to give.
00:39:37
Speaker 3: You like a glimpse into what's happening, like how quickly things change.
00:39:44
Speaker 2: Well, a friend, you know, like this guy, Matt Swish, he's, you know, and he's this dude.
00:39:50
Speaker 4: He's a friend of the show.
00:39:52
Speaker 1: A longtime friend of the show, multi-time guest.
00:39:55
Speaker 5: Yeah, keep going.
00:39:55
Speaker 4: We used to hang out in Abu Dhabi a lot.
00:39:58
Speaker 2: Oh, okay, yeah. Like, I'm no Matt Fawkes.
00:40:01
Speaker 3: I mean, like, now he's like, he gave a talk with our team the other day, and he showed his rack, you know, in his apartment, and he's sitting there, you know, like, hacking stuff, like, while he's sitting there talking. It's just insane. We have people that, let's say, for example, we've been hacking the iPhone. To build a full chain on the iPhone, it's not an easy task. Sometimes it takes several people to do this. And our team was able to do this within.
00:40:35
Speaker 2: A few days.
00:40:36
Speaker 3: Wow.
00:40:37
Speaker 2: You know, as a one person. And we have multiple of these chains.
00:40:40
Speaker 3: It really compresses the time, like the exploitation, the time to build exploits.
00:40:49
Speaker 2: We see customers getting hacked.
00:40:52
Speaker 3: By AI, you know, in the financials, in the banks, basically. You know, defenders cannot, you know, actually, Ty, why don't you tell these stories?
00:41:01
Speaker 2: And, you know, just a little bit, this has happened in the last few weeks, you know. Yeah. how things changed.
00:41:07
Speaker 5: Yeah, basically we saw like, you know, in the OpenAI and HackingFace incident, there's one detail that is very interesting. I'm not sure if people noticed, but OpenAI noticed that like the people who handled incidents on the OpenAI side, they said that they have handled a lot of incidents, but this is the first time they saw that, you know, an incident where you have like a hundred people points within your networks that were compromised at the same time. Like there were a hundred like hackers within your system. Normally you have maybe one or two, right? Now you have a hundred. And I saw that I was like, wow, that was pretty interesting, but maybe because OpenAI have lots of tokens, so their agents can just go, you know, hack everything. And then just two weeks ago, we have like two clients in the banking industry. And they got hacked and they called us to help them with like the incident response. We came in because we do hacking as well. We know like usually it took us like maybe a month or something. Once we come in a network and then took a month for us to spread out. For the AI, it took like, they came in, they brought inside the network on Monday. By Tuesday, they're everywhere.
00:42:23
Speaker 3: Everywhere.
00:42:24
Speaker 5: And like, it's like cancer is spread like at a machine speed. And it was like mind blowing, right? Because how could you defend against that? Like in the past, we were telling our clients, you know, you have to invest in detection. Well, how can you help yourself? If you see it, you can detect it, but there's a hundred of them.
00:42:45
Speaker 3: Noted.
00:42:45
Speaker 4: We have a hundred hackers in our system right now. It's not really helpful.
00:42:49
Speaker 3: Exactly.
00:42:50
Speaker 5: How could you do it, right? And then you have to shut down everything. But shut down is not an option because you are a bank. So that's actually true. requires us to sit down and think about how to redesign the defense system. What we have right now is not working very well. But to come back to your question, as a hacker, how does it feel? I love history. So I always come back to history to see, did it happen before? It actually did happen with chess, for example. You remember in 1997, You know, IBM Deep Blue defeated Garry Kasparov. That was a big moment, right? And then 20 years later, AlphaGo defeated like Lizzie Doe in the game of Go. But those two moments are actually quite different. Like the first moment, the Deep Blue moment, after Deep Blue, the best chess player wasn't a computer because Deep Blue was a mainframe, right? Not something that you can run on your computer. And it took 10 years for someone to go from Deep Blue to develop a computer software that you can run on your computer that defeated everybody. But the 10 years, it took 10 years. During that 10 years, the best chess player was a centaur player. This is a term coined by Garry Kasparov. It's a combination of computer and human. It took 10 years. For AlphaGo, it took like three years. Like after AlphaGo, Then 2019, somebody come up with a computer software that defeated anybody without any human input at all. At Calif, we think like, you know, we are enter the era of the central hackers. Basically, the best hacker right now is a combination of AI and humans. And, you know, our work and our friends' work as well show that AI is powerful. It can hack anything. but it still requires a human expert to arrive at it. And we don't know how long this is going to last, maybe three months. Maybe we come back here and by the end of the year, we tell you we retire now. There's nothing else for us to do. Maybe it lasts like three years. And this is actually pretty exciting to find out. That's exactly what we want to do. And we think like, That's like a pessimistic view, right? AI is going to do everything. Let's just retire and hate AI. Or you can see AI as a tool. It's like humanity just discovered fire. And without fire, we wouldn't be here today. So, of course, we have to think about how to control fire. At the same time, a very important question that we need to ask is, what should we cook, right? That's very important, and we want to find out What can we cook with AI and how can we do it in a safe way that helps everybody?
00:45:53
Speaker 4: Protein that makes our brains get bigger.
00:45:55
Speaker 3: Yeah. All right.
00:45:56
Speaker 4: Well, Ty and Bruce from Calif, thank you so much for coming on All Thoughts.
00:46:00
Speaker 1: That was a lot of fun. You're both excellent. Thank you both.
00:46:04
Speaker 5: Thanks.
00:46:05
Speaker 2: Thanks, Joe. Thanks, Tracy.
00:46:06
Speaker 5: Thank you.
00:46:06
Speaker 2: Bye-bye.
00:46:06
Speaker 3: Bye-bye.
00:46:19
Speaker 4: Joe, I thought that was a fantastic conversation. Not exactly uplifting. I was surprised at Ty's response. I thought maybe people would say we're in this weird, awkward moment where a lot of legacy issues are being discovered. But once AI actually gets deployed and starts redesigning some of the architecture, it'll be a lot more resilient. Apparently that's not the case. That's a little bit worrying.
00:46:44
Speaker 1: I agree. There was so much in there that was interesting. And like I could, you know, all this stuff about how, yes, you could have provably secure software, but it's small. You can't even do it for something that says the browser. I love that last point about how we're in the centaur age where the strongest hacker is the computer working with a human, but that may not be there for much longer. Three months, a year. We have no idea, but the presumption should be that it won't last very long. etc.
00:47:12
Speaker 4: Also, the idea of like a computer, an agent that can code is inherently a hacker. And so you can't just tell it not to hack.
00:47:23
Speaker 1: Just teach it not to hack. What's wrong with you? Just teach it not to hack. If coding is going to be big business, which it is, you're going to be building a lot of good hackers.
00:47:31
Speaker 5: Yeah.
00:47:31
Speaker 4: And so the hackers proliferate. On that note, shall we leave it there?
00:47:35
Speaker 1: Let's leave it there.
00:47:35
Speaker 4: This has been another episode of the All Thoughts Podcast. I'm Traci Allaway. You can follow me at Traci Allaway.
00:47:41
Speaker 1: And I'm Joe Wiesenthal. You can follow me at The Stalwart. Follow our producers, Carmen Rodriguez at Carmen Arman, Dashiell Bennett at Dashbot, Cale Brooks at Cale Brooks. and Kevin Lozano at Kevin Lloyd Lozano. And of course, you can chat about all of these topics 24-7 in our Discord, discord.gg slash oddlots. And for more Oddlots content, you can check out our daily newsletter, which you can find at bloomberg.com slash oddlots.
00:48:05
Speaker 4: And if you enjoyed this conversation, if you like it when we talk about hacking, then please leave us a positive review on your favorite podcast platform. And if you're watching on video, then please leave a comment or like, or better yet, subscribe.
00:48:17
Speaker 1: And remember, if you're a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there.
00:48:27
Speaker 4: Thanks for joining. Thank you.
Speaker 1: Bloomberg Audio Studios. Podcasts.
00:00:05
Speaker 2: Radio.
00:00:06
Speaker 3: News.
00:00:18
Speaker 4: Hello and welcome to another episode of the Oddbots Podcast. I'm Tracey Alloway.
00:00:22
Speaker 1: And I'm Joe Weisenthal.
00:00:23
Speaker 4: Joe, it's October. We're recording this on October 29th.
00:00:27
Speaker 5: Yeah.
00:00:27
Speaker 4: It is also Black October. Have you heard this term?
00:00:30
Speaker 5: Yeah.
00:00:31
Speaker 1: What does that mean? Have stocks gone down?
00:00:33
Speaker 2: No.
00:00:34
Speaker 4: Your mind immediately goes to markets. There are other worlds out there. We're going to be talking about the world of cybersecurity today because one of the reasons people are talking about Black October is we've had a number of recent security issues, the most prominent of which has been a bunch of vulnerabilities discovered in PHP, which is a coding language that apparently is used by 70% of websites in existence. We can get into that stat later. But the really interesting thing about this vulnerability being uncovered is AI was used to uncover it. And so the big question coming back to AI, there's a concern now that you have good actors who are maybe using AI to discover old vulnerabilities and trying to fix them. But you could also have bad actors who are using AI to exploit those same vulnerabilities. So we're back to the question of whether AI makes the internet safer or more dangerous.
00:01:29
Speaker 1: It makes it more dangerous. No, I have no idea. Ever since essentially Anthropic announced that they had this model called Mythos and now it's Project Glasswing, which was rolled out to companies to use for defensive purposes before it was available to the general public, there has been much more awareness of cybersecurity issues as it relates to AI. And there are many things that people debate about AI and some of these bigger existential questions. Pretty much everyone agrees cybersecurity good, more cybersecurity good. I will say I know very little about the world of cybersecurity. I don't know what, for example, a cybersecurity company actually sells. I don't know how if I were buying cybersecurity, I would know whether it's working or whether no one had tried to hack me or not or something like that. I need to learn a lot more, but it's absolutely true about the uptick in the volume of hacks.
00:02:19
Speaker 4: There are hackers out there right now taking notes going, Joe doesn't know anything about cybersecurity, probably doesn't have a VPN.
00:02:25
Speaker 5: I don't.
00:02:26
Speaker 1: That's why my vibe coding projects, I don't want to ever, I have yet to build anything with an account because I don't want to like have to think about that. So it's just like, I don't want to count information with anyone. But I have as a user of technical products, over the last several weeks, you get more emails saying like, Our thing was compromised. You're probably fine, but maybe you should refresh some keys or whatever. This is happening. This is real. And everyone really needs to learn more. And if you think about like black swans and Wall Street and all that stuff, you know, there's been for years a lot of talk about cyber as a major risk out there.
00:03:02
Speaker 2: Yeah.
00:03:03
Speaker 4: All right, well, we do, in fact, have the perfect guest. We're going to be speaking to two people from the company Calif, one of the companies that actually uncovered this PHP vulnerability. The other one was a company called Wiz. We have Tai Yun. He is the CEO over at Calif, which is a hacking firm, and we can get into exactly what that means, as well as Bruce Dang. He is an engineer at Calif as well. So, Tai and Bruce, thank you so much for coming on All Thoughts.
00:03:30
Speaker 2: We're glad to be here. So.
00:03:32
Speaker 4: Just to begin with, because hackers kind of, they always have cool achievements that they like to talk about. Why don't we just go ahead and start with who you are and what are the coolest things you've done in your hacking career?
00:03:45
Speaker 3: Yeah. So I started my career at Microsoft almost 20 years ago in the Microsoft Security Response Center. These are the people that do patch Tuesdays. On the second Tuesday of every month, Microsoft releases an update. So I worked in that team a long time ago. Probably the biggest thing I did there and that I could talk about is I helped reverse engineer the Stuxnet virus, which is one of the one that was related to the supposedly the Iranian nuclear program was used to sabotage the Iranian nuclear program.
00:04:18
Speaker 4: This is the thing that people describe it as the world's first like known digital cyber weapon, right?
00:04:24
Speaker 2: Yeah, yeah, yeah, yeah.
00:04:26
Speaker 3: I'm sure there were many others before, but yeah, that's the first public one, yeah. I worked there for almost a decade. And then I left to do a startup with my friends. And then I joined Apple. I worked on iOS security. And then Ty managed to convince me to follow the Church of AI.
00:04:49
Speaker 2: And so here I am.
00:04:50
Speaker 5: Yeah, it took me four years to convince Bruce. So, you know, when you go to a website or you go to Amazon, even Facebook, Instagram, your computer will share information with the server, right? Yeah. And that would be, the information would travel through many, many other servers, like your ISP, your Wi-Fi routers. So you want to encrypt that information to make sure that only the server can see your data, nobody else. And there's a mechanism for that. It's called the SSL protocol, or the later version is called like TOS, like transport layer protocol. security protocol. So, um, I love those protocols and, um, back in the day, together with, uh, friends, we found a bunch of ways to break the protocols, you know, to decrypt the data. And, um, it's pretty cool. It's actually changed the internet, how the internet works, because we basically, um, showed that in the past that like two protocols, like SSL Versus 3.0, that was invented by Netscape. You remember Netscape? Yeah, yeah. 30 years ago. And then people still use it because it's still working somewhat. But we show that, oh, it's not working very correctly. If you use SSL 3.0, then people can decrypt your data. So we hope the internet really, really removes SSL 3.0. And that's one thing. The other thing that I really enjoy doing was, you know, people think that in the past you have compression, you have some data and you want to compress the data and you have encryption. And both of them actually randomize your data in some way, right? If you look at a zip file, you open a zip file, look at the raw binaries, buy data in the zip file, look random to you. Because... all the redundant information has been removed and encoded in a way to reduce the size of the file. And so one of the basic question in computer security was, would you compress data and then you encrypt or you encrypt then compress? And I took some crypto class at Stanford University and I remember my professor, he asked this question in one of the homework assignments. And the obvious answer back then was you don't want to compress and then you don't want to encrypt and compress because encryption randomize your data. It means that compression would be less effective. So the right answer is you compress and then you encrypt. And that was the correct answer on my homework assignment. I think I took that class in 2011. And then three years later, we found that it's wrong. If you do that, then people can actually decrypt your data. And the way it works is pretty cool. I can explain to you. So compression actually reveals, if you look at a compressed file, you know how well it is compressed, right? Because it's softer. If it's smaller, the compression is very good because there's a lot of redundant information. That's why compression works well. It means that if you look at a compression alone, the size of the compression, a compressed file, you can tell whether the file has a lot of redundant information. And encryption hides the length. So it doesn't hide the size of the encrypted data. When you encrypt a message, you can still see that this message is maybe this long or this message may be this long, right? Because encryption doesn't try to hide that. There's some people try to do that. So it means that if you compress and then you encrypt, it means that I can see where your message has some redundant information, has a lot of redundant information. And that alone is a leakage already, right? That you don't want to review. And I gave you an example. Let's say that you take a message, like a random message that I gave you, and then you put your password together with that message. Let's just say that there's a reason for you to do that. You put your password together with the message I gave you and you compress together. And then you encrypt it. with a key that I don't know. And you reveal the ciphertext, the result encrypted message to me. So supposedly I don't have the key. I shouldn't be able to read the message, right? But because I give you a message and you get my message put together with your password, you compress together. If the compression works very well, then I know whether my message has some redundancy with your password.
00:09:46
Speaker 4: You can reverse engineer between the two relationships.
00:09:49
Speaker 5: Yeah, exactly. And you do that repeatedly. Eventually, we can recover your password. And that's exactly what we found on the web encryption protocols. In the past, that's exactly what they did. They take my data, they concatenate your password, they compress together, and then they send over the wire. And then as an attacker, I can see the length And that's enough for me to decrypt. And the solution was really interesting, because they decided to turn off compression. There's no solution. And then eventually, I was at Google back then. I was born in Saigon, Vietnam. I grew up there. Then I got a job in Mountain View for a consultancy, like what we do. So I moved to Mountain View in 2011. And then I joined Google. later that year. And I did cryptography at Google, basically help people do cryptography correctly, like try to avoid the mistakes that we found. I was there for 12 years. I work on like very last number of products. Google has a lot of products, right? Gmail, Chrome browser that we are using right now. And there's a lot of people out there that want to hack Google and our products.
00:11:17
Speaker 2: Yeah.
00:11:17
Speaker 5: So my job was to hack these products before we released them to the public. And I was there for 12 years and then I left to create Calif. And yeah.
00:11:30
Speaker 1: That's fantastic. I'm already fascinated by this conversation. I want to get into obviously what Calif does specifically. I want to get into some of the latest vulnerabilities. But before we do that, you know, both of you have had long careers that by and large predate what we now call AI. Are the risks associated with AI a sort of continuation on some path that cybersecurity have well understood, or are we in a truly novel area for how we should we be thinking about cybersecurity risk?
00:12:07
Speaker 3: I think it's a combination of the two. We've always known that there are security vulnerabilities in systems. And so we have security engineers who are hackers.
00:12:17
Speaker 2: Who try to find and exploit these vulnerabilities.
00:12:20
Speaker 3: So the system has always been vulnerable to some degree, to a varying degree. What AI has done is it's kind of accelerated the speed in which you identify and exploit these vulnerabilities to a level where it's almost like superhuman. You know, like 20 years ago, Microsoft would release, let's say, maybe 20 updates every month. Now you're looking at like, you know, 1,400 updates per month. And that number is going to continue to grow. Just to give you an idea of the scale of things. And now, for example, to have certain systems, it may take you, let's say, three months with lots of people. Now, that may take you a few days with just one person and a very capable AI model.
00:13:08
Speaker 2: That's the reality that we're in today.
00:13:09
Speaker 3: It even changed the way how Attackers have certain types of systems, like in the banking sector, that our customers are in. You can tell some stories about that time. But that's kind of what I think. I think it's the systems have always been vulnerable to some degree.
00:13:26
Speaker 2: And now it's the AI just... accelerates that and makes it much more visible.
00:13:32
Speaker 3: And it's also made it easier for people to identify these vulnerabilities as well.
00:13:37
Speaker 2: And so, and then, you know, obviously there are pros and cons to that. But yeah, you can go into that later.
00:13:58
Speaker 4: Why don't we talk about discovering the PHP vulnerability then? Because I think this was a joint project between Calif, Wiz, and I think it was OpenAI. How did you find this? What was the process? And then what was the decision? I know you guys have gotten a little bit of pushback about publicizing it before it was fixed. So what was the process of actually putting out the statement saying we found this thing?
00:14:24
Speaker 5: We're really interested in finding vulnerabilities that have wide impact. And we want to understand these vulnerabilities to help inform decisions, defense decisions, how to fix and how to mitigate them for once and forever. On that background, we joined the OpenAI's Pass the Planet program. This is where OpenAI funds companies like Calif to go out and help open source maintainers to identify and fix, mitigate vulnerabilities in their software. And we have this engineer, like Ryan Osteen, who is, he's like a PHP historian. Yesterday, we went to a conference in San Francisco. We basically presented like, you know, PHP 2 shell, and Ryan walked everybody over like the 25 years of history of PHP vulnerabilities. So we have this engineer, Ryan, and this is how we work, basically. Our engineer, they're interested in certain things, and then they just go and hack those things. And, you know, as business people, we try to see, okay, how can we convince some other people to pay us for this work? And that's exactly what we did with OpenAI. We convinced OpenAI, okay, we're going to hack PHP. Can you fund our work? And they said, yes, they gave us tokens and money. And then we go and convince like PHP Foundation, which is the organization behind PHP, that let's work together, you know, we can help you. And it's actually pretty interesting because I mentioned like they're like a history of 25 years of like PHP vulnerabilities. And the person that is behind all of these hacks is like Stephen Escher. He's also at Calif. He joined us, I think, six months ago. Back in the day, he's the PHP security guy. Basically, he hacked and helped, like, harden PHP. So we didn't know anybody at the PHP Foundation. So we asked Stefan to introduce us. And because, you know, even though Stefan stopped working on PHP security for a while now, his name still rings, you know, very... loudly in this circle of the world.
00:16:50
Speaker 4: I saw the comic that you guys put up about this. It was kind of funny.
00:16:55
Speaker 3: Yes.
00:16:56
Speaker 4: He's like Indiana Jones type character.
00:16:59
Speaker 5: Yeah. So he connected us and then we met with Volker Dusch at the PHP Foundation. And yeah, that's how we got started. I think we started in July 30. And then immediately after one or two days, you know, Ryan found the first vulnerability, and he was interested in finding a vulnerability similar to one vulnerability that Stefan disclosed 22 years ago. And to his surprise, there's still a bunch of them like that in the PHP software right now. And so that's how we found it. And then we reported it to the PHP Foundation. because we know immediately that is a critical vulnerability that could affect a large part of the internet. Then the Peace Preach Foundation actually shared with us that they actually found this vulnerability, the same vulnerability, two years ago. But they didn't fix it. Because when they found it, they discovered that the fix is very expensive. And there's just maybe a few options to fix it. and none of them were actually perfect because they need to change a lot of functionalities within PHP. And those changes may introduce new bugs, new vulnerabilities, or new reliability issue. And people would be very upset if they upgrade PHP and their website stop working. So in the end, back in like 2024, The PHP Foundation decided not to fix it because there was no exploit. No one could manage to show that this is actually a real issue. And that's exactly what they asked us. They come back, they ask like, do you have an exploit? Then the next day we share with them the first exploit that showed them that, yeah, someone doesn't need to know anything. can hack your website remotely.
00:19:13
Speaker 4: So just to be clear, the use of AI in all of this, it wasn't necessarily that important in finding the vulnerability. It was more important in developing an actual exploit for a known issue.
00:19:27
Speaker 5: We think like that's probably the right way to think about it. But I think even for finding the vulnerability without AI, it would take a lot more time, right? So you can imagine like PHP was created like maybe 30 years ago. So it's a huge code base with a lot of features. And as a human engineer, you have to dig into this code base and it will take like months, right? And along the way, people may give up. But the fact that PHP Foundation filed the bug in 2024 proved that humans can actually file the bugs, right? And the problem was the exploitation, like you said. And for exploitation, like AI plays a major role in it. It's like becomes an engineering problem where our engineers would tell AI, you know, this is how I want to design our exploit and you go and do it. And then, you know, prove to us that it works correctly. Yeah.
00:20:31
Speaker 1: Let me ask a philosophical question. Is there a reason to believe that defensive cybersecurity can stay ahead of offensive cybersecurity, just generally, and then we can get into some details. But philosophically, is that something that we can reasonably hope for?
00:20:49
Speaker 2: I think so, yes.
00:20:50
Speaker 1: How come?
00:20:51
Speaker 3: Because you can use the AI to build certain types of systems in ways that may be very challenging. for attackers to exploit. Today, you can use AI to build whatever systems you want very quickly. You can experiment, get feedback very quickly. It can also process data very, very quickly and reason about that data.
00:21:19
Speaker 2: And so I think it is possible.
00:21:22
Speaker 3: I think it may not be immediate, but over time, we're going to perhaps use AI to help us write more secure software or build more secure systems. And eventually, I think that should counterbalance the advantages.
00:21:40
Speaker 2: That the attackers have. Perhaps I'm being optimistic, but I do foresee this happening.
00:21:47
Speaker 5: I can add something there. So if you look at the history of humanity, our history is the history of like weaponize our tools, right? If you find a powerful tool, then someone would find a way to use that tool as a weapon.
00:22:03
Speaker 4: Yeah.
00:22:04
Speaker 5: So, and attackers always are early adopters of tools, right? New technology. Defenders move like at much slower pace than attackers. But fundamentally, we believe there are more good guys than bad guys out there. And eventually, the good guys would catch up and build things that help everybody defend off the bad guys. And I don't think in general, cybersecurity, if you see the news all the time, you see we got data breach every day almost, even before AI. So you may feel like it's not working at all. But if you're in the field, you can see the progress, right? In the past, like, 25 years, there's a tremendous progress in cybersecurity. Of course, people still got hacked because this is difficult. Imagine you have a very complex system. Like, imagine you have a house with lots of doors and lots of ways to get in. And cybersecurity must, like, make sure that everything is safe. At the same time, your house or your Office must remain open for business. So it's actually a pretty difficult problem. At the same time, without the progress that the industry made in the past 25 years, things could be much worse. And I think the same thing would be happening here. It's going to take a while for defenders to catch up. Right now, AI helps attackers more than defenders.
00:23:41
Speaker 4: Is the general idea that we're just in this sort of awkward phase where a lot of people in cybersecurity always refer to technical debt, right? This idea that you haven't upgraded a system for ages, like the PHP vulnerability we were just discussing, and so now you have to pay back that technical debt. The idea that we're just in an awkward moment where AI is going to be able to exploit existing flaws in the system is But maybe we eventually move to a place where we upgrade the entire system.
00:24:13
Speaker 1: We reduce our outstanding technical debt.
00:24:14
Speaker 4: We reduce the outstanding technical debt. Exactly.
00:24:17
Speaker 5: That's an optimistic way to look at it.
00:24:19
Speaker 2: Okay.
00:24:20
Speaker 5: You know, we found that there are systems that we thought that, okay, that don't have a lot of technical debt. That turned out that they just like dumpster fire like everybody else. That was a very big surprise to me personally. We know there are places, there are bad problems, and we know that if we just point AI at that thing, there will be a lot of bugs coming out. But there are also places we thought that could be okay because they received a lot of attention over the years already. But it turns out that there's still a lot of bugs there as well. And a lot means a lot more, like 10x more than we expected. And of course there are still like a few small places where there was fewer bugs back then before AI. Now there's still fewer bugs still now, but they're like very small, like very like lit, like they are the exception, not the norms. So that's actually a big surprise because I don't think anybody was surprised to find a critical vulnerability in PHP. because PHP wasn't known for like race security. So of course at the PHP 2.0 scale, then that's a different matter. But if you find another vulnerability in PHP today, then that's not a surprise. But if you go and, you know, find a vulnerability or maybe a hundred vulnerabilities, a thousand vulnerabilities in the Chrome browser for example, that's a surprise. And that's exactly what happened. Like people found like a thousand vulnerabilities in Chrome and you know, I was at Google and I knew, and then the entire industry like assumed that and knew that, wow, there was a lot of investment in Chrome security. And not just like, you know, remove, like eliminate vulnerabilities, but making sure that even if you have vulnerabilities, it's difficult to exploit these vulnerabilities. This is what we call mitigations. You accept that you have vulnerabilities, but then you make sure that like attackers couldn't use these vulnerabilities to control your computer or your browser. We thought that, you know, the defense layers were pretty good and the rate of vulnerabilities were low. But then after AI, we just discovered that either like we, as security engineers, suck at our job. We couldn't find these things. Or they're just like too much code that we didn't have enough time to look into.
00:27:03
Speaker 1: So this raises another question. Like, is it possible to have a provably secure browser or a piece of software? You ship the piece of software and you can prove maybe in the same way that mathematicians produce things and this thing called lean or whatever. Does cyber have an equivalent where you could say, This is provably secure. And we know that no matter how good AI gets, it will never find a vulnerability in this piece of software.
00:27:32
Speaker 2: Oh, man, this is a loaded question, Joe.
00:27:36
Speaker 1: I thought it was a simple question.
00:27:38
Speaker 2: Yeah.
00:27:39
Speaker 3: I don't know, to be honest, you know, like browsers are like, you know, its own operating system, like very, very complex operating systems. Yeah. And today doing, you know, like formally verified software, you know, like it doesn't work at that scale. It's typically smaller, smaller software components.
00:27:58
Speaker 1: So there is such thing as formal verification when it comes to software.
00:28:03
Speaker 3: Yes.
00:28:03
Speaker 1: You could have small piece of software in which you formally prove in the same way that It takes mathematicians seven lines to prove the two plus two equals four. There is some sort of equivalent for software where you can prove these lines of code are secure, but the question is the scale of even something like a browser is just so enormous. That's a very difficult thing to do.
00:28:29
Speaker 4: Yes.
00:28:30
Speaker 3: It's something that I think is being researched right now because I think this is one of the paths forward to build more secure systems. For example, we've had systems where the core of it is formally verified. It's an operating system that's formally verified, but then there are applications running on these things. Those applications... that have issues. So it's complicated. And these are small systems, right? And you're talking about browser. Browser is like many billions of lines of code. And so that's, yeah, I mean, I can tell you how big Chrome is. I mean, they have like zillions of people working on it. And, you know, like lots of people working on Chrome security all these years. But, you know, it's, but long story short, there are such things, but to make it work on a browser or at a very, very large system, Like, this is, we don't have anything like that today.
00:29:26
Speaker 4: Got it.
00:29:27
Speaker 5: I think there's a deeper reason here. You know, the reason like software is eating the world. You can see like, you know, the top, the biggest companies in the world are software companies, right? It's because software is easy to change. And imagine, you know, that allows like software companies to adapt and to learn much quicker than the rest of the world. You have a product, if that doesn't work, then tomorrow, tonight, you can change it. And that's actually allows the software world, you know, to become, uh, to dominate the world basically. So basically fundamentally software is easy to change and people want to change it constantly. And that's the reason why, you know, uh, the tech companies are doing so well. So if you ask like, okay, now let's not change this like frequently, uh, then that goes against the interest.
00:30:21
Speaker 2: Right.
00:30:22
Speaker 5: And that's not going to happen. So it means that even if you have a way to formally verify everything, that would take a lot of time unless you have some sort of AI help you do the formal verification automatically. Otherwise, it's not going to happen because people want to change their software as quick as possible because that's how they do business.
00:31:03
Speaker 4: This is sort of a related question, but does AI start to change, I guess, the way or like the waterfall of the way things get fixed? So nowadays it's like a vulnerability is discovered. The software company issues an alert and a patch. They disclose it and then people go off and fix it. If exploits are coming faster, does that process start to change in any way?
00:31:29
Speaker 5: Yeah, it changed a lot. Basically, back in the day, we have like two kinds of vulnerabilities, like zero-day vulnerabilities that nobody knows about except the person who filed it. And then end-day vulnerabilities, the vulnerabilities have been disclosed and maybe fixed by the vendors and maybe users have upgraded or not. But now we also have something we call like minus one-day vulnerabilities. So these vulnerabilities... happen when, you know, imagine like, let's take the PHP wannabe as an example. Let's say that, you know, because PHP Foundation and PHP itself is an open source project, everything happens on GitHub, right? And imagine, let's imagine that we reported a bug and PHP fixed the bug publicly on GitHub. The moment we found the bug and then PHP produce a patch for that bug, that very moment, that information became public. Somebody who is watching the PHP repository on GitHub, they could turn that away into an exploit within a few hours. Of course, the vulnerability that we found was more complicated than that. But you can imagine that in principle, the window between the passwords being developed, it's not ready yet, you know, for anybody. And before it's become ready for end user to upgrade, there's a window, right? In the past, you know, the attacker has to wait until the past become available. And then it would take them a long time to develop and exploit. So user can have longer time to upgrade. Now that window is collapsing. It is collapsing everywhere, not just in PHP, right? In Windows, in Mac OS, in anything. Because right now there are people who have like what we call like end day factory. You know, they're just waiting for patches from Microsoft, from Apple, from important critical software. Like I'm sure tens of people are watching PHP today. After what happened, they're watching PHP, Python, Java, all of those important languages. And if there is any sign of like, oh, there's a new vulnerability, they're going to like, you know, their AI would go pro and then, you know, bundling tokens to identify that vulnerability, reproduce it, be an exploit for it. So that changed the... the practice, right?
00:34:21
Speaker 2: Yeah.
00:34:21
Speaker 1: I'm curious what you thought after the hugging face incident, because after that moment, there were sort of two schools of thought that I thought there's, or that I perceived. There's one that was like AI creature, there was this sort of, I guess maybe the anthropomorphic Asian view, AI creatures, they want to escape their cages. They're devious, they're tricksters. They want to show, come up with a way to cheat on the right answers, but cheat on the way that the greater, you know, all these stories. And then there are other people that were just like, guys, build a stronger sandbox. Stop talking about all this stuff. Just build a stronger sandbox. Jensen Wong is kind of in that category. I loved his interview with Ezra Klein, where he was basically like, what's your problem? Just make models that don't hack. He's like, just don't ship it. Fix your model. Don't let them hack. And I love that because it's just like, just fix it. And the AI people are like, no, no, no, we can't. You don't understand. We haven't solved alignment. It's very tricky. And he's like, just fix the model. And I'm sort of curious, as people who have come out of the traditional cyber world, but are now in the AI world, Where do you land on this question? Can you just like, guys, just fix the sandbox, or is there something deeper here?
00:35:35
Speaker 5: I don't think that it's simple to fix the sandbox. That's the problem. People think that, you know, maybe people are not insecure. They think that there's a simple solution to cybersecurity. Just do this thing and you're safe. And that's how we have so many problems, you know. Vendors go out and tell people, just buy this box, this magic box. and put it in your network and you'll be safe. No, it's not like that. Even that box has problems. If you look into like, there's a famous idea in security, like product security is not the same as security product. Like security products may not have product security. Even like the firewalls that you buy, they may have problems. And they do. And what I mean here is, so... What we found, like the Hugging Face and OpenAI incident was fascinating, right? To watch. And it's difficult. Enterprise security is very difficult. Anybody saying like, oh, just buy this thing, just do that thing, you're going to be safe. No, it's just like very complicated. And the reason is like, and then people say, oh, just fix the model. And that's a comeback to the- And.
00:36:46
Speaker 1: Just fix the model, just make one that doesn't hack.
00:36:49
Speaker 5: That comes back to the incentive again. And because if you look at like how AI companies, and Jensen must be careful here, right? Because if AI company is not making money, he's not making money as well.
00:37:00
Speaker 2: Yeah, of course.
00:37:01
Speaker 5: And the AI companies, they're making money by selling tokens. And most of the tokens are used for coding today, like programming, building things, right? And if you build, if you want to be really good at coding, you need an agent on AI that understands code understand technical problems, can debug things, can control computers. But that's basically the definition of a hacker. So you cannot really escape this. You cannot have a cake and eat it too, right? You need to be good at coding. Then you have an AI that is good at hacking. So unless you don't want spend more tokens, or you don't want to get into the coding business anymore, you focus on scientific discovery, physical AI, whatever, then maybe you can train a model that is not good at hacking anymore. But as long as you want to automate knowledge work, then eventually your AI must be able to use computers and control computers to understand technical problems. And that's the reason of someone like Bruce. So it's unavoidable.
00:38:17
Speaker 4: So actually, on this note, I'm curious, can you just talk about what it's like being a hacker in the age of AI versus, say, even three or four years ago, right? Like, it's crazy how quickly stuff has shifted. But how has your day to day changed?
00:38:34
Speaker 3: Yeah, I'll just use some background. So I wrote a textbook with some friends on reverse engineering. It used to be like, you know, like you try to like reversing is the practice of understanding software without any source code or hardware systems.
00:38:50
Speaker 2: That used to be like, I wrote this textbook. And then I think with the.
00:38:55
Speaker 3: Advent of AI, I think what, like, you know, whatever, nine months ago, as of nine months ago, like that entire book is deprecating. It takes like, you know, two prompts or one prompt and then bam, it will generate the textbook for you. And it's even better, you know, and it took us, you know, a long time to write. Just that gives you an idea. And then I talk to friends who've been doing this for decades, right?
00:39:16
Speaker 2: They're like specialists in this area, right?
00:39:18
Speaker 3: And they tell me that, oh yeah, man, like I spend, you know, whatever, $ 5 on some open source model and it reverse engineered the firmware on, you know, some Canon or like you know, HP printer and it hacks it, Bruce. You know, and whereas before it would take me, you know, like months to figure some stuff out.
00:39:35
Speaker 2: You know, that's just to give.
00:39:37
Speaker 3: You like a glimpse into what's happening, like how quickly things change.
00:39:44
Speaker 2: Well, a friend, you know, like this guy, Matt Swish, he's, you know, and he's this dude.
00:39:50
Speaker 4: He's a friend of the show.
00:39:52
Speaker 1: A longtime friend of the show, multi-time guest.
00:39:55
Speaker 5: Yeah, keep going.
00:39:55
Speaker 4: We used to hang out in Abu Dhabi a lot.
00:39:58
Speaker 2: Oh, okay, yeah. Like, I'm no Matt Fawkes.
00:40:01
Speaker 3: I mean, like, now he's like, he gave a talk with our team the other day, and he showed his rack, you know, in his apartment, and he's sitting there, you know, like, hacking stuff, like, while he's sitting there talking. It's just insane. We have people that, let's say, for example, we've been hacking the iPhone. To build a full chain on the iPhone, it's not an easy task. Sometimes it takes several people to do this. And our team was able to do this within.
00:40:35
Speaker 2: A few days.
00:40:36
Speaker 3: Wow.
00:40:37
Speaker 2: You know, as a one person. And we have multiple of these chains.
00:40:40
Speaker 3: It really compresses the time, like the exploitation, the time to build exploits.
00:40:49
Speaker 2: We see customers getting hacked.
00:40:52
Speaker 3: By AI, you know, in the financials, in the banks, basically. You know, defenders cannot, you know, actually, Ty, why don't you tell these stories?
00:41:01
Speaker 2: And, you know, just a little bit, this has happened in the last few weeks, you know. Yeah. how things changed.
00:41:07
Speaker 5: Yeah, basically we saw like, you know, in the OpenAI and HackingFace incident, there's one detail that is very interesting. I'm not sure if people noticed, but OpenAI noticed that like the people who handled incidents on the OpenAI side, they said that they have handled a lot of incidents, but this is the first time they saw that, you know, an incident where you have like a hundred people points within your networks that were compromised at the same time. Like there were a hundred like hackers within your system. Normally you have maybe one or two, right? Now you have a hundred. And I saw that I was like, wow, that was pretty interesting, but maybe because OpenAI have lots of tokens, so their agents can just go, you know, hack everything. And then just two weeks ago, we have like two clients in the banking industry. And they got hacked and they called us to help them with like the incident response. We came in because we do hacking as well. We know like usually it took us like maybe a month or something. Once we come in a network and then took a month for us to spread out. For the AI, it took like, they came in, they brought inside the network on Monday. By Tuesday, they're everywhere.
00:42:23
Speaker 3: Everywhere.
00:42:24
Speaker 5: And like, it's like cancer is spread like at a machine speed. And it was like mind blowing, right? Because how could you defend against that? Like in the past, we were telling our clients, you know, you have to invest in detection. Well, how can you help yourself? If you see it, you can detect it, but there's a hundred of them.
00:42:45
Speaker 3: Noted.
00:42:45
Speaker 4: We have a hundred hackers in our system right now. It's not really helpful.
00:42:49
Speaker 3: Exactly.
00:42:50
Speaker 5: How could you do it, right? And then you have to shut down everything. But shut down is not an option because you are a bank. So that's actually true. requires us to sit down and think about how to redesign the defense system. What we have right now is not working very well. But to come back to your question, as a hacker, how does it feel? I love history. So I always come back to history to see, did it happen before? It actually did happen with chess, for example. You remember in 1997, You know, IBM Deep Blue defeated Garry Kasparov. That was a big moment, right? And then 20 years later, AlphaGo defeated like Lizzie Doe in the game of Go. But those two moments are actually quite different. Like the first moment, the Deep Blue moment, after Deep Blue, the best chess player wasn't a computer because Deep Blue was a mainframe, right? Not something that you can run on your computer. And it took 10 years for someone to go from Deep Blue to develop a computer software that you can run on your computer that defeated everybody. But the 10 years, it took 10 years. During that 10 years, the best chess player was a centaur player. This is a term coined by Garry Kasparov. It's a combination of computer and human. It took 10 years. For AlphaGo, it took like three years. Like after AlphaGo, Then 2019, somebody come up with a computer software that defeated anybody without any human input at all. At Calif, we think like, you know, we are enter the era of the central hackers. Basically, the best hacker right now is a combination of AI and humans. And, you know, our work and our friends' work as well show that AI is powerful. It can hack anything. but it still requires a human expert to arrive at it. And we don't know how long this is going to last, maybe three months. Maybe we come back here and by the end of the year, we tell you we retire now. There's nothing else for us to do. Maybe it lasts like three years. And this is actually pretty exciting to find out. That's exactly what we want to do. And we think like, That's like a pessimistic view, right? AI is going to do everything. Let's just retire and hate AI. Or you can see AI as a tool. It's like humanity just discovered fire. And without fire, we wouldn't be here today. So, of course, we have to think about how to control fire. At the same time, a very important question that we need to ask is, what should we cook, right? That's very important, and we want to find out What can we cook with AI and how can we do it in a safe way that helps everybody?
00:45:53
Speaker 4: Protein that makes our brains get bigger.
00:45:55
Speaker 3: Yeah. All right.
00:45:56
Speaker 4: Well, Ty and Bruce from Calif, thank you so much for coming on All Thoughts.
00:46:00
Speaker 1: That was a lot of fun. You're both excellent. Thank you both.
00:46:04
Speaker 5: Thanks.
00:46:05
Speaker 2: Thanks, Joe. Thanks, Tracy.
00:46:06
Speaker 5: Thank you.
00:46:06
Speaker 2: Bye-bye.
00:46:06
Speaker 3: Bye-bye.
00:46:19
Speaker 4: Joe, I thought that was a fantastic conversation. Not exactly uplifting. I was surprised at Ty's response. I thought maybe people would say we're in this weird, awkward moment where a lot of legacy issues are being discovered. But once AI actually gets deployed and starts redesigning some of the architecture, it'll be a lot more resilient. Apparently that's not the case. That's a little bit worrying.
00:46:44
Speaker 1: I agree. There was so much in there that was interesting. And like I could, you know, all this stuff about how, yes, you could have provably secure software, but it's small. You can't even do it for something that says the browser. I love that last point about how we're in the centaur age where the strongest hacker is the computer working with a human, but that may not be there for much longer. Three months, a year. We have no idea, but the presumption should be that it won't last very long. etc.
00:47:12
Speaker 4: Also, the idea of like a computer, an agent that can code is inherently a hacker. And so you can't just tell it not to hack.
00:47:23
Speaker 1: Just teach it not to hack. What's wrong with you? Just teach it not to hack. If coding is going to be big business, which it is, you're going to be building a lot of good hackers.
00:47:31
Speaker 5: Yeah.
00:47:31
Speaker 4: And so the hackers proliferate. On that note, shall we leave it there?
00:47:35
Speaker 1: Let's leave it there.
00:47:35
Speaker 4: This has been another episode of the All Thoughts Podcast. I'm Traci Allaway. You can follow me at Traci Allaway.
00:47:41
Speaker 1: And I'm Joe Wiesenthal. You can follow me at The Stalwart. Follow our producers, Carmen Rodriguez at Carmen Arman, Dashiell Bennett at Dashbot, Cale Brooks at Cale Brooks. and Kevin Lozano at Kevin Lloyd Lozano. And of course, you can chat about all of these topics 24-7 in our Discord, discord.gg slash oddlots. And for more Oddlots content, you can check out our daily newsletter, which you can find at bloomberg.com slash oddlots.
00:48:05
Speaker 4: And if you enjoyed this conversation, if you like it when we talk about hacking, then please leave us a positive review on your favorite podcast platform. And if you're watching on video, then please leave a comment or like, or better yet, subscribe.
00:48:17
Speaker 1: And remember, if you're a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there.
00:48:27
Speaker 4: Thanks for joining. Thank you.
