Ep 32: AI, Security and eBPF: Buying Time Against the Hackers with AI & Security Expert Liz Rice의 전체 녹취록

people can work on the fix. So maybe if I find a bug in a piece of software that you, you know, that you share, that you deliver, I would tell you via your security reporting processes, here is the problem. And you would say, okay, great. Let's keep this under wraps for the next 30 days or 60 days or 90 days, what have you. And. I would agree to that. And I would say, yeah, but if you don't provide an update, I will make it public after whatever timeframe we've agreed. And that puts pressure on you to provide the fix and then your users get the fix in a timely fashion. In this kind of post mythos era, now these kind of initiatives and collaborative organizations are sharing that information about vulnerabilities they've discovered and kind of collectively working on the fixes together, but also allowing security companies to roll out mitigations that can be applied without necessarily, we don't necessarily need to know what the software fix is in order to prevent the exploit if we know what the exploit is. I guess the important thing here is thinking about how we can give end users, consumers, your bank and my bank, I want them running secure software. I want them able to protect my deposits by having, you know, safe software running. And the idea here is that we can, if they can't afford to wait, you know, as soon as the exploit is known. The other aspect of this is good guys are running AI to find these vulnerabilities and exploits, but also the bad actors are running, you know, using AI to find vulnerabilities so they can exploit them. And if you're defending, you have to be right all the time. You know, if you build a castle, every single brick in the wall around the castle has to be intact. Anne: Yeah. Liz Rice: If you're trying to compromise that castle, you only need to find one weak spot in the wall and you can get in that way. So, you know, it's always an uneven playing field between attackers and defenders. Anne: Yeah. And it's interesting how much that the world has changed. I I remember ten years ago, fifteen years ago talking about this kind of stuff. And we always knew if the if the prize was sufficiently attractive, and it to su to such an extent that kind of like state actors wanted to hack you, then there probably wasn't much you could do about it because they'd just generate put they could direct direct enormous resources at finding a tiny chink that the whatever it was, like like like the the Death Star the. The little the little floor that allows you to to to blow it up. But generally speaking, most hackers did not have those res resources. The situation with AI means that hackers have just almost state level resources, effectively. Liz Rice: Yeah. And the cost to develop an exploit, which used to be, you know, you'd have to pay very talented people a lot of money to spend time trying to find out how to exploit something. You can throw AI at it and the cost to develop an exploit is virtually zero. In theory, the frontier models are putting kind of guard rails around. I'm certainly noticing that if I'm trying to, if I'm writing a demo and I want to show an exploit. I'm sometimes running into AI saying, yeah, I can't help you with that. Anne: Yeah. Liz Rice: But not all frontier models will have that capability. I'm also noticing that, you if I run into something like that, I can just back up to an older model and it probably doesn't have the same guard rails. And, you know, I mean, I'm trying to exploit things that are already known and published and I'm doing it for a demonstration. So it's not such a hard problem. But today's models are very, very good at finding creative ways to exploit vulnerabilities or chain together vulnerabilities in new and unexpected ways. Yeah, it's powerful stuff. Anne: Yeah. Yeah, and it's interesting. The we had we had a a guest on, Yan Qing Cheng a couple of weeks ago, who was talking about how if you've got open weights, and of course the the Chinese open models are pretty sophisticated these days, you know, quite some of them are quite close to the frontier models in functionality. But once you've got open weights, then you Liz Rice: some of them are frontier models. I mean, they are frontier models, some of them, you know, Anne: Yeah. Liz Rice: and there's, I don't think there's any, you know, history is showing us that when one model gets to be, you know, the best one, another one comes along later, that's even better. And then another one comes along, you know, I don't think we should think of some of these open models or models coming out of China as kind of second class. mean, was, Mistral just came out. a week or so ago with some really great results, you know, maybe not as good in all respects. You know, I'm not an AI model expert, but you know, definitely really powerful models coming out in the open source world and open weight world and as well as the frontier models. Anne: Yeah. Yeah, yeah, yeah. That's of course that's it. Yes, that is definitely true. And the and and but the open weight ones, it we are also already seeing people being qu pretty good at using the access to the weights to get rid of the restrictions. Liz Rice: Yeah, I mean, if something is open, you can't really control the way that people are using it. That's kind of by definition, isn't it? Anne: Yeah, yeah. It's good and bad. With both your hats on, your open source hat and your security hat on. So, so we so the problem is hacking it's unbelievably accessible now compared to how how accessible it was previously. That the the solution of the past was always patching with the issue that you you have a kind of there's multiple issues with patching. There's the can you get the patch into pr can you get the patch there before the exploit happens? also the people who just struggle to patch. Liz Rice: Well, I mean, usually it means you've got to stop your running application and, you know, replace that executable with a new version or replace the libraries

with new versions. You know, if you're patching, I mean, something that hadn't really occurred to me, I suppose, till we joined Cisco was, you know, the same thing applies to your infrastructure. Hardware, you know, if you have a vulnerability in a switch, then... some point you're going to have to upgrade the software on that switch. And that means all the traffic running through that switch, you know, it can't run through that switch while it's being rebooted. So this ability to find ways to mitigate vulnerabilities becomes increasingly important because we have to do it so much more often now because of the speed at which vulnerabilities are being discovered. Yeah. mean, people used to plan their downtime. weeks and months in advance. You can't wait, you can't afford to wait that long now. Anne: Yeah, so we've we've really talked about two problems there. That the having access to fixes or workarounds for exploits that that we now understand are there. then there's the ability to apply those those patches or workarounds. to those exploits where there are multiple issues there. There's the there's the how quickly can you do it, but also potentially Liz Rice: Mm. Anne: how can it can you do it at all? Liz Rice: Yeah. And actually you also kind of lean towards thinking about there is patching known vulnerabilities, but there's also, you know, with, with modern security tooling and you know, what we can do with EBPF being able to, and other technologies as well, guess, things like set comp in the past and, and, SE Linux and App Armor where you can put some run. some controls on what is permitted at runtime. I mean, at a very, very basic level, know, access has permissions on, know, that not everyone can access every single file, but you can take that to more sophisticated levels with these, you know, runtime guardrails that you might use to say in general. Firewalls is another great example of what people have done for years saying, I'm expecting network traffic between this address and that address, but not everywhere. And for a long time, security has been built on opening up, please open up this port for me so that my application can run and connect to some other address. having network security that is based on IP addresses and ports. has become insufficient in recent years because of the way we deploy software in the cloud native world, where things are coming up and down dynamically. They're getting new addresses. They're reusing addresses for different applications. You can't rely on those IP addresses anymore. And that makes firewalling. There's still scope for firewalling around potentially your perimeter and between. maybe your deployment and external devices, but in your kind of cloud native clusters and deployments, you need something much more sophisticated that can keep up with the way that IP addresses change dynamically all the time. Anne: Yes, so yeah, so so then you've got you've got firewalling as a defense, as you say, in the cloud native world, it's been a while since that's been a really effective defense. then and you've got patching as a defense, but the issue at the moment is how quickly can you patch and maybe the exploit will be exploited before there's even a patch available. So so what's the current thinking? What what's it Liz Rice: Yeah. So, I mean, if we take those kind of in chronological order of where they've been used and, EBPF has been a really great basis for this kind of technology. The first time I saw, we started talking today about kind of 2016, 2017, that kind of timeframe. And around that time is when. the Cillian project, my colleagues at ISAvalent were starting the Cillian project. And I remember seeing Thomas, Thomas Graf speaking at DockerCon and showing this EBPF technology and the way that it could dynamically. put in place network policy to allow certain traffic based on what the actual container workload was. you know, based on labeling those container workloads, you could say, you know, I'm going to allow my front end to talk to the backend, but I'm not going to allow the front end to talk to the payment gateway, that kind of network policy. It doesn't matter what IP addresses those different components have. It's based on the workload that they're actually running. And I remember back then seeing Zillium and thinking, this EVPF technology is really, really cool. But at that point, you pretty much had to compile your own kernel to use it. And it certainly wasn't in any of the Linux distributions being used at the time. That changed some years ago. And now every Linux distribution pretty much has decent EVPF capability. So everybody can be using this. And the way it's working is. loading EBPF programs, Silium loads EBPF programs and maps that describe the kind of network policies, what traffic is allowed to flow and can use EBPF to inspect packets, you know, incredibly efficiently because it's within the kernel and make decisions on whether or not a given packet is in or out of policy and drop it if it's out of policy. So that's been like the modern approach to let's say cloud native micro segmentation really. And that's been proven in production for many years now. It's not the only use of EBPF in network security. Meta is another great example. since... Believe it's 2016. And you can imagine an organization like Meta does compile its own kernels to have exactly the capabilities that they want. And they have been using eBPF to protect their network. Every single packet that has gone in and out of, you know, Facebook and Instagram, I believe since 2016 has gone through eBPF processing for exactly this kind of security reasons. Cloudflare, another great example, they've been using EBPF for a long time to do DDoS protection. So really scalable network security for quite a long time now. Anne: I it's interesting you should say that, because I was at a conference on earlier this week on Tuesday, and one of the engineers at Meta was talking

about their current security policy for AI, actually internally, for using that for for using AI internally. And they were finding basically they use it, they use AI internally. They they have loads of agents running and doing stuff for them. But those agents are all effectively like mini hackers because they they completely don't do what they're supposed to do. They They probe every interface and try and find workarounds and do all that kind of stuff. So they're having to do quite a lot of internal stuff to protect their own systems from one another that would previously clearly they would have done extern only done externally. and it sounded they did he didn't mention EBPF, but I did think at the time it sounded a little bit like they were looking at every packet and deciding whether or not it was actually doing what they thought it should do at the rather than. some completely strange thing that they really hadn't anticipated what it might do. Liz Rice: I think the more difficult thing today with agent traffic is that it's a lot of it's happening at that kind of application level, you know, text based descriptive prompts and responses and information that is not necessarily formatted in a deterministic way. And it's much harder to police, you know, based on. a vague idea of what this is trying to achieve. And you still get the kind of network level security is still incredibly valuable. You you don't want your agents, a lot of the time you might not want your agents making external network requests, for example, or you might say they're only allowed to access certain resources externally. Of course we've seen incidents. where agents have gone a little bit beyond what the humans expected them to do. So for example, with the hugging face incident, they were not supposed to be out on the internet making requests, but they came up with a creative way of using external resources. yeah, it It would have been hard, I think, to predict exactly how they would have chosen to go about doing that. Anne: Yeah, yeah, yeah. Yeah. So yes, it's interesting. The the the MetaGuy was basically saying that they were kind of internally restricting messages and kind of making everything be much more like a a kind of binary style, easily scannable and filterable. interface. So they were they were basically making Liz Rice: Hmm. Anne: their all their internal interfaces much more prescriptive. So that because you as you Liz Rice: Mm. Anne: say, yeah, text-based interfaces, difficult to tell what's gonna happen. Liz Rice: Yeah, I think we will see more. mean, this is me sort of just thinking, you I'm not involved in directly implementing this kind of stuff, but I think we will see more as inference with sort of small local models happens kind of in situ. I think we will see more use of models that can advise or control on, you know, whether or not a particular agent activity is. sensible. Anne: No. Liz Rice: I don't necessarily want to say safe, but sensible. And I think we'll see that more and more that on your phone or on your laptop, people will be running small models that kind of help to police what other activities are going on. Anne: So so rolling us all the way back, that that it would that traditionally we've had two kind of approaches, although you might say, no no no and there are like loads more approaches than that. Two kinds of approaches to security. One is apply patches, kind of like the software approach, just get rid of the hole that that people are coming in through. That requires you to be able to apply patches which very quickly and you know you have the patch and apply the patch, whatever. And there's also the kind of the infrastructure approach where you just go, look, I'm gonna put a firewall, you know, you've got a firewall in, I just don't expect to see any messages coming from from from these guys. The benefit of the latter is you don't have to be able to patch quickly because you're just saying, look, this wraps the whole thing and makes it so you don't have to apply the patch because I'm not even gonna let the the bad packet get to you. So Liz Rice: Mm. Anne: so what's your where are you? Where does EBPF fit into that kind of i or you might want to say, there's another approach as well. Liz Rice: Yeah. And I guess it's also worth just pointing out that defense in depth is a thing. know, it's not like you have one or the other. put as many Anne: yeah, that's cool. Liz Rice: different defenses as are kind of appropriate. So, you know, just because you're patching doesn't mean you don't need to, you need the firewall as well. And you, depending on what it is that you're protecting here, you you might be more or less prescriptive with that kind of network segmentation depending on know, your environment and your needs and whether or not you're in a regulated industry and all that kind of stuff. But yeah, let's, let's think about what EBPF can do to help us with the kind of software vulnerabilities, the kind of pre-patch problem. So we have a project called Tetragon, which is part of Silium and evolved out of Silium. So I talked earlier about Silium being able to look at network packets and you know, determine whether or not they were in or out of policy using eBPF. Anne: So cyllium is a product that's based on EBPF. Liz Rice: Silium is a project, it's an open source project part of the CNCF. So the CNCF is the organization that looks after Kubernetes and Envoy and Silium and dozens and dozens of other projects that people are using in cloud native deployments. In practice, well, a lot of people run open source versions of these projects. A lot of other people will say, actually, I want a commercial product based on that project. Cilium was originally invented by the ISOVALENT team. ISOVALENT continue to be major contributors to that project. And one of our products is a commercial distribution of Cilium. And in order to debug network problems, you needed to be able to see, OK,

what's happening to different network packets? So we have a component in Cilium called Hubble that lets you see different packet flows and what the policy verdict allow or deny is for individual packets. And then at some point, there was this idea of Well, it's all well and good that I've got this out of policy packet, but where did it come from? How did this packet come to be? And with EBPF, when you hook into an event, you get context around that event. And say, for example, you've hooked into a socket where a network packet is being sent or received. You know about the application that has that socket. So you can attach that information about the process, the application process that's responsible for that packet. So we kind of, this idea of providing more and more contextual information around initially network packets and then thinking, well, actually, you know, we can hook into loads of other events. Why only look at network packet events? Why not look at all the other things that are happening in a system? And That was really how Tatchagon evolved kind of out of the Cilium project and into its own kind of independent project. You can now run Tatchagon directly on a Linux machine. You don't have to be running Kubernetes, but if you are running Kubernetes, it can be aware of what pod you're running in, what workload it is, what node and all that kind of Kubernetes information. So what Tetragon can do is you have policies that they really kind of translate into EBPF programs. saying, I want to observe this kind of event. Let's say we want to observe when processes get executed. And we might say, but I'm perfectly happy for, you know, here is the name of my application. Here is engine X, here is Postgres, here is whatever I am expecting to run. Don't bother telling me about those running, because I know that's going to happen, but tell me if something else unexpected starts to execute. know, somebody lands a cryptocurrency miner that starts running. I want to know about it. Somebody starts running a shell unexpectedly. I really want to know about that. So we can use Tetragon to hook into these interesting events and generate alerts or metrics or just send logs to a SIM. And we call that kind of security observability. And then we kind of added the ability to prevent these events from happening. And a lot of those, we make a lot of use of what's called the Linux security module API inside the kernel, where the kernel is making decisions anyway about things like, this... Is this application allowed to open this file, for example? It's the same interface that people use for SC Linux and App Armor, but we can do it dynamically with EBPF. can make decisions. You could pretty much say, if it's a Tuesday, then you're allowed to open this file. there can be all Anne: Yeah. Liz Rice: sorts of policies that can be dynamic and very context specific. So we can use these policies to prevent things from happening that are out of policy. One way we might do

that would be to have guardrail policies. So things like, I'm going to define what application should run. I'm going to define what file access I expect to see. And for common applications, we can kind of pre-provide policies that, you know, if you're running NGINX, we know what that kind of looks like. So we can give you a policy that describes NGINX if you like. If you're writing an application yourself, you... know what files it's going to access. At least in theory, you can write your own policy. So far, so good. We get runtime security using Tetragon. Now let's think about patching. So when there's a vulnerability, when it gets exploited, there is some behavior of that exploit. know, it's Anne: Yeah. Yeah. Liz Rice: maybe it is it's running a shell or you know, it's opening a file unexpectedly. And either through looking at what that kind of end result behavior is, or maybe looking at, how did they, you know, what's the hole in the first place? What is the bad thing? What is the thing that they did that allowed them to exploit that vulnerability? Example of that recent vulnerability called copy fail where All that the attacker had to do really was open a socket with a particular value in a field that actually nobody really ever used. So it was very easy to say, you know what, nobody should ever be allowed to open a socket with that value. So we could create a Tetracon policy that says, hook into the socket opening call. If you see that value, prevent that from happening. And that blocks the exploit. So we can build these policies or shields that are very targeted to specific vulnerabilities and can prevent them from being exploited. Anne: So so that's very interesting, because so that that fits in w what we're talking about earlier, about when you start to find when you found it an exploit, you know what you usually the whole the it usually starts with if I do this, then I can exploit. So you know that really early, before you as you say, before you've got the patch. Certainly before the patch has been written, and definitely before the patch has been widely deployed. So Liz Rice: Hmm. Anne: you've actually it's it's putting another layer in of protection. Which is okay, you could write the protection using this tetragon thing before you've got a patch, and before you've got a patch for it, and certainly before that patch has been applied, if you were fast, you could apply you could just slap on the shield to the the tetragon shield for it. You it doesn't mean that you would never want the patch, you would absolutely say defense and depth. You'd want all the things. You'd want the shield and the patch. But it's just a bit quicker. I see. Liz Rice: Yeah, but it buys you time, buys you time to apply that patch. yeah, exactly. Yeah, yeah, yeah. Anne: Mm-hmm. And that's really interesting. I like that. So that's actually, I like that because that's completely different from the two things that that I I was saying with the two approaches. The one

is the infrastructure, like quite brute force, very, very kind of like all the things on this port, no more things on this port, no more things. It's like, or the really, really tightly defined, here's the patch for it, which you know has to loads and loads of processes need to go in. get to roll out before that patch is in place and protecting you and you can just slap this on in between. Liz Rice: Yeah. Yes. And it immediately protects running applications. You know, don't have to restart your applications because you're sliding it in into the kernel underneath those applications. And yeah, it's... Anne: That is very cool. That's incredibly Liz Rice: Yeah, yeah. Anne: cool. It's interesting, isn't it? I I guess it is a natural extension of what eBPF you're talking about, like the the first use of eBPF was for kind of like fancier firewalling, effectively. this is easy. Liz Rice: Certainly, yeah, an early use case. Yeah, yeah, yeah. Yeah. Anne: Yeah. Yeah. Liz Rice: I guess at the same time as we were doing those early kind of, yeah, let's call it fancy firewalling. As we Anne: Yeah. Liz Rice: were doing that, there was also a lot being done with observability for like tracking down performance issues. So hooking into different events in the kernel. So you might observe, I don't know how many... OOM kills, for example, you know, how often are they happening or what's the kind of rate at which I'm allocating memory or what, why is some particular performance issue happening? And we could use EBPF to kind of get really fine grained metrics on different parts, know, different subsystems within the kernel. So that was being used really early on as well. Anne: And actually we had a I think it was episode three of this podcast, I had Nikki Monolidarki, from the CNCF. She she's one of the maintainers of the Kepler project. And I think Liz Rice: Right, yes, yes. Anne: so Kepler is a monitor effectively a diagnostics monitoring tool for Kubernetes that uses EPPF because it's the only thing fast enough to get that data without slowing everything down to a crawl. Liz Rice: Yeah. And I mean, people do sometimes ask about, you know, what is the performance impact of eBPF? And the answer is a bit of, well, how long is a piece of string? Because you can write some terrible programs if you really wanted to. What we've learned a lot with Tetragon is to filter events within the kernel. So the more data you push out from the kernel to user space, the, you know, that that takes time and resources, particularly sort of transitioning between kernel and user space can be expensive and shuffling data around can be expensive. So the more you can aggregate events, filter them all within the kernel, the more efficient the overall tooling can be. But we can see some incredibly, you know, low overhead protections. with, I can share a blog post that we have with some benchmarking that we did. I won't try and remember, because I'm not very good at remembering numbers, but I can Anne: Okay. Liz Rice: share the link with you later. Anne: sounds good. That does sound good. So now I'm going to ask all the silly questions that that you that I'm going to embarrass myself by answer asking asking the questions. So the whole so eBPF is is is faster because it runs in the kernel. So in in operating systems there's the kernel which is which is very fast and there's user space which is mo what most applications run in and it's much slower. And one of the reasons but it's also much more safe. If people make mistakes in user space, they cr the application crashes, but the the whole operating system doesn't crash. So you Liz Rice: Mm. Anne: sacrifice kind of protections for speed by moving from running in the kernel to running in user space. So how is it that ePPF is is not dangerous? Liz Rice: Yeah. So it's one of the really fundamental pieces of EVPF. It's a component called the verifier. So this sits in the kernel. And when you load a program into the kernel, an EVPF program into the kernel, it first of all goes through this verifying process. And the verifier will check every possible execution path through that program and make sure that it's safe to run, make sure that memory access is safe, no. null pointer dereferencing and it will make sure that there's no possible way for it to just loop indefinitely. And so that verification process guarantees that the program is going to run to completion and run safely. The Verifier can't check for intent. If I write an eBPF program to drop packets, it has no way of knowing whether that's a firewall being good defense or if it's some kind of attack to drop legitimate packets. It has no moral filter, but Anne: Ha ha ha. Liz Rice: it can give you that. guarantees around being safe to run that program and not bring the machine down. Because as you say, if you crash the kernel, you bring down the whole machine, unlike an application, just, know, crashing an application is not great, but it doesn't affect any other applications. Well, probably could have a knock-on effect on other applications. It doesn't definitely affect all the other applications. Anne: Yeah, but yeah, but your kernel did br did give you in the olden days the blue screen of death on on Windows. it brings down your whole machine. Liz Rice: Yeah. Yeah. And, you know, it's only a couple of years ago, we had the, the, incident. won't name the company responsible because it could have happened to, well, it wouldn't have happened to just anyone, but it could have happened to some other people that happened to happen to that particular company. But they rolled out an update that caused kernel crashes and, you know, retail shops and airports and train stations. There were lot of blue screens. Anne: Yes. yeah. but yeah, yeah. It's so yes. So that so mucking around in the in the kernel has was always really, really hard to do. And rightly so. It was really, really hard. There was there was quite a lot of there there still is. I

mean at EBPF you can't do anything. You wouldn't write your app your entire application in EBPF, or would you, these days. But I don't I there's no need to, is there? Liz Rice: You could, if you really wanted to, I mean, a bit like you could, you know, hand code any application in assembler, if you really wanted to. We did a talk, I guess, about a year ago, myself and my colleague, John Fasterbend did a talk about basically how EVPF is Turing complete by... chaining EBPF programs together, we can effectively, we can run Game of Life and then that proves that it's true and complete. So I've yet to see, well, no, actually that's not true. I believe that one of the Linux Plumber conferences, somebody did show Doom running in EBPF. So I haven't actually seen that one myself. I must look up that tool. Anne: That is that was interesting. I would be very interesting to know how fast that could be made to to run. with yeah, directly in the kernel. Liz Rice: I mean, ABPF is not sort of magically accelerating. It's CPU instructions that are being executed on the processor like any other instruction. The reason why it's fast is because we're avoiding all these transitions and because we know that it's not going to hard loop. So it can be very efficient. Yeah, it's not, it's not sort of magical acceleration or something. Anne: No, no, no, yeah. Yeah, you're just getting rid of but it's amazing how much how much slowness is deliberately introduced. I mean, the friction is there for a often for a reason for for safety reasons, that kind of thing. So yeah, by doing your verification up front, you're saying, well look, I've verified up front. So you don't need to have all these points of friction, which just makes sure that everything's okay. Until until the the verifier gets hacked and they and you find a way of aid it. Yeah. Liz Rice: know, the Verify is software, you know, the Verify is software and there have been vulnerabilities found in the Verify. So, you know, we can't say with absolute certainty that it's never, ever, ever, ever, ever going to crash. But we can say that, you know, EBPF programs have been running their millions and the Verify has caught an awful lot of cases that would have crashed a kernel if it weren't for the Verify. It's very well hardened these days just through sheer amount of use. So you very dramatically decrease the likelihood of a kernel crash. Anne: It it is really interesting that in the past well actually around ten years ac actively, that the the Linux operating system has really sick had significant functionality has been added to it without it without having to like rewrite it or anything. like the containers were a completely different approach to using it's not not crazily complicated, but the the ability adding the ability to the ker to to the kernel to say, look, you you as a as a as a container can do this and you one can and do this and you know, mate made a massive difference to security and deployability and all that kind stuff. But E VPF again Liz Rice: Yeah, although, if I'm gonna be pedantic, I'm gonna Anne: Make me done. Liz Rice: say, yeah, the kernel does not have a real concept of what container is. So the kernel knows about namespaces, which was a pre-existing construct in the kernel for this kind of idea of separating what different processes could do. So you might have, I don't know. mount namespace so that different applications could either have or not have access to different mounts. But from the perspective of the kernel, a container is just a process and it just happens to be running in some namespaces and it might be associated with some C groups. Actually C groups got reinvented to, C groups V2 was developed, I would say in large part so that it... fitted better with some of the shortcomings of the first V1 really. It didn't fit so nicely with the concept of containers. But you can't go into the kernel and say, tell me about this container. These days, you can say, tell me about this C group. And often, that C group will map to a container. But that's not strictly the kernel knowing about the container. So if I'm being pedantic, kernel doesn't know. Anne: No, no, it's totally reasonable because I think what you're what you're saying, if I understand you correctly, is the containers totally I mean containers have completely revolution de revolutionized dev and ops since they came in. But actually, it wasn't a massive change to the the to Unix. You know, it it it it was a little Liz Rice: Hmm. Anne: bit of a tweak and a bit of a of a different way of using functionality that was already there. EBPS Liz Rice: I can almost hear your audience going, jails, jails. Anne: Yes, indeed. It's like this there's not it's not like it's a revolutionary new thing. So EBPF is another major change that's completely revolutionizing operating systems, but it's more of a significant change, is w I think what you're saying than containers were. Liz Rice: I wouldn't want to say whether one is more significant than the other. And also it's a little bit arbitrary whether we say this is a kernel feature or not a kernel feature. You know, it's as far as the industry is concerned. Containers are massive. Containers dramatically change the way we do everything. And then eBPF, because it's sitting in the kernel and you know, I just said a minute or two ago that, you know, from the perspective of kernel. A container is just a process and it can see all those processes are managed by the one kernel in that machine or virtual machine. So the kernels got visibility into all of these containers and control over what's happening, control over all these processes. So in some ways containers and EBPF really came hand in hand and EBPF to be a really great way to instrument containers, because you only have to instrument the kernel once and you can see into all of the containers on that kernel. Now, interestingly, particularly because of agent security, we're seeing increased

use of additional isolation. between containers. So things like gVisor and Kata containers, which have been used to provide this stronger security boundary around workloads, stronger than a sort of traditional container. So with these kind of sandboxing, it's like the API is still a container. You're still running a workload from a container image and it's just that the kind of runtime sandboxing around it is different from traditional kind of containers with namespaces and cgroups and root directory. So you might have basically a virtual machine per container, or you might have a sort of different isolation layer between the application and the kernel. So these are all kind of changing the relationship between eBPF and containers and their workloads. It's an evolving field, let's say. Anne: Hmm. That is interesting. I mean it I because I I I remember we always used one of the issues with containers is it's an incredibly useful tool with within a trusted environment. But it d it's not a VM. It doesn't completely seal off the the what's inside the container from what's outside the container. So you can't you you couldn't use it for multi-tenancy, for example, because you couldn't it Liz Rice: Mm. Anne: protect your t your thing that was running in your container from some evil thing running in a container next door to it. but you're thinking with Liz Rice: Mm. Anne: eBPF, it's are you thinking is it moving in that direction? Liz Rice: I guess one thing I'm going to say that's interesting in this interesting development here is a project called Agent Substrate, where actually the problem they're trying to solve is less of, well, it is about isolation, but it's also about the fact that if you're running an agent, it's actually waiting for quite a long time. know, it might be waiting for a response for something from another agent. You know, it might be. spending quite a lot of time just hanging around waiting for an answer. So the idea of this agent substrate is to say, we could schedule agent workloads into pods without having to spin up a different pod for each. It's like another layer of abstraction. Virtual machine containers scheduling agents within those containers, those containers then being scheduled onto machines. Anne: I mean okay, I'll I'll think about that one. I'll Liz Rice: You Anne: have to think about that, how that's how that fits in. but yes, so so fundamentally coming back, we so we talked about the issue with security. So so there is undoubtedly a huge issue with security, which is you find the the the exploit, then the exploit is being used. unbelie it's like as say as the the the the Mandiant report last year was saying it's things are being exploited before they've even been reported, you know, before anybody knew you knew about them there so you need to kind of to get things the to get patches out and to get your system secure in a in a p software patch way, you had to wait for somebody to report it. Then you had to get it fixed. then you had to apply the patch to it to all the machines that might be affected by it. That is a long, long process. And it's time we just don't have anymore. So one of the one of the traditional fixes to that was to get a workaround out quickly so that you kind of go, well, we know what the exploit is, we just don't know how to fix it yet. So there's always been this kind of like, is there a way that we can get a a mitigation out? And And this kind of E BPF way of doing it is is a different way of getting that mitigation out and putting a shield in place while you get the the pa the patch written and deployed. Liz Rice: Yes, and being able to roll out that mitigation without affecting the running applications is, I guess, a big advantage over just rolling out a software mitigation. You don't have to apply any patching. You just apply the shield, and it takes effect immediately over your current workload or your current infrastructure, which I guess probably might be a good... that kind of lead into talking about HyperShield perhaps. Anne: yeah, yeah, yeah, total. Go go ahead, 'cause I was about to say, what Liz Rice: Yeah. Anne: do you well, you know, are there open source, are there products? What's what are the ways in which an enterprise would get their hands on this this thing? Liz Rice: Yeah, so one of the, I guess, visions that Cisco had when they acquired us at iSurveillance was the ability to use Tatragon to provide this kind of runtime protection to Cisco infrastructure. So there's a number of Cisco smart switches now that have this feature called HyperShield Live Protect, where Cisco can provide shields to customers that they can then apply to their switching infrastructure to protect against a vulnerability in those switches. So you don't have to reboot the switches. I mean, at some point you will want to apply the patch, but you can do it at a planned time, you know, under your control. You're not sort of forced to rush into it because of a... a new kernel vulnerability that might be affecting those switches. Yeah, and that's proving to be a really popular feature. Gives the teams some fun to... So we have the Talos team in Cisco who do threat intelligence. And so they have a lot of experience and knowledge of what these vulnerabilities are and how they're exploited. And then... our team who are really expert in EBPF and basically combining that knowledge to build really pinpoint shields so that we can provide those shields to customers with confidence that it's not going to break something else. yeah, that's Cisco Hypershield Life Protect. Anne: So so Hypershield is is is basically just to just to protect Cisco stuff routers, Liz Rice: Yeah, it's being rolled out increasingly to more and more products. I would say come back and ask me in month's time about new environments where life protects can be used. Anne: that sounds that sounds good. But Tetragon, so Tetragon was an open source product. this. That's the question I was

gonna ask. Liz Rice: is still an open source project. Anne: Is it still an open source product product? And that's for that's more kind of generic in that that's pre that wraps any old application, presumably. Liz Rice: I mean, yeah, you can run Tetragon on any device that's running Linux. I really want to see devices all running Tetragon. There was only, what, a week or so ago, there was this exploit in, I guess, French supermarket freezers being Anne: God, yeah. Liz Rice: hacked and having their temperature control. They are running Linux apparently. One of my colleagues was looking into what version of Linux they appear to be running. And it's not that modern, but it is just about modern enough to have some EVPF capabilities. Anne: So, so I'm gonna stop roll back for my for for anybody who hasn't noticed this happening in the last week. I think it was it was thousands, thousands and thousands of supermarket refrigerators across France and a little bit in Germany as well, I think, were hacked to raise the temperature in the fridge to 50 degrees C and everything, you know, thousands and thousands of of it crazy amounts of food spoilage, but it's not just that, it's like You'd be lucky if you ever make that fridge work again after you've got had all that horrible stuff trailing into the, you know, it's it's having a fridge run at 50 degrees C that's supposed to have frozen food in it is a disaster area. I have had that happen to me. but but all of that, yeah, and and the the reality is people wouldn't have had the ability to make that patch in the amount of time that would probably between people realizing that the thing was starting to be exploited and and Liz Rice: In fairness, the particular attack was not a software vulnerability. was poor configuration. was username, admin, password, today's date. So, Anne: Today's date, wasn't it? Liz Rice: you know, no amount of vulnerability patching is going to prevent you from being attacked if you have like loose username and password access like that. But you could, for example, have... guard rails around what executables could run, you could have guard rails around what parameters are allowed to be set. Yeah, I, it feels to me like we could have some useful application of eBPF in a lot of devices. Anne: It's really nice to see a co a a really significant change in the operating system in in Linux arrive quite late in the day. You know, you kind of think, Linux, what exciting things are gonna happen to Linux? But actually loads of exciting things are still happening to Linux. Liz Rice: Yeah, I think in fairness, know, EBPF is not the only area in which Linux is developing. You know, there are lots of different subsystems that have a lot of interesting things going on. I just happen to know more about EBPF than any of the other ones, but I'm sure there are plenty of kernel developers doing, you know, really smart things. Yeah, all across the operating system. Anne: Well, if any of them are listening, come on come on the podcast and you can tell me about all the other the the more exciting things. So that's a challenge for you. Are you doing something more exciting than EBPF to the Linux kernel? Come and tell us about it. Liz Rice: I didn't say they were more exciting than EBPM. I find that hard to believe. Yeah, game on. Challenge. The challenge is there. Anne: So so that's been really interesting, but I I well I would we've been talking for an hour, so this is a good place to s to break. so because we've said so much, we've talked about so much stuff. So I I'm now gonna say thank you very much indeed for being on. What a fantastic episode this was. Thank you very much. Liz Rice: Thanks for having me and it's been fun. Anne: Thank you very much. And thank you to all our listeners and viewers. And hopefully, I will catch you again on next week's Asynchronous and Unreliable. Thank you very much.