KOPYA NG SALIN 🔗
Hey, everyone, welcome back to the show. Today, I want to talk about addressing the symptoms versus addressing the root problem. And I think in application security, or when we talk about secure development, this is something where a lot of times we address the symptoms, but we never really take the step back to address the actual root cause of what's causing those symptoms. And today, I want to actually talk about vulnerable third party components. This is something that has been kind of brought to the attention a lot more in the past few years, made it into the OWASP, top 10. And it's something I think everybody struggles with, we never know when we'll have a vulnerable third party component, because until somebody actually identifies a vulnerability, we just assume that we're good. And then on top of that, if there is a vulnerability identified, then we also run the chances that we're probably not even using that feature. Anyway. So vulnerable third party components are a really interesting aspect, when we think about secure development. Because there is a lot of unknowns, we may know that there's a vulnerability there. But the actual knowledge of do we use that piece and are we vulnerable, can be difficult, which, in the end, ends up adding a whole bunch of extra work and a whole lot of time for us to try to figure this out and address this stuff. And so this is where I talk about addressing the symptoms. In this case, in a lot of places, what we do is we address that symptom, we know that there's an issue of vulnerable third party components, right, that's the symptom, we have a vulnerable third party component. And so most places have some sort of process in place where we're going to identify these right, we're going to scan them all the time, whether using some of the common commercial tools, maybe you're using a free open source tool. But basically, the way it goes is I'm going to scan my repos or I'm going to scan my packages, and I'm going to look for all the dependencies, and then I'll look at their dependencies, and we'll see if there's any known vulnerable components within these right. And that requires having some sort of CVE out there that says, hey, somebody has found this, they've reported it, I remember requiring this to be a reported vulnerability. But we know that there's a vulnerability in this component. And so now every time I scan my package, it goes and looks up to say, hey, there any known vulnerabilities for this package, and then we send it back. And then of course, the difficulty of trying to rate this and understand, you know, is it a low? Is it high? Is it critical? Love these systems, right, from a generic standpoint might say that it's critical vulnerability, but is it really of critical vulnerability? And a lot of cases, probably not, we've had a few that definitely were. But not everything is critical. Even if the CVE listed out, as you know, high or critical, it depends on your environment and how you're using that system. So there's a lot of extra work that goes around that, of course, to try to avoid that, you know, if we could just patch everything right? Wouldn't that be fantastic? But that's a little bit more difficult to get into as well. So we've got this process of we identify this thing, and then what do we do we go create tickets of some sort, send it into the backlog of the development teams. And then you know, they have to go back and address this issue. Now, as we've seen, application security changing a bit over the years, you know, some places were not as separated in that case. So you're seeing things like GitHub or other tools like that, where they actually will identify when you go do a PR, and it'll check everything right then and they'll let you know, or, you know, it just randomly is checking within your repo to be able to alert the development teams right there within GitHub, right? It'll create an advisory to let them know that hey, there's a known vulnerable component in here we got to address this. And the more we embed it, the better. But are we addressing the root cause is really what I want to try to get out here. And that root cause, ultimately, in my mind drops down to do we have proper patching built in to our development process? And we have to consider that as a developer from way, way, way back. You know, I don't think we really thought about that. And I don't think a lot of that has changed coming forward.
So how do we get to the point where we build that maintenance piece into our systems, not saying that every single release were 100% patched, you know, and that's the way it's going to be, there are some of course, downfalls to trying to apply the latest and greatest, as soon as it comes back out. We've seen this with some third party components when a malicious user takes over an account or they hack some third party component, right, and then they put their malicious stuff in there. And sometimes that takes a little bit for the system to realize that. So if you're downloading the latest thing, as soon as it's published, you have a little bit of risk there as well. Because there's time for somebody to be able to identify that. So there's risks both ways. And we want to make sure that we're trying to address this. Now, when we talk about root cause. A lot of times that is just the fact that we don't have really good patching processes in place. Within our development lifecycle. We look at being able to update certain things, right, we work on the code that we're updating, but we don't think about the dependencies that are seeing in our application. How many times have you seen a application that's still running, like the first version of jQuery out there, or application is still running really old versions of Java frameworks for their stuff, because that stuff is just the foundation it works, it's there, I don't have to think about it, you know, and then you don't do that forever. Or maybe you stopped
Load full transcript
development on that for a while and you come back to it. Now, it's very difficult to try to get it back up to speed when you start getting, you know, three, four or five versions back from where it was. So I think it's important that as we're looking at things, and we'll talk about just the third party components right now, but understanding as a development team, what do we need to be able to do, to be able to embed a process to make sure that we're updating our components in a reasonable timeframe, right, not years, not even months, probably. But as we're going through doing our development or doing our sprints, or however, you're running your system, making sure that, hey, I'm going to make sure components are updated. When I go to push this new PR, or I'm getting ready to do a new build, let's look and make sure that we've got the up to date components within the system. And maybe it's a week behind, maybe it's two week behind. But it's not months behind, it's not years behind. It's keeping it updated. But to do that, we have to make sure that we understand that there's a cost to doing this. What happens when we pull in that latest component, and it's a breaking change? Do we just ignore it and say, Oh, I can't move to that, because it breaks my system? Well, that then is going to cause a problem a little bit further down the road, because we just let that keep building and building and building. And then next thing you know, or three years down the road if we've never updated it, and now it's a huge update. And now we really have to figure out how to do this. So from development teams, we really have to be able to start understanding, you know, how can we handle making sure that we're paying the right attention to these components, and that we're keeping them updated, so that we don't get caught way behind. And then what that ends up doing is it turns the scanning that oftentimes the apps get the app SEC team is doing for these dependency things as just a secondary feature of okay, we'll catch something if it's there. And if it's something super serious, we can alert you. So maybe that can get patched quicker. But ultimately, if we have a good patching process and update process within our code system, then we shouldn't have to worry about that. Because it's going to be updated within a limited timeframe anyway, but at least we're aware of it and we know we can track it. And we can verify that yes, we know we have this, here's what's going to be updated, but it becomes less of a backlog issue. And more of a hey, this is what we mean typically do within our systems. The difficult part I think another thing we want to think consider here is around Some apps that maybe we kind of complete with or we're not doing a lot with, maybe we've pushed out the major thing, we've got it to a certain point, it's like, alright, this apps running, it's running smooth. And then all of a sudden, we don't consider maintenance for that app. Anytime after that, right? It's running, it's good, okay, the whole team has moved over here. And if for some reason a user reports a bug, then you know, somebody will go back, and when it's appropriate, they'll go back and fix it. But we got to remember, right? Vulnerable components can pop up at any time, we scan daily, probably on this, so they can pop up anywhere. And so we have to be able to address that. So an app really is never done, which should be common sense anyway, but the app is never really done, we have to have a way to be able to keep these components updated. So that way, we're not running into an issue where something becomes stagnant. And now it's so much more difficult to update. I think a lot of times, we overlook the the true cost of building applications. And when I say that, I mean, the maintenance costs, I'll do another talk about the security costs, all those falls under security. But the maintenance costs of being able to maintain an application, you know how many times you go out and do a WordPress site and just stick it out there. And then all of a sudden, you know, five years later, it's still that same WordPress site sitting out there, that's never been updated, because you didn't turn on auto updates. Now, fortunately, platforms like that now have automatic updates. So a lot of the stuff will automatically update for you. You don't have to go in and do a lot of stuff. But I mean, I know, you know, running some sites like that, you know, I get from my host, hey, we're upgrading PHP, we're upgrading this. There's maintenance to keep it going. That we have to consider how do we handle that going forward? So that way, when we get this vulnerable known component? It's not a surprise. It's not something that's out of our cycle, we understand? Yep, we got to be able to do that new version of dotnet is coming out, you know, or a minor release? How do we make sure that we can stay on that and stay updated with that stuff, to be able to keep going forward? Again, we know there's a cost to that one more time to be able to update the stuff but to the testing time, right? How many times are these breaking changes? When we go update that? I mean, that's a whole nother dilemma of trying to figure out, how do we get components so that way, they don't have as many breaking changes. So that way, it is easier to be able to update those systems without such a mass regression test to make sure that nothing broke. And it's going to cost us all this time to fix this. But again, it's going to help us understand it, if we build that as a root cause if we build up our systems and say, We have to be able to understand how to patch and this is a development thing, this isn't a security thing. This is a development thing of how do we build this process. So we can patch these systems on a timely basis that it's just a recurring thing. It's just part of our process, when we want to go release, we're validating that we've updated our components. If there's updates available, right, there's not then obviously, we need to backlog the issue. But we need to get that next time, we need to be able to identify that, yep, there's updates available to these components, we should be installing these updates. And so building that in and putting that on the development team to say, hey, this has to be part of the process, what's your patching process? And let them figure out the best way to address this. We're saying it's got to be done in a certain way, you know, this is just part of building an application, good patching? How do we get that going and let them drive that, right, let the developers drive the solutions to these problems. But we got to switch from just addressing the symptoms, which we do a lot. And I'll talk about that I think more this year. We're addressing symptoms, we're not addressing the root cause, we need to take that step back and say, okay, that's fine if we want to scan, but we're doing the same thing over and over again, it's this vicious cycle, I'll scan it, I'll give you a ticket, you work it, and then you know, we'll validate it, I'll scan it for you. We just kept going in the same circle without ever coming up with a solution if you know what we wouldn't have as much of a problem. If we had a better way to be able to make sure that we're we're patching and we're keeping updated every time we do a new release. And we're making sure that even if we don't have releases coming up that we have some way to be able to maintain those packages that were done. We don't really we're not doing a lot of active development anymore. But we still know we have requirements for development that can Comments are going to need to be updated, vulnerable or not. We need to be able to keep things updated stay on current frameworks. So what's that process look like? But again, let the development team to find out what that should be looking like. So that way, it's their processes they own that it's going to fit into their workflows versus security, trying to cram something down. There's roads, oh, you need to do it like this. Let them define that stuff. So kind of an interesting topic. I thought today, I was thinking about this morning, I saw LinkedIn post and you know, it kind of sparked thinking about the whole root cause piece. So I hope everybody enjoyed that if you have thoughts on this, please feel free to share them with me. I'm happy to read through them and comment back I'll post you know some information. I'll post a blog that goes with this as well. But thanks for listening and you know, any questions feel free, send them to James at developsec.com and I look forward to seeing everybody on the next episode.