TRANSCRIPT 🔗
WEBVTT
1
00:00:00.000 --> 00:00:52.000
Two bullseyes and one is labeled Compliance, one is labeled Security. I can be bullseye in both of them, but they don't necessarily overlap. Right. So I can be compliant and have a big security vulnerability. I can be more secure and not meet my compliance requirements. Right. So you have to consider both of those things when I'm doing them. So I can't just be compliant and check the box. I can't tell how many times I've walked into a site like, oh, we're, we just passed our NERC SIP audit or whatever thing. So we' like, okay, but how many things are in that? Like, if I go to a power plant, only a fraction of the actual devices are part of the compliance program. So what about the rest of the devices that I can get in from one of those devices that are not part of the compliance program, and then I can potentially impact. So there's all these things to consider as I'm looking at them. Right.
2
00:00:52.000 --> 00:01:10.000
You're listening to Protect it all, where Aaron Crowe expands the conversation beyond just OT delving into the interconnected worlds of IT and, and OT cyber security. Get ready for essential strategies and insights. Here's your host, Aaron Crow.
3
00:01:10.000 --> 00:22:50.000
Hey, everyone. Welcome back to Protected All Podcast. I'm your host, Aaron Crow. Today I want to dive into something I've probably talked about a bit, but it's something that as it gets closer to time, more and more folks and organizations are going to be fighting for. And honestly, it's something that I think in general, we've been kind of dancing around and not actually hitting it directly. And I'm talking about network monitoring in OT environments, specifically east, west traffic. I think we've done a lot. Organizations have done a lot as far as north, south, where we're talking about, you know, I put a perimeter, firewall, even firewall. And if I have a segmented environment segmenting or monitoring that traffic, that's leaving a certain environment or zone in a, in an area. But it's that east, west, right? So things that are on the same switch, switch in the same subnet in the same network. How are you monitoring those things? And in an IT organization, you have, you know, NACs and things that are monitoring. Which Mac address is this? Does it have a policy? Does it have antivirus, like all that kind of stuff? We're not necessarily doing those in our OT spaces. So what's happening inside of your nerxip calls it Electronic Security Perimeter. There's a blind spot, right? There's usually this monitoring or data that we see from the firewall. But everything behind that is kind of a. Like a dark hole of just not exactly knowing. Like you'll get some stuff that's. That's reporting out, but you're not really getting everything, right? So the problem is, is that's your most critical stuff, right? That's the critical systems that are in those spaces. So NERC with electric providers, power utilities, transmission, distribution. Nerczip15 is really focused on attacking that, right? And by attacking, I mean going after knowing. That's a gap in our security posture. And how do we dive into that? What is that? North, south? I call it a trap of, you know, feeling like you've got monitoring or what you do, but you're limited, right? So when I go into most OT environments, power plant substations, manufacturing facilities, and I ask about if they have monitoring, let's say that, you know, obviously many don't even have anything, but if they are doing any network monitoring, nine times out of 10, you know, they're monitoring at that north, south, right? They're only getting things that are traversing through a router, through a firewall, and that's the extent of it, right? They've got ids, they've got, you know, what's going in and out of that esp, and that's. That's a different section in narc, you know, monitoring at that esp, that electric electronic security perimeter. And it's been the standard in power utility and other manufacturing. Manufacturing and other OT spaces as well. But what happens when things are beyond that? Right? What happens when something doesn't go through that path? Like, I can't tell you how many times I've walked into these spaces. And I'm not specifically talking about nercset, but just in an OT environment. Environment where they have a firewall and they are monitoring at that firewall, but then there's a path around it, like they're bringing in a laptop. They have remote access that they directly plug into the OT devices that bypasses all of that stuff. So you're really flying blind because you don't know those things are happening. You're not monitoring those things. You don't know to look for those things, right? And this isn't theoretical, like, we've been watching this play out for years, you know, Ukraine pipe dream. There's all sorts of things. Examples of attackers who have, you know, knocked on the front door and gotten turned away, but found other ways to get in. That's the risk of, you know, living off the land techniques that these guys are using and, you know, using, as we know in these spaces, in OT spaces, we don't have encryption. We don't have a lot of those things that you would have in an IT world. So if I'm inside the protected, you know, space, I have carte blanche to do what I want. I can send a command to a PLC and it is going to do what I ask it to do. It's not usually asking for credentials, it's not asking for authorization. It assumes if I'm speaking its language and I ask it to do something, it's going to do it. So there's one side of me that says, or a lot of arguments in the space of, hey, we need to encrypt everything, we lock everything down. I don't necessarily disagree with those things, but there's a lot of reason why that's a problem in an OT space. But at least we need to be monitoring in these spaces to understand that, you know, we know that the MP3 Modbus, a lot of these things are open
Load full transcript
protocols. How do we know that somebody's in that space and these commands are valid? Your IDS is not going to be triggered by, you know, somebody sending a correct command to an hmi. It's not malware, it's not virus. It is actual commands that should be sent. It just isn't being sent by somebody that should be sending them. And if you're not monitoring that east, west communication, that internal network of things that are plugged into switches, things that are in those spaces, engineering machines, engineering workstations, et cetera, PLCs, HMIs, historians, you know, engineering workstations, you're not going to have visibility into knowing what those things are, right? So you can't tell the difference between operator doing their job and a bad actor, you know, doing something they shouldn't be doing. So specifically, Nercset 15, for those of you in the power utility space, you're probably very familiar with it. Those of you that are not in the power utility industry, NERC and ferc, there is a regulation around, you know, the power utility industry, nerc, SIP is kind of the implementation of those things. And the, hey, these are the rules that you have to follow to be compliant. Everybody knows what compliance standards are, whether pci, whatever. It's very similar to that, right? So in that it is a compliance standard that is held against these organizations, you know, those things. They are a good start. So let's start there. A lot of OT spaces don't have that regulation side of the arm. So you look at water utilities. There isn't any regulatory body that is pushing, hey, you must do these things kind of same thing in manufacturing. Although there's, you know, I 270001, there's 62443 in the UK, there's N2, there's, there are standards out there. Obviously you've spent a lot of time in nerxif, so I have that as an example. I also think it's pretty far advanced as far as, you know, the implementation. It's been around a long time. They've been, you know, constant improving it. So but to be really clear, like there's a lot of noise, like it hasn't been implemented yet. There's a lot of people. Compliance sometimes can be gray. Meaning I interpret this to mean this, right? So there's going to be a lot of vendor driven confusion of hey, yeah, our product will do this, it'll do that and it's going to have to be, you know, kind of played out with auditors and with the regulatory bodies to say, hey, this is what we're looking at. Does this fit? Right, so what are the requirements? The requirements are saying that you need to implement network data feeds. It's going to have to use like a risk based rationale to monitor network activity inside your esp. So you're going to have to really look at connections and devices and communications. You're going to have to be collecting data that gets into that visibility of what's going on behind the firewall on that, you know, that control network, that OT network, you know, and many times at these spaces there's multiple of those networks. I've got various switches spread across my environment. I have to know if somebody's plugging into them, if there's malicious activities going on, those things. It can get really complex, really, really fast, fast. As of right now, it is only being required at medium and higher facilities. So a low impact NERC site today would not need to have this. But it is really good practice. It's just going to be, you know, you look at some of these, I won't call them out by name, but you look at some of these larger, you know, utilities that have thousands of substations across the country, you know, hundreds of power, you know, power generation sites, those are, it's going to be really tedious to deploy this at scale across your organization. And then even after it's deployed, how do you monitor what does response look like? It's not simple. That doesn't mean that we shouldn't be doing it but it is absolutely not simple, right? So you need to be evaluating, you know, anomalous activity, what actions required. You're gonna have to document those things. And the thing around a compliance program is I have to show proof of compliance, right? I have to show that I monitor these things. If I had something come up, what was my reaction? And I have to document those things. So there's this entire documentation side of a compliance program that has nothing to quote, unquote, nothing to do with compliance or nothing to a security and to do with showing that I'm following the rules and I am compliant with this regulation. So that's that yin and yang of I can be 100% compliant and not secure. I can be fairly secure and not compliant. Unfortunately, in compliant organizations like this, 99.9999% compliant is not compliant. You are out of compliance, right? It's 100% or nothing. So NERC is not saying that you have to, you know, do full packet capture everywhere across your entire environment on day one. They're more so saying, you know, deploy where the threats are. Think about where an advisory behavior looks like. So, you know, really deploy into your environment. Make deliberate choices of where you're pushing these, pushing, placing these, you know, network data feeds. And look at the bigger problem beyond just what is the technology that can do network monitoring. A lot of these environments, they have really old equipment. I've been in a lot of places that switches are 20 years old. You've got, you know, old school switches that, yeah, they can do port mirroring or spanning, but they only support 8 out of the 24 ports can be monitored at a certain time. So what are you doing that in that capability? Are you putting in taps? Are you replacing the switch with a more modern switch that supports, you know, full mirroring? And then on the flip side of that, I've also seen spaces where older switches, when they're doing port mirroring or spanning, they're actually using the networking plane to do that, which means you're taking away
throughput or, you know, cycles from your switch to do monitoring, which is a problem. So there's a lot of ways around that. We don't have to go 100% into the architecture of those things, but this is why I bring these things up to just talk about why it's so complex to really dive into these things. So, you know, SIP 15 is, is really applying to high impact and medium with external routable connectivity. If you're in a high impact facility, it is mandatory, you know, good Thing with is there are no high generating sites. It's going to be more control centers and things like that. But there are mediums, there are medium generating sites. Obviously there's a lot of substations and control centers, so backup control centers. You know, it sounds like there's a lot of this in the future. Just be ready. It's going to be deployed I would assume everywhere. So you know, what is the reality check is, is that I've not seen too many organizations sold on a tool that they think will solve the problem. INSM and a network monitoring in these OT spaces is not easy. You know, I was a product vendor every product in the space. It's not only a product problem, there are absolutely products that will do this. And again, I'm not trying to solution or architect this but you know, the whole point is, is that you are going to have to find a product that understands OT protocols that can be in these spaces and determine what good looks like so that I can communicate when something bad happens. Right. If you look at baselining specifically talks a lot about establishing a baseline for your network traffic in this nerczip15 it's, it's one of the requirements so that you can detect anomalous activity. Right. But in an IT environment there's plenty of tools because we know what HTTPs looks like, we know what you know, SSH looks like. Like those protocols are. There's so many products that do those things in an O environment. Obviously we have a lot of OT network monitoring tools out there, but there's not as many as there are in an IT space. So you've got, you know, obviously Those OT protocols, DMP3, Modbus, TCP, you know, Profibus, Ethernet, IP, et cetera, et cetera, et cetera. And each vendor is going to implement things in the way they do things. And the PLCs are a little different. So it's not just the protocol, it's about the function codes that are within them. Like there's just so many things that go into those things. And then the infrastructure challenge that I spoke about, you know, you to a power plant or any of these spaces, I may have 200 switches deployed in all over the place. And there is no, you know, fiber backbone that I can use to do spans or get all this data back. So there's going to be a lot of infrastructure that goes into actually deploying these things, managing switches, span ports, network taps, you know, how am I getting these things back and monitoring them. And then you talk about compliance. I'm not going to put them on a note an IT network
backbone. So it's got to be dedicated. So it's just like all of these problems just kind of compound on top of each other. Now I've implemented this in scale way before. This is a requirement at power plants and it is absolutely doable. But it is, you know, hardware dependent. The infrastructure, the architecture is a little complex. Once you get it, it's there. Right. The cool thing is, is once you implement this type of, you know, monitoring on the, on the network layer, then I can put in a tool that I want. Right? So we, we did this too. We put spans off of every single switch going into a attack like Gigamon or Garland or, you know, any, you know, Ixia, any of those you network taps. We really did that so that we would not break segmentation. But then you go into some kind of port aggregator, right? So I've got 200 switches, we did two spans on every switch. That means 400 spans, right. Going into a port aggregator. So then I could just have say, a tool port. Hey, this is going to, you know, network monitoring tool. This is going to a sim. Whatever you're doing with it, then you have a lot more flexibility down the road to change tools, to be able to add, remove, change without. Once that network architecture layer is done, all the rest of it becomes a little bit easier down the road, right? You're not locked into. Well, I have to use vendor A, on the, on the tool side as far as the monitoring, because I've already implemented, it's too hard to change them and. Or I have to deploy a hundred sensors out in this in the field. If you do the architecture, in my perspective, you should do that architecture at the network layer so that you have, have. You don't have to have a sensor at every switch or at every, you know, substation. Like you could do that a little differently. You're still going to have a lot of implementation, a lot of architecture, but it's going to definitely look different. In an IT world, you're probably just doing a span on a core switch and feeding all of that to one place. It gets a little simpler. OT is, is definitely a little bit more complex just because of the risk of segmentation, the risk of the downtime that that can happen when those things happen, right. And then you do all that like we've monitored the network, we've got the tool, we've set it up, we've tuned it, we know what good looks like. We're monitoring for those things now. The people problem comes into this. Like, who's monitoring it? Who's looking at the dashboard? Like, are there people that understand what good looks like? What are their, you know, playbooks look like, what are the runbooks look like? Right? So you can have the best tools and architecture in the world, but if you don't have somebody sitting in the chair that's looking at it that knows what they're doing, then it's going to be difficult. Like, you're not going to get the same value. Is this normal DMP communication versus malicious ones? Like, and again, some of these are not malicious DMP traffic. Some of this is going to be. We're not an outage. And I know that this site would not be doing those types of commands during this window without a window. So did they trip? Did the unit go down? And now they're doing these things. But you have to understand that at a different level. And this is not a network architecture, it's not a cyber security understanding. This is where I talked about it a couple of weeks ago, where the difference between bringing in an operator, bringing in somebody with the engineering, the operator. I talked about it with Dean Parsons about how important it is to have those teams be commingled and really understand from an operational level what these things are doing. Not just from a cyber level, what good looks like, but also from an operational level, because ultimately that's the real thing that you're doing. Otherwise you're going to be drowned in false positives of, you know, and you're going to miss the good stuff. You're going to be drowning in the things that you think are bad or not really, and you're going to miss the things that really are bad, but didn't, you know, raise a flack, you know, so kind of transitioning that to compliance versus security, which is. Is the real conversation here. Anytime we talk about Nerc sip, there's always that compliance and security lens that you look at. One of my, my previous bosses, he used to have this, this drawing. I think I've mentioned it before. He gives me a hard time about it because he loved the art, the drawing. I hated it at the time. I don't hate it as much anymore. I never hated it, but it communicated the point anyways. The document or the presentation was two targets. So like think of a. An archery target or whatever, right? So a bullseye. Two bull. And one is labeled compliance, one is labeled security. I can be bullseye in both of them, but they don't necessarily overlap. Right. So I can be Compliant and have a big security vulnerability. I can be more secure and not meet my compliance requirements. Right. So you have to consider both of those things when I'm doing them. So I can't just be compliant and check the box. I can't tell how many times I've walked into a site like, oh, we're, we just passed our NERC SIP audit or whatever thing. So we're good, good like okay, but how many things are in that? Like if I go to a power plant, only a fraction of the actual devices are part of the compliance program. So what about the rest of the devices that I can get in from one of those devices that are not part of the compliance program and then I can potentially impact. So there's all these things to consider as I'm looking at them. Right. And like I said earlier, continue say it. 99.999% compliant is not compliant. 100% compliant is not necessarily secure. So you know, the standard of what FERC and NERC of have determined minimally acceptable posture is not necessarily the best thing that you should do or the end goal of what it is, NERC SIP
compliance or any of these compliance program from my perspective is the bare minimum. It is the bare minimum that you can do. It should not be your end state. Like that should be the, okay, we know we have to do this. How much more are we going to do? Where do we feel comfortable? From a risk, from a business risk perspective and from an understanding my environment, what else do we need to do? That doesn't mean you have to go from 0 to 100. But the goal in my perspective has never been that NERC SIP is the end. Like I do that I'm done, right? So NERC SIP 15, you know, organizations really approaching this INSM network monitoring, deploying a tool, configuring some, some alerts, you know, understanding risk based, understanding your architecture. But again you really got to really make sure that somebody understands it's reviewing the data. What does good look like so that you can start alerting on those things. So what is that? Actionable steps, you know, make sure that you've assessed your network, you know, where your systems are in these spaces. You're going to have again narcissip, you're going to have already had, you know, understanding what your assets are, what your, you know, high impact systems, medium impact systems, external routable communications. But bigger picture is really understanding where those segmentations are. What am I going to be monitoring for? And as you're doing, as you're installing this plumbing for all of these things, it is so much easier. It's just like when you're, you know, running wiring for a house. If you need one, run two. Like if you're going to go in this area and you're saying, hey, that's not part of my compliance program, so I don't need it, but it's sitting right next to it. Monitor it. It doesn't have to be part of your compliance program. You can have it in a different environment. Maybe you don't even turn it on, but have the piping there. If you're going to run fiber, run extra fiber. Don't run one pair, run five pair. Because the cost of the fiber is insignificant compared to the how much it costs to run it. Right? You get, you, we know in these spaces you're going to break fiber, you're going to need extra stuff. You never know what you're going to need it for, you know, so really look at those things, you know, assess your network architecture to make sure it's ready for monitoring. Look at the better way to do these things. How can I monitor inside of these spaces? Do my switches support spam ports? Like all these types of things that we talked about. And then the bigger picture is, is baselining what does good look like, right? So you're going to have to understand what these, you know, control system, what the PLC's like. You're going to need an expert in all of those spaces. I go to a site that's got an Emerson and a, and a GE Mark 6, you know, third party controls. I'm going to have to understand each one of those, what good looks like. And I'm going to have different things at every site. The blue boxes, seals like you. What does good look like in all of these spaces? That's going to be the bigger picture. The harder part. Again, we're not talking about only compliance. We're talking about actually securing these spaces and knowing what good looks like so that I can lower my risk and protect these spaces. Visibility is amazing. I'm excited about where this is going to go and how this could take us further in the space. But you know, I also see that this is going to be a big hurdle that's going to take a lot of work to get to. So anyways, all that said, love to hear anybody's perspective on Nerxet 15. How have you been implementing it? What tools are you using? What are the architecture strategies you're looking at? You know, are you replacing switches, are you adding ports and taps and all the things definitely reach out. Let us know. It's definitely something I think is going to be fun and exciting over the upcoming years and it'll probably spread to other, you know, critical infrastructures beyond just nerc, SIP and and Power Utility. All right, Appreciate everyone listening. Definitely dig in and let me know what you think in this space.
4
00:22:50.000 --> 00:23:06.000
Thanks for joining us on Protect it all, where we explore the crossroads of IT and OT cyber security. Remember to subscribe wherever you get your podcasts to stay ahead in this ever evolving field until next
5
00:23:06.000 --> 00:23:15.000
Sam.