TRANSCRIPT 🔗
WEBVTT
1
00:00:00.000 --> 00:00:45.000
They were pulling drives out of one rig just to keep another rig running. That's the reality. Patching Advice walks into offshore Matt McLendon and Josh her join me on rigged cybersecurity, million dollar buttons, security stacks full of gaps, and what actually works out there. Hey everybody, thank you for taking time joining me on another episode of the Protected all podcast. I'm always excited to be able to talk to new folks that I've never met before. I get the opportunity to speak on a lot of different things. I'm excited for today's conversation. I told the guys today that I use the product in the background. Of course you can't see it because it's virtual, but it's running on systems in the background so we'll dive into the thing. So Matt and Josh, why don't you guys introduce yourselves, tell the audience who you are a little bit about your background and what you're doing nowadays where
2
00:00:45.000 --> 00:01:24.000
you're sure I can start it off. So I'm Joshua her I've been working in tech for about 15 years, kind of all over the place. So I started in actually structural engineering software development, then kind of moved from there into the cloud engineering space and from there was kind of a natural progression into security. So it's been kind of a wide breadth of different technologies and different environments that I've worked in over the different years. But it's also helped a lot with security because you see some of these very obscure environments and use cases out there that you don't come across oftentimes in the wild. So I've really enjoyed it. Love where I'm at today and the security space has been like actively exciting all the time.
3
00:01:24.000 --> 00:02:27.000
So love mine's way less succinct so I'm, I'm not gonna. I'll try not to date myself but late 90s, mid to late 90s it like core discipline at a MSP and ISP locally in the Houston area, Linux systems administration, Windows systems administration, really gross references like IP switch products and I've physically played on like Sunspark or Sunos and and old IBM's run in Solaris and things like that and then military component electronics at intermediate level. So for FLIR and radar for about 6 years and then 16ish years in the offshore oil and gas ranging from a major OEM to consulting to deepwater drilling contractor primarily drill ships but also semi subs and jackups at different stages of my career. And then now I'm doing what they never say oil field hands can do and I'm a product manager for a software startup called Portainer, so which is awesome.
4
00:02:27.000 --> 00:02:44.000
Again, I talk about this a lot if listeners have heard me. I'm sorry but it's always amazing to see the trajectory of people's career and what got where you started from. And especially for those of us that have some gray in our beards and have been around the block a few times, the places that we've come from, where we started.
5
00:02:44.000 --> 00:02:45.000
Right.
6
00:02:45.000 --> 00:04:27.000
You know, same. Very similar to you, Matt. Like I've been in this for 30 plus years. You know, mid-90s is when I started out doing this Stu. And you know, started out at Token Ring and coaxial network design and everything from, you know, Exchange and Windows NT351 and you know, novel Netware and all the way up to AI and all the things that I'm doing today and just the vastness of all the way from mainframe to, you know, AI and everything in between. But to your point, Josh also is like there's value in that, right? There's value in and how we've done things in the past and the gotchas. Like it's not like just because a new product comes out that we haven't solved this problem before or had these issues before. Obviously we're able to do different things with new technologies, you know, and we laughed earlier. I gave an example of a use case in OT where when we first started using virtualization in OT, which was like 2015, it's not like virtualization was new in 2015. It'd been around a long time. OT was just still not in a doctor they didn't want to use it to to today we're talking containers and Docker. And again, those are not new things. But still OT is sometimes scared of those things because they don't necessarily understand it. So that's why I'm excited to talk to you guys today about, you know, again, like I said, I've got, I've got containers and things running in the background. We have containers that can run on, on switches and firewalls and there's all sorts of things that we can do today and that, that kind of bridges us into the gap. Like how are you guys approaching and getting into the OT space using this, this new capabilities. Not even new, but new for really what people are, are doing in the OT worlds.
7
00:04:27.000 --> 00:13:02.000
Yeah. So the I, I met the Portainer guys because I am approached them as a customer is how I ended up here. I, you know, used it for from a home lab Perspective for I think I was just looking up, trying to see when a particular server I built was probably like 19ish timeframe. So the reason I found them is because like I'm my prior IT experience was a virtualization focused one, so VMware certified and that whole thing I had a very large cluster in my house just to play with and learn that ecosystem at my own pace. And that was what I was used to, was just abstracting everything with VMs. And so when I started tinkering with some of the early LLM&AI&ML subjects that I was along the lines of, you know, they all tend to be Python projects at the time, you know, things like UV didn't exist and you end up running into this nightmare of like, oh, the Linux kernel uses Python too and if you don't, you know, if you goof up one command and overwrite your, your OS's environment, you know, people are writing tooling and stuff, not assuming, you know, it's in user space. Just an ecosystem rife with pitfalls, we'll say. And so one of the
Load full transcript
mitigating ways to deal with that, at least to give you some abstraction, is to go the container route and you know, not have to spin up VMs every time. And so I remember I looked at, I was doing them just from the CLI and then I started looking at like multi host and stuff like that. And I remember I tried, I think it was called Yacht. It's like deprecated, it's gone. But it was absolutely terrible. And I'm like used to loading up like people's little pet projects and them falling on their face and like requiring weeks of tinkering to get them working. And I remember I found portainer shortly after that and installed it and it worked, just did what it said it was going to do and I could get on with what I wanted to achieve. And that was not something I was used to for, you know, an open source product at the time. And then working for my prior company, ceadrill, I was kind of fed up based on some vendor quotes that we had received about, you know, a pretty simplistic scope to add sensors for the purposes of better tracking our fuel consumption offshore and tying them back into an existing control system that had, you know, spare IO and things like that. So it was like, sell us some sensors, we'll handle class and the pipe work and field cable like so all the labor intensive parts were on us. So it was like, sell us. At the time it was just six sensors because we were just doing supply side and not returns and time back to your control system, display them in the UI and then transit it through the same medium that we get the rest of our data from. And I think the quote was like in the hundreds of thousands of dollars and 12 to 18 months of delivery time. I actually think when I left it so that scope got pushed forward and I think it still hadn't been delivered to that particular vessel yet. And so I got really fed up. That was like, I'd seen it throughout my oil field career. Like people getting, you know, really taken advantage of, like, for like people that own these vessels are the maintainers of them. You know, they're not expected to do what the industrial and process control industry is maybe at least a little bit or used to be a little more tuned for, which was like localized custom solutions and things like that that they would end up owning. And people lean on the OEMs, they lean on system integrators and they get stuck. Add a single tag to a total like physical chain that already exists, but the tag's just not there. 15k for the software change, privilege, things like that. And so I said, okay, look, you know, the usual conversation is the security implications of it. And I was like, okay, so let's just de risk. How do we de risk it? Let's not touch the control system. Do we need a sensor? Okay, cool, let's get the IO too. You know, does it need a plc? Can I tie it back to conventional computing in some way? What's the most elegant way to do that? And that's when like I just through like sheer happenstance, I learned about the. The used
to be five nodes free, then turned into three nodes free. I have my own opinions on that internally too through portainer. And I was like, oh, I'm doing this, but I'm not giving up. My personal email is just horrific with like, nonsense because I've had it too long. And I was like, I'll use my work email. That'll look more legitimate. And you know, they can bug my work email regardless where I am. And I did it. And they reached out and then they started telling me about some of the edge and industrial features and focused kind of roadmap. And I was like, okay, I never even knew about this. Let's, you know, walk me through it. Let's take a look at it. And then I was like, oh, this could actually like make it to where this isn't just something we. The usual thing would be like, I build a. You Know a widget, it solves a problem. Offshore, I standardize it, I spec out all the parts, get the quotes, get the vendors and either I get drug out there or we subcontract to someone to go do the install and then you know, that's fine for one. What about the fleet? Like who you know, and we supplement with documentation and things like that. And it turned into such an easy repeatable process that the proof of concept rig, I mean I've been on that rig before but as part of the initial deployment to dataflow from some switchboards for the system that I designed, I didn't have to set foot on the rig and to do the initial setup of it I had like just hardcore knuckle dragon like offshore electrician guy that was doing the cable runs for me did like some initial boot up and like tell him just reading screens back to me and got connectivity set up and even troubleshot some, some firewall issues that were outside of what he was dealing with with conventional IT and got the system up and running. So it's, it really does both containerization and to that point like the distributed capabilities of more complex handling systems like portainer. The repeatability is massive when you can't walk to the other side of your office to go troubleshoot something or even remote into a terminal. Like these just are not axes that are usually open or people are even comfortable with. Because even in our perspective like the rig in that industry, you should be able to just have the world ending outside of it. And as long as they have like food and fuel, they're achieving technical uptime and they can be autonomous and you know, they report back. They have management onshore, both them like our side and the customer side. So it's not completely disconnected. But there's. I've absolutely been on a rig where we were without VSAT for eight days because of storms. I mean there's like a precarious scenario where you get back to like do we need to start paper logging stuff? Like what is this going to look like when we reconcile all the maintenance data and things like that. But it's a scenario where people make assumptions like everything's a building that they can just very easily go and put
hands on when something goes wrong. And pushing front loading, all that kind of effort to make things repeatable and reliable is a huge value proposition for. I mean you can kind of do it with VMs, but there's a lot of like moving parts there where you could trip up and. Oh man. And then like the time Constraint you're dealing with like, you know what level of abstraction the hardware is. You, you have like faux boot time. Like why is that a thing? And especially for something that's the hardest concept. And this was, you know, a struggle for me coming from virtualization too. And you know, bare metal sysadmin is. And I think this, I watch it in real time and it's like watching a train wreck. But the just I think people understanding that like unless it absolutely needs it, treating every stage of it as ephemeral or programmatic will make for like much better system design and approaches to everything. And so I like the drum that I always beat is one of de risking and parallel interaction. Like don't. Okay, it's a legacy control system. It's unpatched, it's not going to get patched because it's either cost prohibitive or the patch doesn't exist or that company isn't around anymore or okay, so you're not going to make the data request go away. They want the telemetry from it, they want to monitor it, you want the maintenance data from it. So what creative ways can we do that? And having seen those kind of things come down the pipe too, like the amount of conversations that start with like, we're just going to put a tap in here. I'm like, no you're not. You were not a class assessed part of this control system. You're not going to change its design and introduce like failure modes. You're just not. And even to that point, like we had people giving us assurances about them. I've absolutely seen a failure mode on a network tap where no, it didn't lose connectivity when the tap lost power, but it reset connectivity when it came on. And I was like, well there, that's a non starter. And they're like, oh, it's a bug, it's a bug, we fixed it. I'm like, why would the sample unit you give us not. You're telling me you didn't test it? And so I just don't think that people think about that stuff in the same way.
8
00:13:02.000 --> 00:14:13.000
The implication is so much larger in an OT space and that, you know, again, as an asset owner, I was in Texas at one of the large power utility generators in Texas. And I had plants from Odessa to Paris, Texas to you know, almost to Louisiana in East Texas and as far south as, you know, San Antonio. Right. And if you look at Texas on a map, those are pretty vast areas. And my office was in Dallas. You know, it could be eight Hours to get from plant to plant. So if you, if something breaks and I've got a team of six, I've got 40 something power plants, how do I support that? How do I get there in time? And I don't have a corporate jet, I don't have a helicopter. I drive pretty damn fast, but not that fast, you know, if something goes down or you know, I need to update something. That's the things we have to consider. And the other piece to this, and Josh, you hit on it earlier as well, is as cyber practitioners like these, you know, you see the, the fear of selling of, you know, China's going to attack my thing. Sure. Okay. That is, that is a probability that is not zero. It's greater than zero in my environment. But what's more likely to happen. Something's going to impact my operations.
9
00:14:13.000 --> 00:15:12.000
That's why I try and tell people is there's a button on offshore oil rigs that someone can just walk up to like no one's manning the room. That at minimum is costing that operation a million dollars if you push it. Like right. So like what is a USB blocker doing for anyone? But the, but these are the approaches that people default to because the solutions were coming from conventional IT and they don't. They're either the contextual conversation isn't happening or it's one of just, you know, visible compliance. You know, whatever it is, it's going to differ from enterprise to enterprise. But that's always something that you know, the OT guys aren't going to be super supportive of like offering ideas when like that kind of thing's being mandated. And they're so they're the ones that got to walk around and stick them in. And then they're like I have 300 of these removal things. Where, where do these go? Because every blocker kits like four, four blockers and a removal tool. It. Yeah. So not that security theater doesn't exist, you know, in conventional IT spaces too, but it's particularly frustrating.
10
00:15:12.000 --> 00:16:19.000
Different. Yeah. And, and ot. And that's the piece that I think that's how we miss out and that that's what I'm excited about. Having conversations like this of the art of the possible of using capabilities like this. Yes. There capabilities that you can get, you know, analytics out of the environment and understand vulnerability and threat detection and all that kind of stuff. But that's not how you sell the rig and the manager and the operations guy on this tech. Because no, it's like nobody gives a. About your firewall. They care Can I be more effective? Can I. Is there things that I can troubleshoot? Can I get my rig back up and going faster? Can I make more money? Can I be more effective? Like, can I get better maintenance? Like, can I find failures before they happen to. That keeps me up instead of, you know, having a trip or an outage, can I. All of those are the conversations we need to be having. And technologies and capabilities like this sometimes are pushed off and they're presented as a, hey, I'll get you better ash dashboards or whatever for cybersecurity. And the operations guy's like, I don't fucking care. I'm in the middle of the damn ocean half the time. I don't have Internet. Like, what are you talking about?
11
00:16:19.000 --> 00:17:07.000
In industrial spaces, like, poor human factors. Design as
it relates to HMIS and control system is much more mature than I would say conventional it, you know, death by alarm is. Is quite common. Like, I don't. If you can't do meta alarms automatically, like, you're already 30 years behind, because you can't just give someone the fire hose and expect them to be enthusiastic about it, regardless of what visibility you're showing. Like, it needs to be quick, actionable, and not a nuisance. Or it will get ignored. Because that's just like, it's human nature. They're gonna, like, it's gonna become normalized and they will just move on. If the noise, no matter how annoying it is, doesn't tell them what needs to happen right then to address it. Like, people are going to go, well, I don't know what I'm supposed to do. What do I do? What do I do with that information? And so they'll end up getting left vulnerable because of that. Yeah.
12
00:17:07.000 --> 00:17:15.000
Legacy keyboards and control rooms with some of the vendors as we're rolling out control systems and so many keyboards and
13
00:17:15.000 --> 00:17:17.000
monitors that don't need to be there.
14
00:17:17.000 --> 00:17:35.000
Yeah. But even the ones that are there right in the control room. You walk into this control room and there's a literal button on these custom keyboards that are like $10,000 for these older control systems. And there's a mute button. And nine times out of 10, that mute button is the very first thing that breaks on every single one of them because they just smash it constantly.
15
00:17:35.000 --> 00:17:42.000
Alarm acknowledge. Alarm acknowledge. That'll have the text ripped off of it. It'll, like, the waterproof little barrier will be blown out. Yeah.
16
00:17:42.000 --> 00:17:43.000
Yep.
17
00:17:43.000 --> 00:19:01.000
Yeah, it's rough. I mean, we've saw, We've seen it, like, from a systems perspective, I've Been a part of major like contract preparation where we were having to have discussions about, you know, visual clutter considerations for the person operating the equipment. Because every vendor comes in and has their little operantly critical system that's the, in the back end doing something. But then the, the operator needs to be able to see it contextually. At the time that he's doing some particular operation, they were just filling up his visual space in front of him. It's like guys, he's got to be able to see the, the tool and you know, I mean everybody's got cameras but there's always going to be that angle and you know, they're running crews like that's not going to work. And unfortunately, fortunately or unfortunately like industries like that and I'm most of them are probably within this realm except for gotta love the concept but like the dark plants because we're designing for like cohabitated spaces of like heavy equipment and people. You're always going to have like these weird tight areas where you have to consider, you know, the full scope of like what that's going to look like for somebody on a day
to day. Like are they, are they cussing your company's name? Are they shaking their fist at you? Or they like completely don't know who you are because you just fall into the background or are they thinking that, you know, his company spent the money on your product, you know, every day or every time he gets to use
18
00:19:01.000 --> 00:19:40.000
it well, and diving further into Dockers and virtualization and all of this space, right? Is, is the space problem, right? You know, when I'm rolling out hardware across, you know, I did a large program for power utility and we rolled out across 3,000 sites. That was a lot of hardware that we rolled iron for. Imagine if I had to roll iron every time I wanted to do an update. When you start talking containers, I can have one box and it can, I can have it do X, Y and Z today and new capabilities come out tomorrow. You spin up another container, you have another capability. I don't have to go back to site to your point because I've already got something there that can be my host.
19
00:19:40.000 --> 00:23:23.000
There were a couple of specific to oil and gas. There were a couple of vendors that were trying to do this, but I always felt like they were trying to black box a little bit too much and own just owners keep a tight, tight grip on it. And I felt like that hurt them. Like you know, they wanted to have licensing fees for the vendors that were involved when in reality like the end Customer could dictate this stuff. Like if you have some shared services platform with your let your IT do the base level like from bare metal up to VM to whatever and then set security policies for the OS to namespace to user space and containerize to your heart's content and just put in a way you steer your vendors like no, these are the why does your software need that permission? It's going to get taken away. Does it actually need it? Show us why and make them adapt. And even in a container perspective like the answer might not be there is a reason. Like so it's just them changing what their deployment template looks like and go oh yeah, it's fine, just deploy it. And then you know, the speed to new features being added, you know, concurrency of deployment. To your point, like you could horizontally scale at just eye watering density and speed without adding a bunch of physical hardware and it alleviates that kind of, you know, the usual thing which is like oh, either everything's a SaaS which there's industries that just won't work for or everybody's got their little edge PC they want to upcharge you for as part of their, you know, deliverable which you know, in eras like Covid, right? Those were unobtainium. Like all IPCs were just gone. Like when all the small device consumption pivoted initially. And then you know, we had, I was doing some remote projects that were like an early phase of specking parts and stuff and they were pushing our dates out like 18 months for like small component electronics. The closer that the component got to consumer grade in like what hardware was in it or what factory it came from, the worse it was. So eliminating a bunch of stuff like that you get into other things. Like there's this great guy that like every industry has his characters but dynamic positioning has a lot of them. He's like in retirement and he fixated on this like one failure mode and built a little simulator for it. And it was DC fault propagation. Common knowledge or common approach was like on a DC rail you could have a bunch of redundant power supplies because you know you're not phase locking them to each other. It's very simple. You provide for redundancy and then you know what they're fed from can be redundant also. And essentially he like developed a little rig to show you know, DC faults could proposal propagate across reliability groups via common DC rail or shared voltage areas. And so that you know, eventually got pulled into regulatory standard. Not regulatory standards, but like class standards which eventually makes it to insurance companies, which makes it to class renewals, which means it's mandated. And so they like go around and every place that this happens they have to like segment them all. When you're adding a widget, it uses power like AC or dc, like is there available capacity there? Sometimes the answer is no. It's not just about physical space. It might be, is there load space? Is this causing another risk to other equipment just because we plug it in right here? You know, service ports and cabinets aren't meant for just anything. And you know, there's a lot of detail engineering that goes into control systems that try to account for every failure mode of every component in that cabinet. And when you just add some random thing that they never even knew existed, it can. Are you ready to be the one that they're like coming out with your box smoke and being like your system did this to my facility or whatever and cause this much revenue loss. And I don't think people consider a lot of that. But just because you know, specific industry exposure and things like that, that.
20
00:23:23.000 --> 00:24:07.000
That's an interesting point too Matt, because some of the stuff that we're kind of moving to in the broader technology space is running into a lot of these problems. For an example, we were working on some projects with battleships and that was an example where DC power supply is actually like very particular. And so you have to spec out all of the DC power constraints because the last thing you want is for the battleship to lose power because you pull too much of the power supply out or you blow out all the breakers on the battleship. And so those types of considerations at these low levels are becoming more and more prevalent. Especially as we start talking about these data centers, satellite concepts or things out where they are going to be like in space style things and you can't reach them to any capacity.
21
00:24:07.000 -->
00:25:30.000
Yeah, because highly integrated systems like that are. Even if they have an individual function like a vessel, a spacecraft, an airplane will go through final like full integration testing because they have follow on effects to other systems, especially at like power distribution, be it AC or dc, you know, just because it's a natural one where failure can happen pretty frequently. And so yeah, they like bake in safety factor and things like that. With all the detail engineering before anybody buys a piece of kit and coming in there after the fact and adding an edge AI compute node. Right. Like that's pulling 10 times the wattage that like a normal little IPC does maybe wasn't accounted for. And then like, you know there's not every industry has that or company or facility. But the default right now, and this is from a security perspective too, is that the, the person responsible for like the upkeep and maintenance and logging of all that equipment is supposed to be the one that everybody is expecting to push back. And I personally told people like, no, you're putting in a whole rack. You're not, you're not co locating in our control systems racks. I was like, oh, we'll, we'll pull you a breaker if you tell me what your loading is and a network drop. But you're not going in Iraq. And yeah, they, they, you know, and floor spaces at a premium too, and cooling for that matter. But yeah, you can't, there's a lot of dependencies outside of somebody just, hey, let me plug this thing in. And well, and that's a big piece
22
00:25:30.000 --> 00:27:01.000
to this, that many people, you know, I, I travel around, I go to conferences, I'm at, You know, I was at DEFCON in Singapore and I go to DEFCON in Vegas and RSA and black AD and all these places. And I, and I, I have these same conversations with people, especially from the IT side that don't understand all the reasons, like why don't we just rip everything out and put in Windows 11? Like beyond the fact that. No, but imagine how much power and space and like there's just all these other factors that are included that you're not thinking about. And also the risks. Like some of these systems, again, I'm dating myself here. No, that shit would run. Like, I remember systems that had been running for like 10 years and never rebooted. Right. You can't, there's probably not a system alive in an IT world that that has been that. Right. We're patching it, rebooting them all the time. And many of these OT systems you literally don't touch, like it has just been running. Leave it the heck alone. It'll run forever. As long as it doesn't like fry out like electronically, it's going to continue. And we don't have that level of rigor in our traditional IT systems are designed to be replaced every two to three years. They're, the software is end of life, you know, they don't make parts for them anymore. Like, that's just not how they work. And OT is a vastly different environment from a space. Power. Like the hardest thing to find at a power plant is clean power, as ironic as that sounds. But you can't find any power at a power plant.
23
00:27:01.000 --> 00:28:20.000
Yeah, the way I always think about it is the breadth versus the depth and OT is very complicated. So like what Matt was saying, the dependencies along the breadth can be very, very difficult because you have more dependencies to consider like power constraints like we mentioned. Even like you said, physical capacity and physical space in the plan can be a constraint. You don't typically deal with those in the IT space as much. So those are kind of those wide vectors that you have to consider. But then there's also like the depth too. And we come across things all the time like oh, this company's running Windows Server 2012 which is now end of life and not supported anymore and the kernel modules are all really old on it. So like installing new software is almost impossible. But like the vendor, the manufacturing vendor only built it for Windows 2012 and not for any newer versions. So you're also in like these OS lock areas of, of OT Space 2. You're also in these stack lock areas where like you mentioned earlier, someone on the physical plant may have specific security concerns around the physical proximity and things like that, but they're not thinking about the networking layers on top of that and someone else is thinking about the networking layers. You end up building all these gaps into your security stack because there's just different people at every layer thinking about them and no one thinking about them from that 360 degree view. So we see that all the time in our space.
24
00:28:20.000 --> 00:30:12.000
Well, and that's like the, the, you know, the minute you let the floodgates of vendors come through your gates, the expectation is that you're the filter and that that person may not be there or have the, even the expertise to ask the right questions. The vendor may not know. I've seen in the 8 and 9 figures quotes where yes, same exact example where a Windows Server version that was already end of life or about to be was included in the scope by name. And then I had to explain to the vendor that LTSC was a thing that existed. So it's just their default. Like it's what they were comfortable with. You know, they had. NET developers or whatever the dynamic was that made them end up at that. But yeah, I think it is generally like, you know, Microsoft won the general familiarity wars and so you end up with systems that are designed on something that they think is supportable because people know how it works. But yeah, truly it's you know, to general purpose and that's, you know, to the legacy, like the real legacy control systems, those were very narrow scope operating systems. And so when you don't have a bunch of like Interaction that isn't required. You know, that's a complexity reduction. It adds to stability, reduces security exposure to an extent. Yeah. And it, you know,
they can go for a really long time with the appropriate, you know, documentation. And I love those because they're like. It's like reminiscing. You, like, go into the, the document for, like, how to recover a system and you see these, like, old DOS commands and stuff because it doesn't have like. And you're like, ooh, this is, this is going to be an interesting one to tinker with. But it happens all the time. There was. I had a colleague when I was a consultant that was supporting a. A platform that had their primary, like, drilling control system was running on, you know, the derivative. There's a precursor to NT, but it was like Windows 3 base, which I think it was NT 3.5, but it was like it had the old.
25
00:30:12.000 --> 00:31:41.000
Yeah, yeah, well, and that's the thing. I've literally walked into plants of all kinds and had Sun Microsystems or an NT system still running. And, you know, at the end of the day, again, an IT person walks in and they're like, how could you possibly allow that? And the answer, you know, on the, on the OT side and the operation side, it's more like, if it ain't broke, don't fix it. Like, why do I need to upgrade it to something that I have to patch and is unreliable and I have to do all these things? This thing, it's so old, it's actually probably more secure than newer things because it has, like two vulnerabilities and it's not on the Internet. Right. It doesn't have the capacity to do anything but what it does. And that's all I need it to do. I don't need it to do anything else. I can put another device, like what you guys are talking about. I could put something else that gets me the data out of it without having to replace this thing. So let's keep this thing really reliable. It's like, you know, do you want to run your. The analogy I always give, like, you're in an airplane and the airplane control system has a vulnerability. Do you want them to patch it while you're in the air, or do you want to land the fucking plane? Let's land the plane. But take that a step further. Do you want them to replace the control system with, you know, a Windows 11 device? I don't. I'd rather keep it on the device that's been running in a Boeing for 30 years. Because yeah. Is it the most secure? No. But is it reliable? It flies planes really, really, really, really well. And they've worked out lots and lots of bugs.
26
00:31:41.000 --> 00:32:44.000
There was I just that reminded me of like my favorite example is, you know, I thought I was escaping it when I joined the military. And then I got to my first duty station and there's like this particular workbench that I plagued my entire military career repairing electronics and flir. And I remember reading like one of the benches like boot and troubleshooting steps to like recover from an image and they had the big cartridge CDs for the image on it. And like it was. Someone had just made a Word doc and like wrote the commands you gave it. And I'm looking at it, I'm like, this thing's running Linux. And it was like just the most basic, like switching to the kernel that was on the disk to boot from it and copy it over. I was like, oh wow. And no one there had any like context of what they were typing in. They were just repeating what was on the page. I just thought that was hilarious to. Yeah, that was probably my first. I'm not going to say first, but like first at that cost level of like capex on a piece of equipment where I was like, you really can't just like build anything you want with you know what's out there and nothing should be a black box in that regard.
27
00:32:44.000 --> 00:32:45.000
By obscurity.
28
00:32:45.000 --> 00:32:52.000
Yeah, exactly. I think all three of us can lament the fact that Microsoft kind of won out the war on.
29
00:32:52.000 --> 00:33:20.000
They're putting. I don't know if you like I'm just. What's the terminally online, the whatever Microsoft calls their thing this year. I know it was like they have connect insight and whatever. I think it was the developer one grep is going to be part of PowerShell now. So we've come full circle. They're taking open source CLI tooling and pulling it back into PowerShell. And no, I don't mean WL. I mean in PowerShell you'll be able to grab stuff.
30
00:33:20.000 --> 00:33:21.000
Yep.
31
00:33:20.000 --> 00:33:27.000
Core utils don't tell them about Rip Grab. So yeah, they're. They're already 30 years behind, I guess
32
00:33:27.000 --> 00:34:02.000
but that's what we see all the time. Like to your point Matt, is oftentimes you get pin locked into these Windows operating systems and Windows is notorious about like poor forward compatibility and they'll break their kernel drivers all the time. So Windows is very finicky if you want to try and upgrade the operating system. But Linux in Particular is a lot easier for that. So I think the OT space in some parts are starting to shift that direction because they're realizing that too. But you'll get more stability and the ability to upgrade if you move to Linux from Windows. So we've been encouraging that for some of our OT customers as well.
33
00:34:02.000 --> 00:37:48.000
But we did kill like an entire generation of like that level of like detail curiosity. It felt to me like in the 90s that was like. But you know, we didn't have the bandwidth to download a Linux image but we still managed to pull the. I remember compiling the kernel from the mailing list. So just being able to do that was like that wouldn't work today but was such a eye opener at the time and I as soon as like someone would hear about it you'd have like the mega thick books you're digging through. Like it just was a, a mindset that I feel like we lost with the yeah like Microsoft push the gas pedal and you know, the whole world got familiar enough with it and they get you know, locked in. There's been exclusions and things like that with some strange touches of POSIX compliance with macOS and things like that. But the yeah, I don't
like you just don't. It is having this weird resurgence because of like gaming and Windows being particularly aggressive now. But it's going to be you know, a 10 to 15 year delay before we see those people in the workforce. And so what we've seen in the industrial space is people, they lack that middle generation of sysadmins essentially of like people that could had enough shell scripting like because your best sysadmins your laziest, right. Like it's the one that had his scripts and his cron jobs done so that he didn't have to do anything. And he may have been like troubleshooting some edge case or something like that. That was your guy. That's who would be designing these base OS infirmity images and keeping them up to date. But what's ended up happening is like they're going to the embedded OS stuff which you know is a extrapolation of like DDWRT and things of that. And then it gets more polished like PTX disk then busy Yocto is a much more structured approach and we are, I am seeing that so even at that layer. But then they go a little bit too in the deep end in my opinion in that as soon as they make that transition they all like they want rootless everything. And I'm like, you can bifurcate your implementation and cause yourself less development headaches. But I am personally seeing that since being at portainer like going to conferences from this side and talking to some of our hardware partners, the amount of people that are either have options in that realm or are moving in that direction and then quickly progressing to them. They are still running I assume hardware shortage is going to fix this. But like they're very much on still like legacy ARM architectures. So like stuff predating like seven which makes it a bit rough sometimes. So you'll like you might have an ARM compatible container but it won't work. Yeah unless they specifically compiled inside of it. So you can still run into some pain points there like if you're, if it's not, you know, if the vendor provided container is somewhat abstracted in a binary but from their perspective like again with the containerization and their development process, that's something that you could potentially feed back to a vendor that you have a, a big impactful enough relationship with to say hey I have these ARM devices. Can you see if you're able to compile and make a image that you could tag as this and, and some of them will pursue that. You know, it's just about how much of their customer base is represented in it. You know, of course they're going to start at like the more popular populated portions of the market but we like, we are seeing that the amount of people that even now like I feel like YOCTO is their flavor of the, the last few years, most of them, even if they had a Linux flavor as their base before, are moving toward YOCTO as like a way to handle the security better and be able to have essentially like whitelisted repos and package installers and things like that and give some extensibility to the customer without them, you know, just being able to go and tinker with everything.
34
00:37:48.000 --> 00:39:15.000
The irony of it, a couple things off of that, like the irony of I literally just did an assessment on a site and we did. You know, I had multiple sensors out there grabbing PCAPs off of network switches to try to get network flows and, and all the things on it. I've got a terabyte of data that I'm going through, you know, with, with all the communications just on the OT side of the network for multiple data points in those. And as I'm looking at it and I'm parsing through flows and PCAPs there's like 37 devices that are on the OT network that's supposed to be Isolated and they're all phone and home to Microsoft, right. And it's just like oh my God like we can't, we've got to cut these things off. But Microsoft bitches about it but we're scared on the, on the Linux side because like you said we don't have these admins like all of those, those sensors in the back there. I have to your point, I'm the lazy guy. So I've created scripts. So I just click, I literally pull up a script. It gives me a menu and it says are you starting a new one? It wipes it, it builds it from scratch. It wipes you know, employment or the, all the data off of it. It puts it at the latest ver it start and I put it back in the case like I'm done. It uploads the data for me. I've got parsers built into it. Like I've done all of this because I did it once and I didn't want to have to do it the second time so I spent a little bit longer created it so that I could script it so that the third time I did it it's just put pushing a button, maybe doing some troubleshooting. But at the end of the day when I'm done you can say an
35
00:39:15.000 --> 00:39:16.000
you're not going to hurt my feelings.
36
00:39:16.000 --> 00:39:40.000
But it's little things like that problems. Yeah, no, no, it's not that. It's like dude, you know we've got Portainer running on it. We've got, I've got local scripts on things. I'm running Raspberry PI boxes. There's literally onlogic and other hardware vendors that have a DIN mounted DC powered Raspberry PI that is, it's got inputs and outputs like a plc.
37
00:39:40.000 --> 00:40:07.000
Like yeah, there's a, there's one of our. I'm not going to call them out, it's a work in progress but one of their fancier power supplies has like a little aftermarket but I guess optional like clip on thing that just snaps in and then gives you a MQTT or OPCA client endpoint. And yeah, I'm pretty sure it's like a little embedded. I started trying to pull it apart but then I was like I don't want to break this one. But yeah, it just gives you a little web UI and it's dead simple. I'm sure it's running on a microcontroller
38
00:40:07.000 --> 00:40:09.000
of some sort and probably ESP32.
39
00:40:09.000 --> 00:42:35.000
Yeah, it's just, you know, and from their perspective like that's maintainable code. It's very simplistic. It's like
right. Sizing the device. It's well beyond commodity hardware at this point. Not super secure but also like if they're at that network layer, like that's not to, to my point, like once you're in the envelope of like layer one, you're in that envelope. So it's when we egress it is where you need to tamp things down that are phoning home and, or, or control for the end point that you do put into that space for whatever value add that you're getting out of it. Whether it be extensibility of like the existing system or modernization or just security blogging or you know, data analytics. And that's there's a lot of even in you know, oil and gas typically is like one of the industries that does industrial controls where people are like oh you guys could afford it. But the reality is, is like even during the times when we could, you can't afford the out of service time to do the upgrades. And so you prioritize some of them and then you start triaging for these. Like even from the control system manufacturers they have these like staggered hardware like life cycles that are extension beyond and they're doing just like the software vendors where they're like, okay, if you pay us this much more, we'll hold, literally hold these controllers in a factory in Norway for you for when they, they, you know, based on their failure rate that you'll be covered for this amount of time. And then at the extremity we have a rig was getting cold stacked and possibly turned into razor blades and we sent people out to gut physical hardware and it wasn't even like it was VFDs. We were pulling VFDs out of a rig to be able to keep another whole rig running for like a contract period. So if that's the kind of stuff we're doing at that. No, I don't care about your patches on your ipc. Is it working? No, I'm not going to touch it. So like what's the actual solution? Because to me like to say that software development, ie distributed systems is inflexible in finding solutions for things that like satisfy dependencies and security is silly. That just means you're not trying because there's always a creative way to do that. Is it a, you know, an arms race? Absolutely. But that's like the most fun part is being clever enough to be the one that's a couple steps ahead at least or be Able to tell when you fell behind in a safe zone.
40
00:42:35.000 --> 00:42:54.000
So, so what do you guys see? This is, this is usually my, my wrap up. I used to do the next five to 10 years but things, things are just changing so fast. I think that's a dumb, a dumb perspective because who the, who the hell knows. What's one thing that's coming up over the rise you're excited about and one that's maybe concerning. And I'll give you both opportunity to.
41
00:42:54.000 --> 00:42:56.000
I want to hear the encryption guys first.
42
00:42:56.000 --> 00:43:15.000
Yeah, I think the thing I'm probably most excited for right now, candidly, and I kind of teed this off is honestly I'm seeing a big switch, like Matt
was saying, from the Windows ecosystem into the Linux ecosystem. By and large there's a number of factors for that. I thank Valve for a decent chunk for helping with Linux.
43
00:43:15.000 --> 00:43:16.000
Thank you, Gabe.
44
00:43:16.000 --> 00:43:43.000
But that one, that one is the most optimistic to me. I think where I'm more concerned is definitely on the AI front. I feel like we're creating new exposures and attack surfaces everywhere and we're not closing a lot of them and so we're just like kind of building up more exposure over time and AI I think is just another vector of, of attack exposure that we have to deal with and we're not currently. So that's, that's kind of where my mind's at in the future focus right now.
45
00:43:43.000 --> 00:47:53.000
Yep, I agree with like the Linux transition. I hope that the people collectively with the, you know, the purse strings behind that realize that even though that has been like the historic open source project of our lives with the thankfully a benevolent dictator. Go look at the, the maintainer roster and look at their email domains. There are every blue chip company you've ever heard of that leverages Linux, contributes their employees time to maintaining open source projects. And if you keep wrapping them up into your industrial products, some of that has to go back or you're just putting yourself at risk down the line if you don't in house the expertise. And so I hope that they have some realization of that, that it's not just, you know, a free MIT license lunch to do with what they will because it is like it's been something that has been around. Like my first Linux distro was Slackware 2.3, I think was the suffix version. But so like my entire professional life has had some touch on it and then every time I think I escape it, there it is again. It really does run the world regardless of like what people are familiar with. So I'm interested in the AI one honestly, not because of what people are doing with it, because I have a intimate understanding of how it works, but in how it changes the way people think about things. So I already think that we're in a especially an open source that has the potential to hurt us because I feel like we're moving into what's going to end up being kind of a post trust ecosystem where without like vetting or some other set of automated tools for either ensuring secure deployments of even at the extremity, like malicious code. Yeah, I think that it's going to hurt us overall because it okay, let's be honest, like NPM was going to happen. It's the most hilariously insecure by design thing. Its creator regrets the way that he structured it. Everybody just tries to keep band aiding stuff on it but it's like I don't know if you guys saw but this morning now the aur yeah and it still is propagating via NPM and I think more of that's going to happen. You know we're at the, the whims of front end developers right now and JavaScript unfortunately. But you know, as time goes on some of that's going to fall back to you know, the more core and machine layer languages and I think that that could be rough because I don't want to, I don't want industrial people to like get freaked out and revert back to something they think is more supported or secure when it's actually probably still a lot worse. We just don't get to hear about it because they're allowed to keep it secret. So yeah, I, I, I'm interested to see what kind of systems and assessments people develop around that to be able to navigate the new landscapes and then just, I don't know, like I have some personal ones like for from stuff we're working on and tinkering with that I'm actually surprised we will be moving forward with and excited about that is touching on all these subjects. Josh knows what I'm talking about. But we're working with one of our partners to do some integration work and scoping for some customers to do that transition in a smooth manner. So from bare metal or VM regardless of the deployed solution into VM first and then containerized and repeatable and programmatic and then have security be a native part of that so that like whatever you're pulling from can be secure at the moment of deployment and you don't care where it came from. You know, kind of like how I just throw a random AI tool of the week on my home server to have a play with because I've appropriately isolated it, but at an individual container level. Very excited for stuff like that because I think it gives people that tooling where like, okay, if I just know that I'm walking through a minefield, make sure that I approach it this way or at the extremity. I think use. Use LLMs for what they're good at, which is pattern matching. You know, even hackers have primitives at some layer, like something that's not, you know, giant SaaS running at ring zero because we've seen how that doesn't work well. But like at a tooling layer that doesn't care whether it's a container, a vm, something like that, I think we're rife for solutions of that nature that are not a cash grab and are actually like trying to be at least somewhat altruistic given the landscape we're moving into.
46
00:47:53.000 --> 00:48:06.000
100%. Well, what's the call to action? How do people find out more? How do I use more Linux, more containers, more Portainer in the OT space? And what, what are my pet projects like? What is that?
47
00:48:06.000 --> 00:49:09.000
We're Portainer IO. I'm our edge product manager. If you engage with us in a professional capacity, I will end up talking to you in some form or fashion. I talk like this in our initial customer meetings. I'm, I love everything about my career history and what the. What tooling and skills I've learned over time. And I love brainstorming architectures to solve problems. Like, I think I've gone one customer onboarding call, went three hours one time on a Friday, just because we were talking about all sorts of stuff that he could like take different approaches with his architecture. Just. I thoroughly enjoy it. I feel like all of our FAEs are kind of in the same boat. You know, we're, we're, we're about solving problems and, and making it to where people can, you know, spend their, their brain power where it adds value, not in like maintaining their infrastructure. Because at the end of the day, like, yeah, infrastructure's fun because I'm weird. But if your job or your, your passion is delivering a software product or a service or something like that, the infrastructure should be what you're relying on, not what you're managing.
48
00:49:09.000 --> 00:49:10.000
Yeah, for sure.
49
00:49:10.000 --> 00:49:32.000
Yep. And I'm over at exceed, which is xd.com so we're doing a lot of work right now with portainer around connectivity and post quantum resistance, which is less. Actually less important for OT spaces mostly, but still very interesting work. If you come and reach out to Exceed, you'll definitely end up talking to me because I do all of our customer integration work and that's my favorite thing to do is dine in those environments.
50
00:49:32.000 --> 00:49:55.000
So we'll put all the details in the show notes, guys, and definitely reach out. It's exciting to see all the different things that we can do. I absolutely think Quantum and all those other pieces are coming to this space as well. We just happen to be behind the eight ball. Some of these things we can't be behind too far or it's going to be. It's going to impact us negatively. So we need to be thinking about things before. Before it's left of bang instead of right of bang. Right.
51
00:49:55.000 --> 00:49:56.000
Absolutely.
52
00:49:56.000 --> 00:50:03.000
Awesome. Well, thanks again for your time today, guys. Nice to meet you both. I'm sure we'll cross paths again in the future.
53
00:50:03.000 --> 00:50:06.000
Absolutely. Appreciate it.
54
00:50:06.000 --> 00:50:20.000
That's protected. All for this week. One favor. A quick review on Apple or Spotify takes a minute and helps other defenders find the show. New episodes every Monday. Follow wherever you're listening. I'm Aaron Crow. Take care, stay safe, and I'll see you next.