บทถอดความฉบับเต็มของ #236: How Nevada Recovered from a Statewide Cyber Attack in 28 Days (And What Every CIO & CISO Should Do Before It Happens to Them)

maybe that hearkens back to my time in the Marine Corps. In the military, it's not just one branch, it's not just one type of unit that can be deployed and expect to be successful. It really takes combined arms for a military unit to be successful. It takes land, sea, and air. So that's exactly what we did here. It wasn't just the GTO team internally. It was also our executive branch partners. It was also our vendor partners from across the spectrum. And we all needed to show up and they did. But definitely hats off to my GTO team. Those first few weeks, those first couple weeks especially. I had folks coming in doing 18, 20 plus hour days. And that really helped as well because they really had a high degree of care for what was going on. It's their environment and [00:07:00] they felt slighted that this happened to their environment. And so they had like a personal stake and making sure that this infrastructure, was restored and back to serving Nevadans. And so I had folks doing 18, 20 plus hour days and we had to kick them out of the office to make sure that they got rest and recovery so they can get back in the next day. So, one of the things that really benefited us there is that when our, incident response and recovery vendors, would give us tasks. We actually had, internal folks that could do that 24 by seven during the initial and most critical parts of the incident. And so there was never any time in the, those first two weeks that we were not making progress in the trace routing and, and whatnot through that incident. So I think that really helped us, speed up the recovery process. But it was it was definitely a challenge. I think communications was a key [00:08:00] challenge for us because our constituency is used to other types of events, in our state. They're used to wildfires, they're used to other natural disasters where government can really come out and give all of the information that they have regarding an incident a cyber incident is a little bit different. A cyber incident is more akin, similar to what I was just discussing. It's more akin to a battle. You know, we have an active adversary and an active adversary group active adversarial ecosystem that is watching us, that is looking out for any kind of weakness that they can exploit. And so we had to be very cognizant of what information that we were putting out there. So every step of the process, we had to be very diligent. We had to have clear decision gates on what information could be released to the general public and we'd have a balancing act. Is the information that we're putting out, is it helpful for the constituency? Is the information that we're putting out [00:09:00] potentially causing us to be more vulnerable to additional attacks? Are we prepared to withstand any of those additional attacks if they do happen? And so as we're preparing for communications. Each one of those checks. Each one of those decision gates would have to be cleared before we could

clear information to be released to the public. And it was important that we could release as much information to the public as we could because Nevadans were suffering from fear, uncertainty, and doubt. They were scared. They were scared that services weren't going to be restored. They were scared that their personal information was being exploited by criminals. And we wanted, with every fiber of our being to provide reassurance. They were suffering from fear, uncertainty, and doubt. And the only way that you can mitigate that, the only way that you could minimize that fear, uncertainty, and doubt is by providing clear, concise, and truthful information to them. And so that's really why we were, motivated to get as [00:10:00] much information out as we could. And that really led me to the conversation that I had with Info-Tech, and really the development of this after action report was, we want to tell the full story. We've heard of other incidents which will remain nameless throughout the nation that, you know, there was, there was not a lot of communication, there was not a lot of transparency. And that only bred more questions that only increased the appearance of coverups, et cetera. And, that's the last thing we wanted to do. We wanted to show Nevadans that, we were looking out for their best interests every step of the way, and that we were doing everything that we could to protect them and restore this infrastructure as quickly as possible. And we wanted Info-Tech to come in and be that third party to do an analysis of our entire event and help us tell that story. Joe Toste: That's really great. So you said a lot there that I could turn into a three hour podcast episode. But no, seriously, I've done one of those before. I think it was like three and a half [00:11:00] hours, but, a couple things I just wanna highlight that you said there. So one was partnership. Mark Hellbusch: Yep. Joe Toste: So a lot of times on the podcast, I often will talk about, this is kind of the human side, right? There's not anything technical about the partnership side, but getting buy-in from the executive branch, super, super important. Cause if you don't have that you're kind of dead in the water. And I had a great actually cybersecurity episode with Governor Ducey the former governor in Arizona getting really the mindset of what it takes for the executive branch to get that buy-in to sell it. And so getting out, you had that in advance, right? Getting the buy-in. I love what you said what they said to you. What do you need? Timothy Galluzzi: Yeah. Joe Toste: Not you're done. What do you need? What do you need? Right. I love that. And then partnership you mentioned the vendor partners, right? Really distinguished. They're both kind of the short term, the long term, and you probably now said, all right, these vendor partners are now gonna be really trusted long term because they showed up during this crisis. And then the third partnership level

[00:12:00] is with your team. Which for sure your military background went from peacetime CIO to wartime, CIO. And so I love that you highlighted the partnership there. Bringing the Avengers together, right. Kind of wanting that, wanting that super team to come together to take down Thanos. My last piece that you said that the team took it personally is, speaks to my heart. You know, I talk about this all the time. I coach high school basketball and there's nothing better than when the team wants to own it, not just the leader. Right. So, before we jump to Mark. I wanted to talk about the incident response plan. You had that plan built. How much did that plan match up with reality? Timothy Galluzzi: There's always going to be some differences between, you know, what you put on paper, what you practice, and then when the excrement hits the atmosphere mover, right? And so I think it was close because. We really focused on when we built our incident response plan was, let's be practical about it. What are the key lines [00:13:00] of communication that we need to focus on? Who's responsible for what. And so I think that there was pretty good alignment there. When we built out that incident response plan we were really building it out with an agency level response in mind. And so we definitely had to bring it up to the next level to really accommodate for a statewide incident. And so I think that's, that was really the only major delta in our incident response plan is that we had to lift it up one more level to accommodate for the entirety of the state. Some of the things that changed were call cadence and, who were the key stakeholders that were engaged in those conversations. At the very beginning, one of the things that we did to kind of highlight the partnership aspect that you just tapped on was we had daily conversations in, in the first couple weeks with all of our agency CIOs. So with all of the executive branch agency IT leaders, we would have daily touch base [00:14:00] calls with them just to give them an update. All right, here's where we're at, here's what we're working on, here's what our priorities are, and here's where we can use your help. And so that became really the heartbeat of our operation every day. So we were always working towards that update call, always working on, all right, what's the next thing we need them to work on? And then, you know, really focused our operations on that because we can't do it alone at GTO. I've got a team of 200 and we're trying to support an executive branch of almost 20,000 executive branch employees. We really needed all executive branch agencies, IT teams, to really be on board with us. And we knew that it wasn't gonna stop at this incident. We know we needed to only strengthen those partnerships with those executive branch agencies moving forward. And the only way that we were gonna do that is by communicating with them all the way throughout. Joe Toste: I love that. My last follow up for you Tim, I was thinking a lot about this was the community at NASCIO. Could you just speak to, I'm sure there's like a wealth of information. Can you just talk about that community and how that helped assist [00:15:00] you, in the state of Nevada? Timothy Galluzzi: Yeah. That, that community they really showed up too. So even if it was just the friendly reach outs, like, Hey, Tim, we're here for you. Or some of the more tangible calls that I had with some of my peers in neighboring states, where they could potentially share some intelligence, share what they're seeing even offers of tangible, like boots on the ground support. The NASCIO community really showed up. And, through that network we were actually able to share some intelligence. I, I can't speak in, in detail, but we were able to share intelligence about the event that we were going through to help out other municipal entities across the nation. The NASCIO community it's there for a reason. It's there to connect peers to, share information and to, really build up that network. Joe Toste: And the network really matters. Not necessarily in good times, but exactly right there when you know, you don't have time to build new relationships, right? You gotta be able to pick up the phone and call somebody. So I wanted to jump into when [00:16:00] did Info-Tech first come on the scene? And when did you realize that they weren't the typical vendor partner? Timothy Galluzzi: I've been engaged with Info-Tech now for over two years. We did a comparison between some of the similar vendors in this space and we just really looked at Info-Tech as really the best fit for us for operationalization of our strategic initiatives. It was just a better fit. And so when this event happened, they had already settled in as a extension of our team at this point, just because that's the relationship that we have built with direct support with our executive counselor, Patrick and our executive advisor Corby and the rest of the team. So when we were looking for any support on, hey, who are the best vendors in this space? And this was early on in the incident. They were able to immediately jump on that for us. If I needed any information polls and I didn't have the time to go and, [00:17:00] do that research myself. All it took was a quick email and the team jumped on that right away. And then as we were getting further on in the incident we started looking at, alright, what are we gonna do as we kind of wrap this up? And that's when the conversation of the after action report really started getting a little bit more in depth. And that's when we got introduced to Mark. Joe Toste: I love that. Now, Mark, you're coming on the scene. So Tim, I I didn't even ask you, I was gonna ask you afterwards, but I guess I could just ask you right now. There's the after action report, which is public. And then there was a letter that you wrote to Info-Tech. So you had your account executive. You had your

executive counselor. They're serving you now. We have this ransomware incident. Mark's coming on the scene. Mark, now you're jumping into the story. Talk about the weight that you were feeling when you got on the scene and you coined the term, we were talking about this phrase, 28 days of success. Unpack that a little bit for us. Mark Hellbusch: So I actually heard about this when it happened on the very day one. Okay. Because reality is, [00:18:00] you know, you can't keep things like this a secret in America. So once it happened and how big the state is and who they're connected to and, and all of our other members that are part of the state of Nevada. So we had heard about it through some other members of ours. Uh, so when it came about a month later, it came back to us. I was like, okay. So I was already very familiar with what was happening. I was following it along and seeing, potentially how I could, figure out who they were hit by. And I was doing a lot of that work, that first week, seeing everybody else that was going through similar things. So when I got the call to, to be brought into this I had kind of a general understanding. But one of the first things, and I've gotta hand this to Tim and his entire staff, was that I was coming into it with a view of, I've been through this before and I have learned over time that people, they just want to know . They just want to be informed. So when Tim and I sat down with his staff, that was one of the very first things we all agreed upon [00:19:00] was that, hey, Tim was very adamant about that. You know, we're a state, we have citizens. We need to just be honest and disclose what happened. We were there supporting that all the way. And as a result, that's what built the after action report. And I wanna comment something real quick, is that forced to get to that place, to be that successful? Tim actually has already mentioned it. He had already been doing instant response tabletop exercises the year, two years before. Working with our company. Alright. As he mentioned, Patrick, executive counselor from Info-Tech, he's been doing this a long time. He's actually assigned to Tim on a monthly basis. They work together so right there, Patrick knows Tim very well, knows all the people, knows the operations, knows what they need and what they don't. He reached out to me, giving me the 2 cents, brought me in, and it was wonderful how we all came together. Something that Tim hasn't mentioned, but I want to kind of bring it up here, is, we're always talking about Tim and I. The reality is this was [00:20:00] very big. You have to kind of keep in mind Tim and his staff, he talked about the 200 people, but what he doesn't talk about is we had the governor involved in quite a bit. We had the attorney general, we had the state communications officer. We had multiple other agencies that were all involved too. So it took a little bit of an effort to bring that together. But it was done in such a way that, Tim, I gotta hand it to you. Organizing that, having it come through, we were able to get all the data we needed very open about what we wanted to get accomplished. And so he gave us opportunity, you know, go out with this. And, and I was more than happy to. Creating the report, we worked through, I think almost daily for like two weeks. We were meeting daily to just make sure that we were getting all the information, we were disclosing the right amount of information. ' cause here's the other thing we wanna make sure is that we don't wanna give credit to the bad people, the bad actors out there. So at the end of the day, the state recovered successfully, but nobody needs to know, [00:21:00] you know, who it was. One of the big concerns I think as we move forward is the fact that Tim knows, you have bad actors in your environment. That's concerning because they learn a lot, right? So as we move forward, it's what do you do, right? And how all the other vendors came to play was really amazing and, and people need to understand that today and the work, and Tim knows this is when you start off with your cyber insurance. There's certain things that we all kind of abide by. So, reached out to them, got a couple of big reputable firms on site. You had your vendors of your operating system on site and letting them just do what they needed to do and then listening to them and their recommendations at that time, because I, you know, we were able to, Tim was able to get some improvements in right away as we're now developing more of a long term strategic plan. So, Timothy Galluzzi: And one of the. Things that was incredibly impressive with all the vendors that came in to support the initial response and then the recovery efforts [00:22:00] is how well they work together. I think, one would automatically assume that you know these companies that are natural competitors because they operate in, in the same space that, there'd be some mud slinging or whatever, but no we were very clear like, all right, this is your lane. This is your lane. This is how you guys are gonna work together. And they were great. There was clean handoffs when it moved from, you know, one phase to the next, or when one deliverable move to the next company, to the next company, or to the next phase of operations. And they all worked incredibly well together. And, hats off to, to all of our vendor partners that were engaged. Joe Toste: That was really great. So Mark, real quick, for those folks out there, CIOs CISOs, they might be numb to the fact they hear cybersecurity, they hear ransomware. It's just another, incident on the books. What's the TLDR on this, like, if, if to if they're not gonna read the whole report, give us the short executive summary on why they should. Mark Hellbusch: The executive summary really is this is if you truly believe that it's just a matter of time and [00:23:00] there's, we have this saying, it's not if it's when you're gonna get hit by

ransomware. If that truly is your thought process out there, then Hey, you have an instant response plan, but you practice it. That's the key. You have to practice it once a year and you just can't pull it out and say, oh, you know, I'll get five people together and no, um, just like Tim, you know, he does this. Tim correct me if I'm wrong, but I think we had 160 applications or so that you were somewhat connected to either directly or indirectly, at least, right in the multiple teams. I mean, when you start thinking about how complex a state is, all the different services that they provide, all the different groups that would have some ownership in those things, all had to be communicated with. Okay. Every single day as they went through and recovered from this. That is a big ask. That's a lot of work. As Tim talked about 20 hours a day for people for literally a month, if [00:24:00] not longer. 'Cause even when they were up, the work didn't stop, you know? So, yeah, there's a lot that goes into this link to be successful, and as Tim talked about, it literally changes your reality, meaning everything that Tim was working on in August, I personally, I would ask him, did you ever get back to any of that? Timothy Galluzzi: No the course has definitely shifted. Yeah. Uh, because after this incident it's very public knowledge that we had a special session that was called by our governor, and we were able to introduce Assembly Bill One. And it's very telling that it's called Assembly Bill One. It's the first bill on the assembly side, that was specifically geared towards cybersecurity. And that bill, I'm happy to talk about this constantly. It received unanimous support from our legislature and was signed by the Governor, but that bill supports a statewide security operations center. And that's legislation that we've been fighting for for years. It supported funding for significant cybersecurity enhancements to support the executive branch. [00:25:00] It supports a cybersecurity quick reaction force. If an event ever happens that's at a significant scale, I can immediately call up IT professionals from around the states to keep boots on the ground to help immediately respond to that type of event. The legislators, the Governor, everybody showed up for us when we needed them. I couldn't feel better supported. Joe Toste: I love that. And we had talked about on episode 218, I think we talked about the statewide SOC as an opportunity that you were looking at in the future. And, uh, well, what's the phrase? Never waste a good crisis and so able to Timothy Galluzzi: Yeah, it really just, highlighted the need for it. And Joe Toste: yeah, Timothy Galluzzi: highlighted the fact that cybersecurity is a team sport and no one is immune. No one is immune no matter how much how many resources you, you throw at cybersecurity. Unfortunately, all it takes is one errand click. One, one bad download, and your organization can be victimized. Joe Toste:

And just to highlight, I say this a lot and I love that you said team sport. Because within that, but coaching high school [00:26:00] basketball, team sport, you know, there's a couple things. Number one is we talked about practice. This is not a joke because you start to learn the high schoolers and at college NBA if don't practice, like really practice at game speed, you're never gonna rise to the occasion that you need to or you're only gonna rise to your best practice. And so I love the practice and the communication. This is like the number one thing, right? Communication balls, turnovers only happen because folks aren't communicating. They're on the same page. And I love that you said that the vendors knew their lane, their role of where to go. And then there were clean handoffs, there was great communication. I think that is so understated especially in cybersecurity, where a lot of folks, you want it to be technical, but, it's often the human, the human side of it as in team sport. And I thank you for highlighting. I remember researching, so Assembly Bill One, we're gonna link to that in the show notes too. So thank you for highlighting that, Tim. As we jump along what's the real [00:27:00] takeaway? What does this mean for folks around the country? Because there's a lot of, whether it's a state CIO or state CISO or a local government. It could be Bob Leek in Clark County. Shout out to Bob. And even in higher education, a lot of folks are, okay, so maybe I need to call Info-Tech. So Mark, for someone who isn't currently a member or hasn't worked with Info-Tech, what does the relationship piece actually look like? What does Info-Tech do for public sector when they're not in crisis? Mark Hellbusch: Oh, absolutely. And, and that's a great question. Thank you, Joe. You know, what we do is actually, as I mentioned, you know, we're the fastest growing research and advisory when it comes to IT and security in the globe. We operate on virtually every continent throughout the globe. We have major conferences here, but what we really do, Joe, that makes us very different is we partner up with our members, right? Our members, engage us to get a variety of information and we size it all according to what their desire is, meaning here's examples. [00:28:00] You know, let's talk about instant response. That's what we're talking about here. If you join Info-Tech you get access to all the information that we have about that, all the things that we can help you deal with. Instant response. However we call that, you know, entry level is kind of DIY, do it yourself, right? But then we have multiple areas that we can help you with. Technical counselors, executive counselors that are people that actually partner up with people like Tim on a regular basis, helping them being their strategic advisor. We also offer workshops on a variety of topics. Tim actually took advantage of one workshop in the past. Practicing instant response, so these are all different kind of services. And then you get finally down to me where we're the consulting group. And I always like to tell people this is, we're the big gorilla. We do everything that's custom, everything that, you know, doesn't quite fit in our membership roles. We come in and help out. And this is one of those examples where Tim needed after action report. Well nobody has a service line that says After action [00:29:00] report. We just knew he needed help. We knew what he wanted to get accomplished and we were there to support him every way, every step of the way. And honestly, what I wanna emphasize, this is our involvement was not just show up that day and produce this after action report. Our involvement started a couple years ago. Okay. We've been meeting with Tim every month, every year we've been talking with him. We've been building a relationship with him through various services. You know, like Patrick, Dennis, Corby, Richard Bateman. I mean, these are all people that have intimately gotten to know Tim over the years based on our, his membership and the services we provide. Bringing me in was new. But Info-Tech had already been there helping him, too. Joe Toste: Yeah. Tim, I'd like for you just to hammer home the difference between a trusted partner and a vendor partner. Can you just really take that home for us. Timothy Galluzzi: Yeah. A vendor is a very transactional relationship, right? A vendor is someone that you know, might blow up your inbox, with cold call emails when they, find out [00:30:00] that you're going through something difficult. A trusted partner a real partner is someone that shows up for you and someone that will be creative to fulfill the need for you, that will work with you that isn't afraid to, as Mark said, go outside. Just, what they have as lines on a sale sheet that will work with you to fulfill the needs of your organization. And that's the difference for me. And as someone that, I can call up when we're in need and they show up they're on flights, they're really delivering and they understand who we serve. When Nevadans need their services. If they can't get to the DMV, if they're worried about their benefits they understand that need and they understand where our motivations are for getting our services up and you can really feel that out of them. That's the difference between a partner who [00:31:00] really is an extension of your team and a vendor who is just there for the transaction. Joe Toste: Yeah. Thank you for hammering that home. I feel the same way. I get to interact with a lot of different vendor partners, whether it's like third party conferences or my own live podcast tour, just around, around the country traveling. And I hammer home, that you really need to be a trusted partner. You really need to care about what they care about and when it's a transaction, it's just. I'm very non-transactional, so you just spoke to my heart, Tim. I'm very non-transactional and so I often tell

people like, nice part about me is a little bit like Info-Tech, uh, Info-Tech being I think a hundred percent technology agnostic. I am, I only report to my wife. I don't think people realize, like I, I'd probably fire, the bottom 25% of my clients every year if they're not really focused on the mission. Which I think is shocking to a lot of folks, but. I only really wanna work with people that actually really care. Which is very weird in [00:32:00] 2026, you know, I, I just, I don't know, I just don't believe that money is the prime asset in life. And so, I actually care, this is why I coached basketball, right. Um, on the ground. So I thank you for talking about the trusted vendor. And I think that the trusted vendor partner piece, I really hope people take that home because I think would really do public sector very well to really lean into that. So something I also wanted to bring up Mark was, we don't have a lot of time, but as we look next for a lot of public sector folks, can you just talk about. Zero Trust, AI security. Again, this is another podcast episode, that we can dive into, but as folks are looking to what should they be aware of next? Sometimes they hear these words, but let's, you know, maybe just get a brief overview. Mark Hellbusch: So great question. I know that's, we're dealing with a lot of that questions these days. I'll be honest with you, Joe. Here's where the world really is, in my opinion, is we've spent the last 30 years doing a strategy, a security strategy that's called defense in depth, which really is just let's [00:33:00] layer security one thing on top of another, right? And now we have so many. Security companies out there, all their different software. You have to have Pam, you have to, all these things to protect the reality of the world is now it's just become overly complicated. So many organizations have tools that don't work with each other because of the complexity. I mean, I have people calling me about, they have like three products, and they're like, oh, how can I get all three to work together? And my first question is, well, if you're looking to the future, I would consider getting rid of 'em. Okay. Because the reality is, this is now with the introductions you mentioned, AI, artificial intelligence, that brings a whole different dynamic. Okay. You still have the same problems that we've had for the last 30 years. Okay? Now we have AI on top of that right now. Fundamentally, every organization knows this, is that they need to move to what's called Zero Trust Architecture. That is the new security model everybody should be adopting and let's be clear, it's not [00:34:00] new. But it's designed specifically for these kind of environments, right? With AI, you've gotta go in there making sure that you know your users are always being authenticated. You gotta make sure that, you don't trust, you know, you're always verifying transactions and things like that. Your system, you're taking that risk out of your environment. I mean, the reality is this is if you went to a zero trust architecture, you literally will reduce your risk by 80%. And on top of that, you probably could recoup half the cost on your security software budget. And what I've been able to show is about $1.5 million in ROI. A, you're more secure and you're gonna actually recoup some money. That's the future and that's the direction we are advising all of our members to go in. Okay. To ignore it today just means again, that a few years from now, you're going to be sitting potentially in Tim shoes, but not with a good outcome. Okay. And [00:35:00] that's, I'm advising people . Timothy Galluzzi: That's actually speaking right to the direction that the state of Nevada's going in right now. Not only is that called out in our after action report that we're I believe it's called out, that we're driving towards Zero Trust and a lot of our upcoming investments and, some of the stuff that we did in the immediate response to this event. But as we're looking forward, kind of more on the statewide strategic level. I'm driving towards more simplicity in our infrastructure, in the tools that we're deploying across, uh, the entirety of the executive branch to remove, uh, duplicative security tools to remove duplicative tools across the entire executive branch to lower the potential threat vectors because of just that, right? The environment is just overly complex and it's harder to secure. Joe Toste: That's also another, another podcast episode. And as a reminder for all of the folks listening we, so yeah, zero trust architecture there. There's been several conferences, events, but when you think about, again, securing an entire state and [00:36:00] all of the people involved. On a daily basis, I'm reminded when I'm outside of TechTables. No one's really that familiar with MFA, no one's familiar with zero trust architecture. Go talk to somebody in a different agency and they might look at you very strangely. So, on the tech side, I think a lot of folks we're all familiar. I have, you know, have these conversations all the time, but when you kind of step out and think about securing an entire state or, or county or local government or higher ed institution, it takes communication, right? It's gonna, yeah. Yeah. A hundred percent. So this is a great conversation. I could actually go three hours on this. Well not have the opportunity. Thus I feel bad for Tim. I'm not doing you quite, quite the service I want to, but, you know, hey, I'm trying my best in an hour. But my call to action is Tim, I, I believe you're gonna be at Info-Tech live in Las Vegas. Okay, I'm gonna give you this offer right now on the podcast. So do you, I am hosting TechTables at Info-Tech again. Uh, I just hosted this in New Orleans. If you would like, you can come back on the podcast and we can have a couple other friends from around the country who will join us in Las [00:37:00] Vegas. Timothy Galluzzi: Let's Joe Toste: do it. Um, we can, yeah, let's do it. I love it. Timothy Galluzzi: Game on. Joe Toste: Game on. I love it. So we'll, we'll tee that up. But with that, thank you both for coming on. Uh, the Public Sector Show by TechTables. Timothy Galluzzi: Thanks Joe. Appreciate you, Mark. Good seeing you. Mark Hellbusch: Tim. Call these, it's a pleasure, sir. And Joe, thank you so much.