Lessons Learned From Incident Response [The Industrial Security Podcast]

WEBVTT
1
00:00:03.799 --> 00:00:05.879
If you didn't listen to a single thing I said,
2
00:00:06.519 --> 00:00:13.599
you can listen to these three things, Collaborate, plan, and practice.
3
00:00:14.439 --> 00:00:18.359
Welcome listeners to the Industrial Security Podcast. My name is
4
00:00:18.440 --> 00:00:22.120
Nate Nelson. I'm here with Andrew Ginter, the vice president
5
00:00:22.199 --> 00:00:26.399
of Industrial Security at Waterfall Security Solutions, who's going to
6
00:00:26.480 --> 00:00:30.120
introduce the subject and guest of our show today. Andrew,
7
00:00:30.320 --> 00:00:30.679
how are you?
8
00:00:31.960 --> 00:00:34.320
I'm very well, Thank you, Nate. Our guest today is
9
00:00:34.439 --> 00:00:37.920
Chris Cistrunk. He is the technical lead of the Mandient
10
00:00:38.399 --> 00:00:41.840
ICs or ot security consulting team, whatever you wish to
11
00:00:41.880 --> 00:00:46.359
call it. Google purchased Mandian in twenty twenty two, but
12
00:00:46.560 --> 00:00:50.119
they're still keeping the Mandian name, so he still identifies
13
00:00:50.240 --> 00:00:55.719
as technical lead of Industrial Security Consulting at Mandian. And
14
00:00:55.759 --> 00:00:58.679
our topic. You know, they as part of their consulting practice,
15
00:00:58.679 --> 00:01:00.479
they do a lot of incident response. He's going to
16
00:01:00.520 --> 00:01:04.799
talk about lessons learned from incident response in the industrial
17
00:01:04.840 --> 00:01:05.599
security space.
18
00:01:06.480 --> 00:01:09.159
Then, without further ado, here's your interview.
19
00:01:12.319 --> 00:01:16.599
Hello Chris, and welcome to the podcast. Before we get started,
20
00:01:16.680 --> 00:01:18.760
can I ask you to say a few words for
21
00:01:18.799 --> 00:01:21.599
our listeners about your background and about the good work
22
00:01:21.680 --> 00:01:22.719
that you're doing at Mandian.
23
00:01:23.519 --> 00:01:29.079
Okay, thanks Andrew. I'm again at Mandient on the ICSOT
24
00:01:29.319 --> 00:01:32.400
consulting team, been doing that for over eleven years now,
25
00:01:33.200 --> 00:01:38.519
focus on ICSOT security consulting around the world with every
26
00:01:38.599 --> 00:01:45.560
type of critical infrastructure, doing incident response, strategic and technical assessments,
27
00:01:45.920 --> 00:01:50.079
and doing training as well. Before that, I was electrical engineer.
28
00:01:50.560 --> 00:01:54.319
I still am, but for a large electric utility energy
29
00:01:55.079 --> 00:01:58.840
was there over eleven years as a senior electrical engineer
30
00:01:59.079 --> 00:02:04.000
Transmission dish should skate a substation automation and distribution design.
31
00:02:04.640 --> 00:02:07.959
So that's a little bit about me and again just
32
00:02:08.080 --> 00:02:10.680
working for Mandian, part of Google Cloud.
33
00:02:11.080 --> 00:02:14.879
And our topic is incidents. It's lessons from incidents. But
34
00:02:14.919 --> 00:02:18.000
let's talk the big picture of incidents. I mean, Waterfall
35
00:02:18.039 --> 00:02:22.479
puts out a threat report annually. I'm one of the contributors.
36
00:02:22.879 --> 00:02:27.000
You know, we go through thousands of public incident reports
37
00:02:27.039 --> 00:02:29.479
looking for the needles in the haystack, the incidents where
38
00:02:29.479 --> 00:02:32.759
there were physical consequences, where there were shutdowns, where you know,
39
00:02:32.800 --> 00:02:38.120
sometimes equipment was damaged, and we rely on the public
40
00:02:38.280 --> 00:02:44.840
record on public disclosure. And so I've always believed that
41
00:02:44.919 --> 00:02:49.319
we were under reporting because I'm guessing again, I don't
42
00:02:49.319 --> 00:02:52.879
have that many confidential disclosures that people tell me about,
43
00:02:53.080 --> 00:02:55.319
but I'm guessing that there's a lot more out there
44
00:02:55.360 --> 00:02:58.800
that never makes it into the public eye. You folks
45
00:02:58.879 --> 00:03:01.960
work behind the scene, you know, without reaching any non
46
00:03:02.000 --> 00:03:05.159
disclosure agreements or anything. Can you talk about the big picture.
47
00:03:05.319 --> 00:03:09.520
Do you see incidents, especially incidents you know, in the
48
00:03:09.520 --> 00:03:15.400
industrial space with physical consequences, incidents that triggered shutdowns, incidents
49
00:03:15.520 --> 00:03:18.840
that are not public reported. How many are there, what
50
00:03:18.879 --> 00:03:21.159
do they look like? Can you talk anything about sort
51
00:03:21.199 --> 00:03:23.960
of what I would not see by looking at the
52
00:03:23.960 --> 00:03:24.599
public record.
53
00:03:25.360 --> 00:03:28.280
Sure, thanks for the question. You know, I think we're
54
00:03:28.319 --> 00:03:31.879
talking about cybersecurity incidents here, and there's many incidents that
55
00:03:31.960 --> 00:03:36.800
happen every day, right but life goes on. Squirrels happen
56
00:03:36.919 --> 00:03:42.240
right in the grid. But for cybersecurity incidents, I do
57
00:03:42.439 --> 00:03:47.560
believe we're seeing an increase. I can't go into how many.
58
00:03:48.439 --> 00:03:53.360
We actually have a report that m Trends Many has
59
00:03:53.360 --> 00:03:55.240
put out every year. It's going to come out later
60
00:03:55.280 --> 00:04:00.240
this month and for RSA, and this is a yearly report.
61
00:04:00.759 --> 00:04:09.039
We report on the different themes, the different targeted victims,
62
00:04:09.439 --> 00:04:14.639
the different threat groups, the TTPs. But for cyber attacks
63
00:04:14.759 --> 00:04:20.480
that impact say production or cause of a company to
64
00:04:20.680 --> 00:04:25.120
shut their operations down. I don't have any hard fast
65
00:04:25.240 --> 00:04:31.079
numbers to talk about, but we have seen an increase
66
00:04:31.720 --> 00:04:35.839
and you can look in not just our report, but
67
00:04:36.079 --> 00:04:41.560
also the reports of others IBM X Force, Verizon, d
68
00:04:41.639 --> 00:04:49.120
b I, r Drago's others. There are increasing reports of these,
69
00:04:49.199 --> 00:04:50.879
and a lot of it has to do with things
70
00:04:50.920 --> 00:04:57.399
like ransomware UH and ransomware either directly impacting the control
71
00:04:57.480 --> 00:05:03.120
system environment, which we have responded to in a manufacturer
72
00:05:03.959 --> 00:05:10.319
and a few others, but we have seen in the public,
73
00:05:11.199 --> 00:05:13.920
you know, news where a company might have to shut
74
00:05:13.959 --> 00:05:18.800
down operations due to indirect impact. Maybe their enterprise resource
75
00:05:18.879 --> 00:05:24.560
planning software or manufacturing execution software was impacted, which is
76
00:05:24.600 --> 00:05:29.000
an indirect impact to the OT critical data flowing that
77
00:05:29.160 --> 00:05:33.199
was halted, which means I can't produce my orders anymore,
78
00:05:33.399 --> 00:05:36.920
or track shipping or logistics things like that. So we're
79
00:05:36.959 --> 00:05:40.439
seeing a lot of those. There's others in the electric
80
00:05:40.480 --> 00:05:42.920
sector that they kind of have to be reported to
81
00:05:43.279 --> 00:05:48.480
OE four seventeen reports. If there's a material impact, obviously
82
00:05:48.480 --> 00:05:51.560
they'll be filed in the U or they're supposed to
83
00:05:51.600 --> 00:05:55.199
be filed in the eight K or ten K with SEC,
84
00:05:57.160 --> 00:05:59.680
and so I think if you take all of those
85
00:05:59.680 --> 00:06:04.079
source and look together and see, we see there's an
86
00:06:04.120 --> 00:06:11.240
increase of operational impact. Mhm. But it's I think the
87
00:06:11.279 --> 00:06:14.199
engineers are doing a good job of UH and the
88
00:06:14.240 --> 00:06:21.199
folks that run these systems are minimizing the impact in
89
00:06:21.240 --> 00:06:26.439
these situations, and especially for electric and water and other
90
00:06:26.600 --> 00:06:31.519
critical infrastructure. Manufacturing is critical. But I'd say it is
91
00:06:31.519 --> 00:06:37.079
probably the highest targeted outside of you know, healthcare and
92
00:06:37.120 --> 00:06:38.879
other other areas.
93
00:06:39.439 --> 00:06:41.680
So work with me on on the numbers, just you know,
94
00:06:42.120 --> 00:06:46.439
for one more minute. I'm on the record in the
95
00:06:46.560 --> 00:06:49.879
in the the Waterfall Threat Report speculating as to what's
96
00:06:49.920 --> 00:06:54.920
going on with public disclosures. It's my opinion, but you know,
97
00:06:55.000 --> 00:06:57.439
I have limited information to back it up. It's my
98
00:06:57.519 --> 00:07:01.199
opinion that the new disc closure rules in the SEC
99
00:07:01.319 --> 00:07:04.759
and other jurisdictions around the world are in fact reducing
100
00:07:04.759 --> 00:07:07.399
the amount of information in the public domain rather than
101
00:07:07.480 --> 00:07:10.839
increasing it. And the reason I suggest this is because
102
00:07:10.879 --> 00:07:14.000
it seems to me that with the new rules, every
103
00:07:14.000 --> 00:07:18.040
incident response team on the planet roughly you know, I overgeneralize,
104
00:07:18.040 --> 00:07:20.480
has a new step two in their playbook. Step two
105
00:07:20.519 --> 00:07:23.240
is called the lawyers, and what the lawyers say, they
106
00:07:23.279 --> 00:07:28.680
say say nothing, because if you disclose improperly, if you
107
00:07:28.759 --> 00:07:31.519
fail to disclose widely enough, you can be accused of
108
00:07:31.560 --> 00:07:36.680
facilitating insider trading. If you disclose too much information, you
109
00:07:36.759 --> 00:07:39.240
might get sued. I mean people have been sued for
110
00:07:39.759 --> 00:07:45.079
disclosing incorrect information about security into the public. People buy
111
00:07:45.079 --> 00:07:48.000
and trade shares, and then they know, they find out
112
00:07:48.040 --> 00:07:51.000
the information was incorrect and they get sued. And so
113
00:07:51.519 --> 00:07:54.480
to me, the mandate for the lawyers is say the
114
00:07:54.519 --> 00:07:58.439
minimum the law requires, because if you say too much,
115
00:07:59.279 --> 00:08:02.040
you risk make a mistake and getting sued, and you
116
00:08:02.040 --> 00:08:05.639
don't want to get sued. And if you you know,
117
00:08:05.639 --> 00:08:07.600
if you say too little, you're going to get sued.
118
00:08:08.120 --> 00:08:10.879
You know, the lawyers minimize. And if you have, you know,
119
00:08:10.920 --> 00:08:13.360
a material incident, you must report it. If it turns
120
00:08:13.360 --> 00:08:16.560
out the incident is not material to the finances of
121
00:08:16.600 --> 00:08:19.560
the company, you don't have to report it. And again,
122
00:08:19.600 --> 00:08:22.920
to minimize the risk of getting sued by reporting incorrect information,
123
00:08:23.439 --> 00:08:27.040
you report nothing. So my sense is that we're seeing
124
00:08:27.160 --> 00:08:30.680
fewer reports because of these mandatory rules, not more of
125
00:08:30.680 --> 00:08:33.879
than what do you see? You know, you see this
126
00:08:33.919 --> 00:08:35.799
from the other side does this make any sense? Do
127
00:08:35.840 --> 00:08:36.960
you have a different perspective.
128
00:08:38.039 --> 00:08:40.960
I can say that, you know, as an incident responder
129
00:08:41.080 --> 00:08:45.600
working with you know, victims in critical infrastructure, but also outside.
130
00:08:45.600 --> 00:08:49.000
I think this is a broader question you bring is.
131
00:08:50.240 --> 00:08:53.679
I can definitely confirm that we work with external counsel
132
00:08:54.080 --> 00:08:57.720
that a victim may have hired to bring in to
133
00:08:57.840 --> 00:09:03.120
handle a lot of these reporting or not reporting requirements.
134
00:09:03.240 --> 00:09:09.000
I can't say or confirm that the lawyers themselves external
135
00:09:09.039 --> 00:09:13.399
counsel under reports. I can't say that I don't know.
136
00:09:14.200 --> 00:09:16.679
I'm not a lawyer, nor do I play one on Facebook.
137
00:09:16.960 --> 00:09:20.200
So I will just stick to say yes, we have
138
00:09:20.320 --> 00:09:26.080
worked with external counsel, and usually we do not say
139
00:09:26.080 --> 00:09:30.639
anything in public for us as the incident responder, unless
140
00:09:32.080 --> 00:09:37.759
the victim company or our client asked us to, because
141
00:09:37.799 --> 00:09:42.039
sometimes sharing information is a helpful thing, especially if it's
142
00:09:42.080 --> 00:09:48.120
a big breach. Sharing that lessons learned about what has
143
00:09:48.120 --> 00:09:52.360
happened to them with others, just like we did back
144
00:09:52.360 --> 00:09:55.360
in the day when we had the solar winds breach.
145
00:09:55.559 --> 00:09:58.519
So you know, there's two ways of thinking of that,
146
00:09:58.720 --> 00:10:02.320
and maybe you can pull on that throw with some
147
00:10:02.360 --> 00:10:05.240
other experts, but not me. I don't know about the
148
00:10:05.240 --> 00:10:06.320
External Council part.
149
00:10:09.399 --> 00:10:13.679
Andrew, you'd referenced Waterfalls Annual Threat Report, a report which
150
00:10:13.799 --> 00:10:17.080
I've covered in the past for dark reading. I'm not
151
00:10:17.120 --> 00:10:20.159
sure i've seen this year's iteration, So maybe you could
152
00:10:20.559 --> 00:10:23.679
tell listeners just a bit about what the report covers
153
00:10:23.840 --> 00:10:25.799
and what the numbers are showing lately.
154
00:10:26.559 --> 00:10:31.440
Sure, the report uses a public data set. The entire
155
00:10:31.480 --> 00:10:33.320
data sets in the appendix. You can click through to
156
00:10:33.320 --> 00:10:38.960
it if you wish we cover we count in our statistics.
157
00:10:39.000 --> 00:10:43.799
We count deliberate attacks, cyber attacks with physical consequences, not
158
00:10:43.840 --> 00:10:48.360
stole some money, physical consequences in heavy industry and critical infrastructure,
159
00:10:48.519 --> 00:10:52.120
the industries we serve in the public record. Okay, no
160
00:10:52.519 --> 00:10:59.200
confidential disclosures. The numbers last year were seventy two attacks,
161
00:10:59.240 --> 00:11:03.120
I believe with I don't know some you know, a
162
00:11:04.879 --> 00:11:06.879
one hundred, one hundred and fifty something like that. I
163
00:11:06.879 --> 00:11:11.639
forget the numbers sites affected. Many of the attacks affected
164
00:11:11.720 --> 00:11:15.720
multiple sites. This year, you know, we're up from seventy two.
165
00:11:15.879 --> 00:11:19.519
We have seventy six attacks affecting a little over a
166
00:11:19.600 --> 00:11:22.200
thousand sites, So there was more sites affected, but the
167
00:11:22.320 --> 00:11:28.799
number of attacks did not increase sharply. And this is why, Again,
168
00:11:28.879 --> 00:11:32.200
I speculate, why have we sort of seen a plateau.
169
00:11:32.399 --> 00:11:35.600
We went up from you know zero essentially give or
170
00:11:35.639 --> 00:11:42.159
take in you know, let's say twenty nineteen to you know,
171
00:11:42.240 --> 00:11:45.960
seventy two, and then you know in twenty twenty four
172
00:11:46.000 --> 00:11:49.200
to seventy six. Why do we seem to have a
173
00:11:49.200 --> 00:11:51.320
bit of a plateau? And I'm speculating it has to
174
00:11:51.320 --> 00:11:55.600
do with the SEC rules. People are now legally obliged
175
00:11:55.679 --> 00:11:58.000
not just in you know, the United States, the Security
176
00:11:58.039 --> 00:12:01.480
and Exchange Commission, They're legally not just in that jurisdiction,
177
00:12:01.559 --> 00:12:04.000
but in other jurisdictions around the world. There's similar rules
178
00:12:04.039 --> 00:12:06.759
around the world. If you have an incident that is
179
00:12:06.840 --> 00:12:10.600
material that any reasonable investor would use as grounds to
180
00:12:10.799 --> 00:12:14.600
buy or share or sell or value shares, you must
181
00:12:14.679 --> 00:12:18.159
disclose it. But I have the sense that we are
182
00:12:18.279 --> 00:12:23.200
seeing fewer disclosures because by law you're required to disclose
183
00:12:23.320 --> 00:12:28.399
material incidents. And again because I speculate that because the
184
00:12:28.480 --> 00:12:33.720
lawyers are involved, we are seeing fewer disclosures. You know,
185
00:12:33.799 --> 00:12:36.679
they disclose the material incidents and they squash everything else.
186
00:12:36.919 --> 00:12:39.480
Is the sense I have. But you know, you asked
187
00:12:39.480 --> 00:12:44.879
about the numbers seventy six last year. Nation state attacks
188
00:12:44.879 --> 00:12:48.159
are up from you know, there were two the year before,
189
00:12:48.240 --> 00:12:50.960
there were six last year? You know, is this a trend?
190
00:12:50.960 --> 00:12:56.919
It's still small numbers, who knows? And industrial control system
191
00:12:57.080 --> 00:13:01.799
capable malware malware that understands industrial protocols and that is
192
00:13:01.879 --> 00:13:07.039
apparently designed to manipulate industrial systems is up sharply with
193
00:13:07.159 --> 00:13:11.639
there were three new different kinds of malware disclosed last
194
00:13:11.720 --> 00:13:13.679
year or you know, found in the wild last year
195
00:13:14.000 --> 00:13:17.159
that had that capability versus you know, seven in the
196
00:13:17.200 --> 00:13:21.840
preceding fifteen years. Again, small numbers. Is it a blip?
197
00:13:22.039 --> 00:13:25.480
Is it a trend? Is AI helping these people write stuff?
198
00:13:25.480 --> 00:13:28.399
We don't know? So these are all sort of you
199
00:13:28.440 --> 00:13:29.960
look at the numbers and you scratch your head and
200
00:13:30.039 --> 00:13:32.879
you go, I wonder this. Doesn't you know what's going
201
00:13:32.919 --> 00:13:37.120
on here? So that's that's the threat report in a nutshell.
202
00:13:37.320 --> 00:13:39.159
There's other statistics in there, but those are sort of
203
00:13:39.159 --> 00:13:46.159
the headlines that leads us into the topic of the show,
204
00:13:46.200 --> 00:13:49.120
which is lessons learned from incidents. You folks do incident
205
00:13:49.120 --> 00:13:52.000
response all the time. You know, can you talk to me?
206
00:13:52.360 --> 00:13:54.240
What are you seeing out there? You know, is there
207
00:13:54.240 --> 00:13:56.639
an incident or three that sticks in your mind. As
208
00:13:56.720 --> 00:13:58.879
you know, Andrew, the most important thing I have to
209
00:13:58.879 --> 00:14:01.039
tell you is and you know or the most recent
210
00:14:01.440 --> 00:14:02.559
Where would you like to start?
211
00:14:03.480 --> 00:14:08.679
Okay? Sure? We have been doing OT incint response since
212
00:14:08.720 --> 00:14:12.840
I've been here, and I can give you a few examples.
213
00:14:13.919 --> 00:14:17.559
Last year in twenty twenty four, we responded to a
214
00:14:17.639 --> 00:14:24.240
North American manufacturing company that had their OT network for
215
00:14:24.320 --> 00:14:30.279
looking at a Purdue model. It's the third layer or
216
00:14:30.519 --> 00:14:34.519
level three of the network was directly impacted by the
217
00:14:34.600 --> 00:14:41.480
Akira ransomware game. And what had happened was an unknown
218
00:14:41.559 --> 00:14:45.320
Internet connection was made by this third party who was
219
00:14:45.399 --> 00:14:49.120
running the site. They had put in their own Cisco
220
00:14:49.240 --> 00:14:55.159
ASA firewall, and it just so happened to be that
221
00:14:55.759 --> 00:15:00.399
there was two critical vulnerabilities in that firewall at the time,
222
00:15:01.120 --> 00:15:07.320
and the Cure ransomware game was targeting targeting those exposed firewalls.
223
00:15:08.200 --> 00:15:13.519
So don't necessarily think this was a targeted manufacturing OT attack.
224
00:15:13.759 --> 00:15:18.080
It's just ransomware gangs doing what they do, trying to
225
00:15:18.120 --> 00:15:21.679
make money, and so they were able to log in
226
00:15:21.840 --> 00:15:27.120
and get in through these vulnerabilities and deployed the ransomware
227
00:15:27.240 --> 00:15:32.320
on directly on the OT network, which was flat and
228
00:15:32.799 --> 00:15:36.159
every system but about five or six or seven were
229
00:15:36.240 --> 00:15:43.240
completely encrypted, including their o T DCS vendors, And there
230
00:15:43.279 --> 00:15:46.919
was multiple not pick picking on any one in particular,
231
00:15:47.000 --> 00:15:53.240
but ge A, B. B Rockwell several others that were there,
232
00:15:54.320 --> 00:16:00.279
and the backups server was impacted in the back up
233
00:16:00.600 --> 00:16:03.399
of the backup server was impacted. They were all on
234
00:16:03.440 --> 00:16:08.279
the same flat network. So this was a really tough
235
00:16:08.320 --> 00:16:16.240
situation since the company manufacturing did not have any backups
236
00:16:17.720 --> 00:16:22.480
that were offline. The OT vendors, like I mentioned for
237
00:16:22.600 --> 00:16:26.000
I had to come on site to completely rebuild the
238
00:16:26.039 --> 00:16:33.200
Windows systems, the Windows servers, the engineering workstations, the hmis,
239
00:16:33.360 --> 00:16:36.879
all the things that were Windows and or Linux that
240
00:16:37.000 --> 00:16:40.919
had to completely rebuild. Client didn't pay the ransom in
241
00:16:41.000 --> 00:16:45.559
other words, and so the lessons learned here. Work with
242
00:16:45.639 --> 00:16:52.000
your OT vendors and OEMs and even your contractors to
243
00:16:52.000 --> 00:16:56.000
make sure that your Windows systems and Linux systems have antivirus,
244
00:16:57.879 --> 00:17:02.639
make sure that you have OT backups that are segmented
245
00:17:02.679 --> 00:17:08.599
from the main OT network, and keep offline backups and
246
00:17:08.680 --> 00:17:13.319
test them on a at least a year basis. Backups
247
00:17:13.359 --> 00:17:16.400
will get you out of a bad day, even if
248
00:17:16.400 --> 00:17:19.480
it's an honest mistake at five o'clock on Friday. So
249
00:17:19.640 --> 00:17:23.480
this is a basic win here having good backup strategy.
250
00:17:24.640 --> 00:17:30.319
And then in the last case here we recommended they
251
00:17:30.640 --> 00:17:36.440
eliminate this external firewall and leverage the existing itot DMZ
252
00:17:36.599 --> 00:17:42.839
firewall that came in from the main owner of that site.
253
00:17:43.279 --> 00:17:48.319
And so they had a back door essentially that this
254
00:17:48.480 --> 00:17:52.279
third party contractor had installed in a new Internet connection.
255
00:17:52.440 --> 00:17:56.200
So give away from the shadow it go back to
256
00:17:56.279 --> 00:18:00.680
your normal itot dmz with jump box two, factory authentication
257
00:18:00.799 --> 00:18:04.519
and all those things. But if you do the basics
258
00:18:04.559 --> 00:18:09.079
and do them well, keep good segmentation, have backups, and
259
00:18:09.559 --> 00:18:12.960
patch your firewalls on a regular basis, I think that
260
00:18:13.039 --> 00:18:16.440
will go a long way, especially in this case.
261
00:18:19.920 --> 00:18:23.680
You know, I feel like I've heard of variant of
262
00:18:24.000 --> 00:18:28.039
that advice that Chris just gave a million times, and
263
00:18:28.359 --> 00:18:31.839
I don't work in industrial security, so you folks must
264
00:18:31.839 --> 00:18:34.359
hear it all the time, or it must just be
265
00:18:34.880 --> 00:18:37.640
such basic knowledge that you don't even think about it.
266
00:18:38.039 --> 00:18:42.119
So are there really industrial sites out there that still
267
00:18:42.160 --> 00:18:44.480
need to hear that you shouldn't be making an Internet
268
00:18:44.480 --> 00:18:46.359
connection from your critical systems?
269
00:18:47.160 --> 00:18:53.519
Short answers, Yes, you know, people who do security assessments,
270
00:18:53.559 --> 00:18:55.640
you know, not just incident response that we're talking here,
271
00:18:55.680 --> 00:19:00.319
but security assessments come back and say they regularly find
272
00:19:00.920 --> 00:19:05.759
connections out to the IT network and occasionally straight out
273
00:19:05.799 --> 00:19:08.680
to the Internet. The connections to the IT network tend
274
00:19:08.680 --> 00:19:13.319
to have been deployed by the the you know, the
275
00:19:13.359 --> 00:19:16.519
engineering team or the IT team to make their lives easier.
276
00:19:17.720 --> 00:19:20.920
You know, people with gray hair enough gray hair like me,
277
00:19:21.079 --> 00:19:24.200
they talk about, you know, how the systems used to
278
00:19:24.240 --> 00:19:27.079
be air gapped. This was a very long time ago.
279
00:19:27.119 --> 00:19:29.559
We're talking thirty forty years. The systems used to be
280
00:19:29.559 --> 00:19:33.839
air gaped, and you know, people with gray hair like
281
00:19:33.920 --> 00:19:37.039
me might assume that's still the case. It's not. You know,
282
00:19:37.119 --> 00:19:42.359
everybody who does audits reports these connections. The really disturbing stuff,
283
00:19:42.519 --> 00:19:45.880
you know, yes, it's it's disturbing that there are connections
284
00:19:45.880 --> 00:19:48.440
to the IT network that are poorly secured, that, you know,
285
00:19:48.799 --> 00:19:51.720
But the really disturbing stuff is the vendors going in
286
00:19:51.880 --> 00:19:54.279
and if you do an audit on a site time
287
00:19:54.359 --> 00:19:57.720
and time again, I hear people saying, yeah, they discovered
288
00:19:57.799 --> 00:20:01.559
three different Internet connections. The vendors stuck in there, and
289
00:20:01.599 --> 00:20:05.079
you're going, well, wouldn't you notice if there was a
290
00:20:05.119 --> 00:20:08.079
new Internet connection. I mean, no internet service provider gives
291
00:20:08.079 --> 00:20:10.720
you a connection for free. You got to run wires,
292
00:20:10.759 --> 00:20:12.799
you got to You got to pay for this thing.
293
00:20:12.839 --> 00:20:14.400
Every month. It's showing up on your bill.
294
00:20:14.440 --> 00:20:14.920
No, it's not.
295
00:20:16.119 --> 00:20:18.359
You know, there's a lot of wires being run while
296
00:20:18.440 --> 00:20:20.880
stuff is being deployed. You don't notice a new wire.
297
00:20:21.440 --> 00:20:25.480
And the vendors pay month after month for the Internet connection.
298
00:20:25.680 --> 00:20:27.920
Doesn't even show up on the bill of the owner
299
00:20:27.960 --> 00:20:32.279
and operator. Because the vendors are providing a remote management
300
00:20:32.400 --> 00:20:35.880
or vote maintenance service and they want to minimize their costs,
301
00:20:35.880 --> 00:20:38.799
they want to maximize their convenience in terms of getting
302
00:20:38.799 --> 00:20:42.200
into the site, so they deploy rogue DSL routers, they
303
00:20:42.200 --> 00:20:47.119
deploy rogue firewalls to the site's Internet connection. They deploy
304
00:20:48.640 --> 00:20:52.920
they might deploy rogue cellular access points where you know,
305
00:20:52.960 --> 00:20:55.359
there's not even wires to run. It's just a box
306
00:20:55.519 --> 00:20:57.960
sitting there that has a label on it saying, you
307
00:20:58.000 --> 00:21:01.119
know important, do not remove. And of course God makes
308
00:21:01.160 --> 00:21:03.799
it invisible to everybody who's looking at it says, oh what,
309
00:21:04.039 --> 00:21:09.200
don't touch that one. Yes, it's very common. The advice
310
00:21:09.359 --> 00:21:12.680
I try to give people is when you do like
311
00:21:13.119 --> 00:21:16.559
a risk assessment or a walk through or an audit
312
00:21:16.640 --> 00:21:22.200
of your site look for these rogue connections. Unfortunately, you're
313
00:21:22.279 --> 00:21:26.279
probably going to find one or two of these contractual
314
00:21:26.319 --> 00:21:32.920
penalties with the vendor help, but they're no guarantee. You
315
00:21:33.000 --> 00:21:39.039
said that the victim decided not to pay the ransom.
316
00:21:40.319 --> 00:21:44.960
You know, do you see victims ever paying the ransom
317
00:21:45.440 --> 00:21:49.000
to recover an OT network, to recover the HMI, to
318
00:21:49.079 --> 00:21:52.039
recover worse than that, the PLCs and the safety systems.
319
00:21:52.480 --> 00:21:57.480
You know, why would does anyone trust a criminal to
320
00:21:57.640 --> 00:22:00.359
take the tool the criminal provides and light it on
321
00:22:00.400 --> 00:22:04.880
their safety system and you know, restore it because they
322
00:22:04.880 --> 00:22:07.279
trust the criminal. Does anyone trust the criminal that far
323
00:22:07.279 --> 00:22:08.200
as that does that happen?
324
00:22:09.240 --> 00:22:16.400
We have seen traditional I T systems where they pay
325
00:22:16.519 --> 00:22:23.039
the ransom and get access back to these systems and
326
00:22:23.119 --> 00:22:30.400
some that are OT adjacent such as colonial pipeline and hospitals. Right,
327
00:22:31.519 --> 00:22:35.240
we do know that those systems and both of those
328
00:22:35.359 --> 00:22:39.359
incidents or well those examples the are colonnal pipeline are
329
00:22:40.240 --> 00:22:44.039
name a hospital breach. In some cases there were OTS
330
00:22:44.319 --> 00:22:50.680
type data OT type critical information that was impacted and
331
00:22:50.759 --> 00:22:55.440
so they of course paid and due to the fact
332
00:22:55.440 --> 00:23:03.240
that there's if someone didn't trust these ransomware gangs to
333
00:23:03.400 --> 00:23:05.000
do what they say they were going to do, they
334
00:23:05.000 --> 00:23:08.440
would those ransomware games would be out of business. So
335
00:23:09.319 --> 00:23:13.960
if you pay them, they don't, they decryptor doesn't work,
336
00:23:15.000 --> 00:23:19.119
then it's no good in their ransomware gain job is
337
00:23:19.160 --> 00:23:25.359
over with at least for this thing. But for OT
338
00:23:26.440 --> 00:23:30.240
in this instance I talked about, they did not pay.
339
00:23:31.880 --> 00:23:36.839
I don't have enough data to know if OT direct
340
00:23:37.160 --> 00:23:42.440
on the control systems themselves, the Windows, HMI s, engineering workstations,
341
00:23:42.519 --> 00:23:48.400
DCS servers, SKATAS servers, if they've paid in those instances.
342
00:23:50.279 --> 00:23:55.160
But I'd say it's it's plausible, And it really comes
343
00:23:55.200 --> 00:23:59.160
down to the business decision of the plant owner, the
344
00:23:59.200 --> 00:24:06.200
CEO of the cut based on what the engineers at
345
00:24:06.240 --> 00:24:09.279
the lower level, Hey, can we get do we have backups?
346
00:24:09.319 --> 00:24:12.880
Can we get the vendors to come in? And so
347
00:24:13.160 --> 00:24:16.519
I really don't have enough information to say about do
348
00:24:16.680 --> 00:24:23.000
OT asset owners like a plant, like a site, like
349
00:24:23.079 --> 00:24:29.559
a you know that directly operates h O T, if
350
00:24:29.599 --> 00:24:34.839
they trust these ransomware games or not. It may just
351
00:24:34.960 --> 00:24:41.440
roll up higher than them about that. Uh. Also, there's
352
00:24:41.680 --> 00:24:48.880
usually an advice from a ransomware negotiator that's a third
353
00:24:48.880 --> 00:24:54.559
party that specializes in negotiating with ransom so they may
354
00:24:54.720 --> 00:24:58.359
advise to pay or not to pay, or to get
355
00:24:58.359 --> 00:25:02.759
a reduced pain as well. So it's very, very complicated.
356
00:25:03.559 --> 00:25:09.200
I know I didn't answer your question directly, but in
357
00:25:09.359 --> 00:25:13.240
the instances we've seen, we have seen them not pay
358
00:25:13.319 --> 00:25:17.680
and we have seen them pay and what's OT or not.
359
00:25:18.279 --> 00:25:21.880
So so coming back to our theme here, you know,
360
00:25:22.000 --> 00:25:25.680
lessons from incidents. You know, the lesson from this incident is,
361
00:25:26.039 --> 00:25:29.680
you know, get rid of that firewall, use the existing infrastructure,
362
00:25:30.119 --> 00:25:33.519
and you know, look at your backups. I mean, if
363
00:25:33.559 --> 00:25:35.839
the backups are encrypted, it's it's all over. That makes
364
00:25:35.839 --> 00:25:39.240
perfect sense. What else have you got? What else you
365
00:25:39.240 --> 00:25:41.640
know have you been running into lately? That that's that's
366
00:25:41.680 --> 00:25:42.720
interesting and noteworthy.
367
00:25:43.119 --> 00:25:46.440
Yeah, I mean it's basically boths down to either ransomware
368
00:25:46.519 --> 00:25:51.160
or commodity malware. So I've got I've got another example
369
00:25:51.200 --> 00:25:56.000
about ransomware. Electric utility was impacted by ransomware on the
370
00:25:56.039 --> 00:26:00.319
IT side, but they had a good incident response plan
371
00:26:00.960 --> 00:26:07.640
and they severed the IT and OT connections there and
372
00:26:08.839 --> 00:26:13.200
even down to the power plant type networks, and so
373
00:26:13.319 --> 00:26:17.160
that was really amazing, and so that's a good story,
374
00:26:17.720 --> 00:26:22.680
and we were able to actually verify the IT team
375
00:26:22.720 --> 00:26:25.599
were able to verify that the threat actor, the ransomware
376
00:26:25.640 --> 00:26:31.200
game Quantum, were scanning the OT DMZ, but they didn't
377
00:26:31.200 --> 00:26:34.839
get a chance to get be let through. We did
378
00:26:34.920 --> 00:26:40.880
do a full assessment of their DMZ and looked at
379
00:26:40.960 --> 00:26:46.519
the domain control of the firewalls and even the domain
380
00:26:46.559 --> 00:26:51.480
controller and the firewalls and others down inside the OT networks,
381
00:26:52.359 --> 00:26:54.920
and we found that that they were actually pretty lucky
382
00:26:54.960 --> 00:26:58.400
because they had some weakness in some of the firewalls.
383
00:26:58.400 --> 00:27:01.960
So eventually do if they had enough time, if the
384
00:27:02.119 --> 00:27:05.440
ransomware actor had persisted long enough, they could have gotten
385
00:27:05.440 --> 00:27:09.119
through that firewall, made it to the DMZ And the
386
00:27:09.160 --> 00:27:12.960
active directory had some weaknesses as well, and they could
387
00:27:13.000 --> 00:27:18.359
have gotten domain access domain admin and pivoted to the
388
00:27:18.400 --> 00:27:22.920
OT network. But the great thing is highlight again that
389
00:27:23.160 --> 00:27:25.720
they had a good instant response plan. They were able
390
00:27:25.759 --> 00:27:29.279
to segment quickly and then they were able to have
391
00:27:29.680 --> 00:27:34.640
their OT vendor in this case it was Emerson Ovation
392
00:27:34.880 --> 00:27:39.920
were able to go on site and they were able
393
00:27:40.000 --> 00:27:43.519
to not only take the IOCs that we had from
394
00:27:43.519 --> 00:27:48.759
the ransomware, but they were able to sweep because it
395
00:27:48.799 --> 00:27:52.000
was their contract to do so, to look in the
396
00:27:52.000 --> 00:27:56.759
POC logs, the OT workstations, endpoint protection and all that stuff.
397
00:27:57.119 --> 00:27:59.920
So we all worked in concert together in this incident.
398
00:28:01.119 --> 00:28:05.119
And then actually they hardened the firewalls, hardened the domain controllers,
399
00:28:05.200 --> 00:28:11.799
hardened the workstation configurations before doing anything else. They did
400
00:28:11.839 --> 00:28:18.000
all of that. When the ransomware was eradicated and hardened,
401
00:28:19.559 --> 00:28:22.440
then they said, okay, now we'll reconnect everything back the
402
00:28:22.440 --> 00:28:25.640
way it was. So that was a really great lesson
403
00:28:25.720 --> 00:28:30.640
learned with another ransomware, and it wasn't a direct impact
404
00:28:30.640 --> 00:28:37.160
to OT, but this is a great opportunity to leverage
405
00:28:37.200 --> 00:28:40.400
that incident response plan that they had.
406
00:28:43.440 --> 00:28:49.119
So nate the concept of separating it from OT networks
407
00:28:49.160 --> 00:28:52.920
in an emergency. This is the concept that I see increasingly.
408
00:28:53.839 --> 00:28:56.559
I mean, I think we've reported on the show here
409
00:28:56.680 --> 00:29:00.559
a few times that this is what the TSAD demands
410
00:29:00.599 --> 00:29:05.160
of pipeline operators, petrochemical pipeline operators ever since colonial the
411
00:29:05.240 --> 00:29:08.000
ability to separate the networks in an emergency so that
412
00:29:08.079 --> 00:29:10.799
you can keep the pipeline running wild it's being cleaned up.
413
00:29:12.359 --> 00:29:18.200
I haven't I'm told I haven't actually read the translated
414
00:29:18.200 --> 00:29:20.680
the Danish law, but apparently in Denmark there's a recent
415
00:29:20.920 --> 00:29:24.720
law in the last twelve months saying exactly the same thing.
416
00:29:25.000 --> 00:29:27.559
You know, the TSA applies to pipelines and rails, in
417
00:29:27.640 --> 00:29:31.440
Denmarket applies to critical infrastructure, and it says, in an emergency,
418
00:29:31.480 --> 00:29:33.119
you have to be able to separate. They call it
419
00:29:33.240 --> 00:29:39.039
islanding the industrial control network, and as Chris points out,
420
00:29:39.119 --> 00:29:45.920
it can be effective, but it relies on really rapid
421
00:29:46.279 --> 00:29:50.359
intrusion detection and rapid response because as Chris said, you know,
422
00:29:50.480 --> 00:29:53.839
the bad guys had been testing the OT firewall. If
423
00:29:53.839 --> 00:29:55.839
they had had just a little bit longer, they could
424
00:29:55.839 --> 00:30:00.400
have gone through. So, you know it, even though it
425
00:30:00.400 --> 00:30:05.119
it's you know, imperfect, it is a measure that I'm
426
00:30:05.200 --> 00:30:10.119
seeing increasingly required of critical infrastructure operators and you know,
427
00:30:10.240 --> 00:30:14.079
recommended to non critical operators as a as a measure
428
00:30:13.920 --> 00:30:20.119
that helps, especially on the incident response side. Have you
429
00:30:20.119 --> 00:30:22.039
got another example for us? I mean three is the
430
00:30:22.079 --> 00:30:24.240
magic number you've given us, you know, sort of two
431
00:30:24.319 --> 00:30:26.480
sets of insights. What what else have you got for us?
432
00:30:26.519 --> 00:30:28.079
And in terms of lessons.
433
00:30:27.799 --> 00:30:32.319
Learned, yeah, there's lessons learned I can just name a
434
00:30:32.319 --> 00:30:36.000
few other lessons learned from it, just about any attack, right,
435
00:30:37.119 --> 00:30:41.759
making sure that you have these at least Windows systems
436
00:30:41.799 --> 00:30:44.839
with antivise. In a lot of cases the OT network
437
00:30:44.839 --> 00:30:49.960
didn't have anti just basic antivirse, not necessarily an agent
438
00:30:50.160 --> 00:30:54.839
or EDR solutions. If you have those, great. If you
439
00:30:54.920 --> 00:30:58.720
don't have any antivirse, that's you need to get at
440
00:30:58.799 --> 00:31:04.079
least corner version of Windows or operating system and with antivirus,
441
00:31:05.599 --> 00:31:10.799
having good backups, having good that good vendor support. Now,
442
00:31:11.880 --> 00:31:17.279
this last incident we responded to was using a living
443
00:31:17.319 --> 00:31:21.599
off the land attack, So we responded to electric utility
444
00:31:21.799 --> 00:31:28.720
in Ukraine in twenty twenty two and it was a
445
00:31:28.759 --> 00:31:33.559
distribution utility that the attacker came in through the IT
446
00:31:33.880 --> 00:31:41.359
network deployed their typical wiper malware. This was the group
447
00:31:41.480 --> 00:31:45.400
APT forty four or Sandworm team, which has been targeting
448
00:31:45.400 --> 00:31:50.240
critical infrastructure around the world for quite a while, and
449
00:31:50.440 --> 00:31:55.440
they were able to pivot to the Skater system and
450
00:31:56.279 --> 00:32:02.960
used the feature of the Skater system to trip breakers
451
00:32:04.319 --> 00:32:07.000
using a tool that was built in the skate of
452
00:32:07.079 --> 00:32:10.519
system itself, So just giving it a list of breakers
453
00:32:10.519 --> 00:32:16.920
to trip and calling that executable in the system to
454
00:32:17.359 --> 00:32:23.440
trip those breakers on. Behalf of the attackers is the
455
00:32:23.799 --> 00:32:32.160
too long did read of that incident? And so the
456
00:32:32.279 --> 00:32:38.000
lesson learned here is targeted attacks. They're going to not
457
00:32:38.160 --> 00:32:42.039
use malware. They're going to use the features or the
458
00:32:42.240 --> 00:32:46.880
inherent vulnerabilities in an OT network stealing valid credentials like
459
00:32:46.920 --> 00:32:54.359
an operator workstation or an engineering administrator account. And if
460
00:32:54.400 --> 00:33:01.279
you can even spearfish an engineer or administrator network AM
461
00:33:01.519 --> 00:33:04.960
on the IT network and you don't have good segmentation
462
00:33:05.119 --> 00:33:09.759
of roles from IT to O T, then that's that
463
00:33:10.680 --> 00:33:13.880
attacker is going to use every one of those tools
464
00:33:14.640 --> 00:33:22.359
to evade detection, to bypass your normal detections because they're
465
00:33:22.559 --> 00:33:27.240
they're coming in as a valid user. So the lessons
466
00:33:27.319 --> 00:33:34.680
learned there is to limit the amount of administrative access.
467
00:33:34.960 --> 00:33:38.440
And this is you know, role based authentication, right and
468
00:33:39.000 --> 00:33:41.680
does the person that got promoted and now is in
469
00:33:41.680 --> 00:33:45.640
a different department, does he still need ADMIN rights? You know,
470
00:33:46.480 --> 00:33:51.039
does this person having enough control for just their area
471
00:33:51.119 --> 00:33:55.119
only or their responsibilities too wide? And now we say, okay,
472
00:33:55.119 --> 00:34:00.720
we need to reduce the amount of admin. Do we
473
00:34:00.759 --> 00:34:05.839
require two factor authentication or even hardware two factor authentication
474
00:34:06.079 --> 00:34:11.760
to really reduce the attacker down to an insider threat,
475
00:34:12.440 --> 00:34:16.360
because remotely that's very hard to do to to to
476
00:34:16.360 --> 00:34:24.199
to bypass hardware token based two factor authentication. And so
477
00:34:24.280 --> 00:34:28.360
there's some there's some living off the land guides out there.
478
00:34:28.679 --> 00:34:33.519
The US government DEWE has put out a threat hunting
479
00:34:33.559 --> 00:34:38.599
guy for living off the land attacks after the volt
480
00:34:38.639 --> 00:34:45.480
typhoon announcements last year. But I would also go as
481
00:34:45.559 --> 00:34:53.199
step above and beyond that, learning good ways to detect
482
00:34:53.440 --> 00:34:58.480
anomalous logins, even from your own folks. If it's out
483
00:34:58.519 --> 00:35:04.000
of a normal time, out of a normal location, you're
484
00:35:04.039 --> 00:35:06.519
really gonna have to have some tuning on some of
485
00:35:06.559 --> 00:35:11.719
these detections. And the only way to really test those
486
00:35:11.800 --> 00:35:15.639
is with a red team that's trying to be quiet
487
00:35:15.679 --> 00:35:23.400
and not trigger your detections. And that is some of
488
00:35:23.440 --> 00:35:29.199
the more advanced asset owners and end users they're using
489
00:35:29.559 --> 00:35:33.679
leveraging red teams, hiring red teams like what we do
490
00:35:33.719 --> 00:35:36.320
at Mandiant to come in and see if we can
491
00:35:36.639 --> 00:35:39.480
do living off the land attacks to bypass their detections.
492
00:35:42.639 --> 00:35:46.400
Since Chris mentioned it but moved on before we could
493
00:35:46.400 --> 00:35:49.440
actually define it, let me just for listeners, living off
494
00:35:49.480 --> 00:35:53.920
the land is the process by which an attacker, rather
495
00:35:53.960 --> 00:35:58.960
than using their own malicious tooling, would make use of
496
00:35:59.280 --> 00:36:04.280
legitimate software or functionality of the system they're attacking to
497
00:36:04.400 --> 00:36:09.840
perform malicious actions on it. It's been a growing trend
498
00:36:10.639 --> 00:36:15.320
in recent years, I believe, because it's so effective in
499
00:36:15.360 --> 00:36:20.039
that it is so difficult to detect. You know, you
500
00:36:20.039 --> 00:36:23.280
could spot malware with certain kinds of tools, but can
501
00:36:23.280 --> 00:36:27.519
you spot somebody doing things with legitimate aspects of Windows
502
00:36:27.599 --> 00:36:31.199
or whatever you might be using. It sounds though like
503
00:36:31.760 --> 00:36:36.840
Chris is talking about detecting living off the land tactics,
504
00:36:36.840 --> 00:36:38.280
which seems difficult to Andrew.
505
00:36:38.880 --> 00:36:41.079
That's right, I mean, I you know, have been have
506
00:36:41.159 --> 00:36:45.320
been following Living off the land to a degree. You
507
00:36:45.360 --> 00:36:49.679
know the what's the right where the short The short
508
00:36:49.679 --> 00:36:52.599
answer is you run an anti virus scan on a
509
00:36:52.599 --> 00:36:54.800
machine that's been compromised by a living off the land
510
00:36:55.199 --> 00:36:59.400
attack and it comes up squeaky clean, there's nothing nasty
511
00:36:59.440 --> 00:37:02.360
on the machine. And what I heard Chris say is
512
00:37:02.400 --> 00:37:08.000
that this is because the bad guys are using normal mechanisms,
513
00:37:08.079 --> 00:37:12.599
especially remote access, to log into these systems as if
514
00:37:12.639 --> 00:37:16.440
they were normal users and use the tools on the
515
00:37:16.480 --> 00:37:20.320
machine to attack the network or you know, to wait
516
00:37:20.400 --> 00:37:22.920
for a period of time until it's opportune, and then
517
00:37:22.960 --> 00:37:27.440
attack the network. And you know what I heard him
518
00:37:27.440 --> 00:37:31.079
say is that because it's a lot of a lot
519
00:37:31.119 --> 00:37:34.679
of this is remote access, he says, you can detect
520
00:37:34.679 --> 00:37:39.280
this by focusing hard on your remote access system. A.
521
00:37:39.440 --> 00:37:41.800
You can prevent it by throwing in some hardware based
522
00:37:42.000 --> 00:37:44.159
two factor. You know, that will solve a lot of
523
00:37:44.199 --> 00:37:46.800
the problem, not necessarily all of it. There's always vulnerabilities
524
00:37:46.800 --> 00:37:50.360
and zero days, but two factor helps enormously. It's way
525
00:37:50.360 --> 00:37:53.880
better than not having two factor. But that's preventive. On
526
00:37:53.920 --> 00:37:57.320
the detective side, he said, pay attention to your remote
527
00:37:57.360 --> 00:38:03.719
access If normal users are logging in at strange times,
528
00:38:04.639 --> 00:38:07.960
that should raise a red flag. If normal users are
529
00:38:08.199 --> 00:38:11.119
logging in from strange places, the IP address coming in
530
00:38:11.199 --> 00:38:14.559
is from China, Well is Fred in China this week?
531
00:38:14.719 --> 00:38:18.400
No he's not. So you know what I heard was
532
00:38:18.440 --> 00:38:22.639
one way to to you know, help detect living off
533
00:38:22.639 --> 00:38:25.960
the land techniques is to pay close attention in your
534
00:38:25.960 --> 00:38:30.880
intrusion detection system to the intelligence that you're getting about
535
00:38:31.119 --> 00:38:37.079
remote users logging in. So one more question, you know,
536
00:38:37.119 --> 00:38:38.519
we talked to a lot of folks on the on
537
00:38:38.559 --> 00:38:40.880
the podcast, A lot of them are our vendors with
538
00:38:41.239 --> 00:38:44.519
technology that we talk about, and you know, sort of
539
00:38:44.559 --> 00:38:47.719
a consistent theme for most of these vendors most of
540
00:38:47.760 --> 00:38:54.840
these technologies is operational benefits. Yes, the technology whatever it is,
541
00:38:54.840 --> 00:38:59.760
is helping with cybersecurity, but often this stuff helps with
542
00:39:00.440 --> 00:39:05.440
just general operations in sometimes surprising ways. We've been talking
543
00:39:05.440 --> 00:39:08.679
about incidents and lessons, and you know, a lot of
544
00:39:08.719 --> 00:39:13.239
what you do is incident response. Are there operational benefits
545
00:39:13.000 --> 00:39:15.719
that you run into that people say, you know, I
546
00:39:15.800 --> 00:39:18.800
did what you told me and everything is working smooth
547
00:39:19.039 --> 00:39:21.440
more smoothly than not just on the security side. Can
548
00:39:21.480 --> 00:39:23.880
you talk you do you have anything like that for us?
549
00:39:24.360 --> 00:39:27.480
Oh? Absolutely. And one of the things that I always
550
00:39:27.719 --> 00:39:31.440
teut is looking at your network, looking at the packet
551
00:39:31.480 --> 00:39:37.679
captures in the network can aid in not just cybersecurity benefits,
552
00:39:37.719 --> 00:39:41.760
but these operational benefits. You can see things like switch
553
00:39:41.800 --> 00:39:48.199
failures happening, TCP retransmissions happening. All this traffic may be
554
00:39:48.320 --> 00:39:51.880
went like your Windows hmis maybe trying to reach out
555
00:39:51.880 --> 00:39:55.280
to Windows Update, but it's blocked by the it OT
556
00:39:55.440 --> 00:40:00.440
firewall or anything else. It may not have a connection
557
00:40:00.519 --> 00:40:03.480
at all as trying to reach out all this unnecessary
558
00:40:03.599 --> 00:40:12.159
traffic or indications of proper configurations, misconfigurations and things like that.
559
00:40:12.440 --> 00:40:15.000
So just looking at your network with some of these
560
00:40:15.039 --> 00:40:19.079
tools that are out there, free tools, pay tools, ICs
561
00:40:19.079 --> 00:40:22.559
specific tools, or IT specific tools. It doesn't matter if
562
00:40:22.559 --> 00:40:24.360
you look at If you take any one of those,
563
00:40:24.400 --> 00:40:28.079
say just even wireshark and look in your OT network,
564
00:40:30.239 --> 00:40:34.920
you can get an idea on what traffic doesn't need
565
00:40:34.960 --> 00:40:39.159
to be there that you can eliminate it, make your
566
00:40:39.400 --> 00:40:44.440
improvements to the system, and now I have better visibility
567
00:40:44.519 --> 00:40:49.079
if there is an incident. I can easier to detect
568
00:40:49.119 --> 00:40:52.719
if there's a cyber incident or if something's operationally wrong
569
00:40:54.199 --> 00:40:57.960
like a switch failure or something, and so there's a
570
00:40:58.039 --> 00:41:02.760
really great benefit there. Also helps improve reliability. We've done
571
00:41:02.840 --> 00:41:11.000
an assessment at a company that had a conveyor belt
572
00:41:12.039 --> 00:41:14.920
that they were having problems with. If the conveyor belt
573
00:41:15.000 --> 00:41:17.320
wasn't timed exactly right, if they had too much latency
574
00:41:17.320 --> 00:41:20.639
on the network, the conveyor belt would stop and all
575
00:41:20.639 --> 00:41:23.400
the things on the conveyor belt would just go everywhere,
576
00:41:23.440 --> 00:41:25.880
and it was a disaster. So we just looked in
577
00:41:25.960 --> 00:41:29.159
the network, Oh, you got all these TCP re transmissions
578
00:41:29.239 --> 00:41:32.480
is and you look at the map in the in
579
00:41:32.559 --> 00:41:34.559
the software and say, oh, it's coming from these two
580
00:41:34.599 --> 00:41:38.840
IP addresses. Oh, we know what those equipment is. And
581
00:41:38.880 --> 00:41:40.920
we had the network person coming, Oh, I've been trying
582
00:41:40.960 --> 00:41:44.639
to figure this out for weeks and just looking looking,
583
00:41:44.840 --> 00:41:47.760
just using a duel like that. Uh, they were able
584
00:41:47.800 --> 00:41:50.199
to find and fix the problem and they fixed their
585
00:41:50.280 --> 00:41:56.639
latency issue because of that. So going back to you know,
586
00:41:57.000 --> 00:42:04.079
incident response, having these having an incant response plan. A
587
00:42:04.119 --> 00:42:13.400
lot of ot already has this because of disasters, fire, floods, storms,
588
00:42:16.320 --> 00:42:23.800
it spills, air releases, safety issues, and that's all part
589
00:42:23.800 --> 00:42:29.880
of their normal disaster recovery or incident response plans. If
590
00:42:29.920 --> 00:42:32.199
you already have one of those, you've already done ninety
591
00:42:32.239 --> 00:42:35.559
percent of the work to have a cyber incident response plan.
592
00:42:36.400 --> 00:42:40.960
You just now have a cyber incident response added to that.
593
00:42:41.199 --> 00:42:44.039
So that's the whole premise behind things like ICs or
594
00:42:44.199 --> 00:42:50.679
ICs incident Command system for industrial control systems and having
595
00:42:50.719 --> 00:42:56.840
that say, chief person in charge of cybersecurity for a
596
00:42:57.000 --> 00:42:59.679
site for a paper mill, for a power plant, for
597
00:43:00.559 --> 00:43:04.519
manufacturing facility, even though that's not your day to day
598
00:43:04.599 --> 00:43:08.599
job all the time. If you say the lead that
599
00:43:09.679 --> 00:43:14.440
and you say you have multiple plants, have multiple leads
600
00:43:14.800 --> 00:43:21.159
for those plants. That still the every decision will go
601
00:43:21.239 --> 00:43:24.079
through the plant manager, the general manager of the plant
602
00:43:24.559 --> 00:43:28.440
or site. But at least you have someone that is
603
00:43:28.480 --> 00:43:32.000
in charge of cybersecurity, just like you have a designated
604
00:43:32.000 --> 00:43:39.480
firewatch person or anything else. So if you take safety
605
00:43:39.519 --> 00:43:42.639
culture that we've known about for over one hundred years
606
00:43:43.840 --> 00:43:49.599
and mold your cybersecurity culture to fit with that, things
607
00:43:49.639 --> 00:43:53.119
will make a lot more sense. We've already we've already
608
00:43:53.119 --> 00:43:57.000
invented this. We're not reinventing the wheel here now. We're
609
00:43:57.039 --> 00:44:02.519
just including another paradigm of cyber security, network security, and
610
00:44:02.679 --> 00:44:07.000
endpoint security into these things that we have been doing.
611
00:44:08.199 --> 00:44:11.960
There's a fire, okay, let's put out incident response. So
612
00:44:16.199 --> 00:44:19.119
and if you have a plan, that's great. If you
613
00:44:19.119 --> 00:44:24.039
don't have a plan and you run around, that's not good.
614
00:44:24.280 --> 00:44:26.320
So if you have a plan, you can at least
615
00:44:26.360 --> 00:44:30.079
prepare for it, and sometimes that's the win. Being prepared
616
00:44:30.880 --> 00:44:32.239
is better than not being prepared.
617
00:44:33.280 --> 00:44:35.719
Well, this has been tremendous. Thank you Chris for joining us.
618
00:44:35.960 --> 00:44:37.760
Before we let you go, can you sum up for
619
00:44:37.760 --> 00:44:39.840
our listeners what you know? What are the key points
620
00:44:39.840 --> 00:44:40.840
we should take away here.
621
00:44:41.639 --> 00:44:44.239
Sure if you didn't listen to a single thing I said,
622
00:44:45.280 --> 00:44:51.880
and you can listen to these three things. Collaborate, plan
623
00:44:52.039 --> 00:44:56.760
and practice. So collaborate. You get your IT teams, talking
624
00:44:56.800 --> 00:45:02.400
to your OT teams, talking to your manufacturers, and identify
625
00:45:02.480 --> 00:45:05.920
the right roles within each of those and make sure
626
00:45:06.039 --> 00:45:08.960
you get together and talk about these things. Have some
627
00:45:09.079 --> 00:45:14.559
donuts and coffee. So collaborating. Knowing who is in charge
628
00:45:14.559 --> 00:45:18.159
of what is half the battle. Knowing who to call
629
00:45:18.199 --> 00:45:24.719
when plan, having an incent response plan, or including OT
630
00:45:24.920 --> 00:45:29.079
security in your incident response plan and or engineering procedures.
631
00:45:30.000 --> 00:45:36.960
That's gonna help when an incident impacts OT directly or indirectly.
632
00:45:38.280 --> 00:45:43.320
And then practice even can start with a simple question, Hey,
633
00:45:44.079 --> 00:45:47.320
what would we do in an incident or even going
634
00:45:47.440 --> 00:45:53.440
to having a tabletop exercise collecting logs from a PLC
635
00:45:53.800 --> 00:45:56.880
of security logs? How long does that take? How many
636
00:45:56.880 --> 00:46:00.119
devices do we have? If the general manager says how
637
00:46:00.119 --> 00:46:01.480
long is this going to take to pull all the
638
00:46:01.519 --> 00:46:04.719
logs from all of our systems, you won't be able
639
00:46:04.719 --> 00:46:07.360
to say I don't know. You'll have to say no,
640
00:46:07.440 --> 00:46:11.599
this will take two hours and forty five minutes. Because
641
00:46:11.599 --> 00:46:18.719
we've tested it so collaborate plan and practice. If you
642
00:46:18.920 --> 00:46:22.880
need help with OT security or IT security, we do that.
643
00:46:22.960 --> 00:46:29.079
At Mandian. We offer incidant response retainer that covers IT
644
00:46:29.360 --> 00:46:31.760
and OT. There's no separate retainer. If you have an
645
00:46:31.800 --> 00:46:35.559
IT incident and don't need OT, not a problem. If
646
00:46:35.599 --> 00:46:38.639
you have OT only incident, not a problem. If it's
647
00:46:39.000 --> 00:46:42.800
IT cloud and OT all at the same time, we
648
00:46:42.880 --> 00:46:45.719
can help you around the world twenty four to seven.
649
00:46:47.400 --> 00:46:50.239
And lastly, if you want to learn more about this,
650
00:46:50.400 --> 00:46:54.000
you can reach out to me Chris's drunk at Google
651
00:46:54.239 --> 00:47:00.679
dot com, my email LinkedIn social media blue Sky, and
652
00:47:01.079 --> 00:47:03.679
we check out some of our blogs on the Google
653
00:47:03.719 --> 00:47:09.360
Cloud or Mandian security blog. We have great content out
654
00:47:09.400 --> 00:47:16.239
there that is actual actionable not marketing fluff. It is
655
00:47:16.360 --> 00:47:23.079
actual actionable reports. The next M Trends Report is coming
656
00:47:23.119 --> 00:47:27.480
out next week r s a time prime end of April,
657
00:47:28.000 --> 00:47:31.760
so that's a free report. It's a great report to
658
00:47:32.119 --> 00:47:35.119
look at and gain some insights on what we've been
659
00:47:35.159 --> 00:47:38.920
responding to over the last year. And with that, I
660
00:47:39.159 --> 00:47:45.280
appreciate it. Collaborate plan practice, Andrew.
661
00:47:45.320 --> 00:47:49.440
That just about concludes your interview with Chris Sistroke. Do
662
00:47:49.480 --> 00:47:51.800
you have any final words to add on to his
663
00:47:52.039 --> 00:47:53.960
to take us out with today.
664
00:47:53.559 --> 00:47:57.360
I mean Chris, Chris summed up collaborate plan practice. You know,
665
00:47:57.760 --> 00:48:00.880
what I heard, especially earlier in the interview, was, you know,
666
00:48:01.559 --> 00:48:05.440
do the basics, guys. They some people call it basic
667
00:48:05.519 --> 00:48:08.559
hygien is. Basically do on an OT network as much
668
00:48:08.599 --> 00:48:10.320
as you can of what you would do on an
669
00:48:10.360 --> 00:48:13.360
IT network. Put a little anti virus in on the
670
00:48:13.360 --> 00:48:17.880
systems that tolerated, you know, get some backups, get some
671
00:48:18.039 --> 00:48:20.679
offsite backups so that if the bad guys get in
672
00:48:20.719 --> 00:48:23.639
they can't encrypt the off site backups. There's somewhere else,
673
00:48:25.880 --> 00:48:31.880
you know, look for the vendors leaving behind internet connections,
674
00:48:32.639 --> 00:48:36.119
get rid of them, you know. And you know, on
675
00:48:36.159 --> 00:48:38.400
the in terms of living off the land, you know,
676
00:48:38.440 --> 00:48:42.800
he gave some very concrete advice that I've never heard before, saying, look,
677
00:48:43.239 --> 00:48:46.280
these people are coming in as users. Get two factor.
678
00:48:46.360 --> 00:48:49.559
Two factor will do a lot to breaking up living
679
00:48:49.559 --> 00:48:52.519
off the land attacks. And in your intrusion detection systems,
680
00:48:52.559 --> 00:48:55.639
look hard at what your remote users are doing, and
681
00:48:55.639 --> 00:48:58.719
if it seems at all unusual, that's the clue that
682
00:48:58.760 --> 00:49:02.239
you're being a tact And you know, in terms of
683
00:49:02.280 --> 00:49:05.519
his collaborate, plan and practice. I really liked his his
684
00:49:05.519 --> 00:49:12.599
his uh the fire warden analogy, saying, look, you know,
685
00:49:12.639 --> 00:49:16.320
if you have an industrial site that is flammable, okay,
686
00:49:16.840 --> 00:49:20.039
your firewarden does not just sit on her hands until
687
00:49:20.039 --> 00:49:22.639
the place bursts into flames. Okay. The fire warden is
688
00:49:22.679 --> 00:49:26.360
someone who's active in terms of actively h you know,
689
00:49:26.960 --> 00:49:31.079
looking at managing, raising the alarm when they see dangerous
690
00:49:31.119 --> 00:49:34.920
practices in this flammable plant. It's not you know, it's
691
00:49:34.920 --> 00:49:38.639
not just a reactive position. It's also a proactive position.
692
00:49:39.159 --> 00:49:45.960
And we need that for cybersecurity because basically every site is,
693
00:49:46.119 --> 00:49:52.480
you know, in a sense, a flammable cybersecurity situation. So
694
00:49:52.559 --> 00:49:55.079
it's not just that they sit on their hands until
695
00:49:55.079 --> 00:49:57.920
there's an incident and then they're in charge. They are
696
00:49:58.000 --> 00:50:01.079
actively looking around, just like a fire warden wouldn't say,
697
00:50:01.119 --> 00:50:03.760
you know, we shouldn't be doing this. You know, my
698
00:50:03.920 --> 00:50:06.280
job is not just to put the fire out when
699
00:50:06.320 --> 00:50:09.480
it occurs, or coordinate putting the fire out. My job
700
00:50:09.559 --> 00:50:12.159
is to help prevent these things. And so I love
701
00:50:12.239 --> 00:50:15.119
that analogy. That makes so much sense. Anyhow, That's what
702
00:50:15.159 --> 00:50:16.119
I took from the episode.
703
00:50:16.880 --> 00:50:20.000
Sure well, thank you Chris for speaking with us and
704
00:50:20.000 --> 00:50:22.679
Andrews always thank you for speaking of it.
705
00:50:22.679 --> 00:50:23.960
It's always a great pleasure.
706
00:50:24.000 --> 00:50:24.280
Thank you.
707
00:50:25.199 --> 00:50:29.559
This has been the Industrial Security Podcast from Waterfall. Thanks
708
00:50:29.559 --> 00:50:31.239
to everyone out there listen.