WEBVTT
00:00:03.680 --> 00:00:06.000
Sometimes it's a policy shift.
00:00:06.240 --> 00:00:08.480
Sometimes it's a system that fails.
00:00:08.720 --> 00:00:10.960
Or it could be that a market moves.
00:00:11.519 --> 00:00:14.080
Often leaders find out too late.
00:00:14.320 --> 00:00:19.679
The signal was there, but nobody turned it into something they could act on before the moment passed.
00:00:20.320 --> 00:00:25.039
What the hell happened is a weekly briefing for leaders who would rather be ready.
00:00:25.440 --> 00:00:32.399
I'm Michael Crackett, and I've spent over 30 years working in situations where planning gets interrupted by reality.
00:00:32.799 --> 00:00:38.399
This program looks at not just what the hell happened, but also what can be done about it.
00:00:40.159 --> 00:00:42.960
Welcome back, everybody, to another episode of the podcast.
00:00:43.119 --> 00:00:49.679
I'm truly excited to have you here with me again, and I appreciate your interest and taking the time to listen to these episodes.
00:00:49.759 --> 00:01:09.760
It means a lot to me, and I really appreciate any comments that you'd like to leave, and it'll help drive us to future episodes, any topics that you may be interested in hearing about, and also give you a chance to leave a review, let other folks know what you think, and give me a thumbs up if you like what you're hearing, or let me know some things that I can do to improve.
00:01:09.840 --> 00:01:11.120
It all helps us get better.
00:01:11.200 --> 00:01:13.040
So anyway, I'm glad that you're here.
00:01:13.200 --> 00:01:14.560
Let's jump into this episode.
00:01:14.719 --> 00:01:19.760
The title for this episode is When Crisis Don't Wait for Their Turn.
00:01:20.640 --> 00:01:23.040
What exactly does that mean?
00:01:23.439 --> 00:01:31.359
What it means is a lot of plans, most plans in fact, are written around focusing on one crisis at a time.
00:01:31.439 --> 00:01:38.640
When that crisis turns up, they activate the plan or the plan goes into action and the crisis is dealt with.
00:01:39.040 --> 00:01:52.079
But what a lot of plans, in fact, most plans don't address is how to deal with a situation that comes up, you're working on it in the plan, and all of a sudden something else goes wrong, and now you're facing two crises at the same time.
00:01:52.159 --> 00:02:01.040
They may or may not be related and they may or may not be connected, but they will have an impact on each other and definitely have an impact on your organization.
00:02:01.120 --> 00:02:03.439
So that's what we're going to talk about today.
00:02:03.840 --> 00:02:07.920
So with each episode, I try to open with a single moment.
00:02:08.159 --> 00:02:10.639
A blue to gray illusion is what I call it.
00:02:10.800 --> 00:02:19.599
It's a blue sky moment where the status says that everything was fine in the organization, but in reality the skies were already turning gray.
00:02:19.759 --> 00:02:23.360
Things were getting dark, and things were happening behind the scenes.
00:02:23.599 --> 00:02:26.719
So here's this episode's blue to gray illusion.
00:02:27.919 --> 00:02:32.080
The leadership dashboards showed the inventory was moving normally.
00:02:32.560 --> 00:02:40.400
The distribution centers were packed with the product, but it couldn't be shelved, and the store shelves were actually empty.
00:02:41.280 --> 00:02:46.719
The system reported one reality, but the warehouse showed an obvious different reality.
00:02:47.680 --> 00:02:53.039
The status and all the indicators said that everything was okay, but the reality was much different.
00:02:53.520 --> 00:02:55.360
So what the hell happened?
00:02:55.680 --> 00:02:58.000
Well that's what we're going to talk about today.
00:02:58.240 --> 00:02:59.840
First let's look at readiness.
00:02:59.919 --> 00:03:02.319
Here's the readiness portion of this episode.
00:03:02.479 --> 00:03:05.439
Readiness is where we're going to look at what actually happened.
00:03:05.680 --> 00:03:13.680
We'll talk about the sequence of the events and the decisions that were made, and then we'll look at some of the moment that things went wrong.
00:03:14.159 --> 00:03:19.759
Today in readiness we're going to look at two organizations and three different incidents actually.
00:03:20.000 --> 00:03:25.840
They affect different industries, they happened during different decades, and they impacted different people.
00:03:26.240 --> 00:03:29.840
But they made the same catastrophic mistake in a similar way.
00:03:30.000 --> 00:03:31.680
So let's start first with a question.
00:03:31.919 --> 00:03:34.159
You have an organization and they hit a problem.
00:03:34.400 --> 00:03:38.400
A competent team makes the appropriate response according to the plan.
00:03:38.479 --> 00:03:42.719
You have confident people and they work fast to try and solve the problem.
00:03:43.199 --> 00:03:48.240
They run all of their standard corrective procedures and the procedures only make it worse.
00:03:48.639 --> 00:03:58.560
This doesn't happen because the procedures were slow or the response was delayed, and it didn't happen because the responders or the folks managing the situation didn't care.
00:03:59.039 --> 00:04:11.919
They were operating on a picture that was based on information that they thought was actually true, when in fact the information that they were operating on wasn't the picture that was actually true in reality.
00:04:12.400 --> 00:04:17.360
So what happens when the correction is based on the wrong diagnosis?
00:04:17.600 --> 00:04:23.519
A diagnosis that's based on information that's incorrect, inaccurate, or not timely.
00:04:24.319 --> 00:04:26.639
Let's look at these two case studies.
00:04:27.199 --> 00:04:29.360
The first one is Knight Capital.
00:04:29.839 --> 00:04:35.839
It's an investment group, and the problem happened on October first, 2012.
00:04:36.480 --> 00:04:44.959
On this date, Knight Capital deployed a new trading software and it went out to eight servers, but one server got missed.
00:04:45.120 --> 00:04:56.959
It had a legacy code on it called PowerPeg, and that stayed active on that one individual server while the new trading software deployed as it was supposed to to the rest of the servers.
00:04:58.240 --> 00:05:04.399
At eight oh one AM there were ninety-seven emails that were generated that showed an error.
00:05:04.560 --> 00:05:08.240
They were automated emails, and they identified the problem.
00:05:08.800 --> 00:05:17.199
The problem here was, though, that they were not configured as high priority alerts, and so nobody actually read the emails before the market opened.
00:05:17.920 --> 00:05:20.319
The market opened at 9 30 AM.
00:05:20.800 --> 00:05:26.879
The unpatched server began to execute millions of erroneous trades automatically.
00:05:27.279 --> 00:05:34.079
The team seized the problem and they ran their bad deployment response as it was outlined in their plan.
00:05:35.040 --> 00:05:39.920
It called for them to uninstall the new code from the functioning servers, so that's what they did.
00:05:40.560 --> 00:05:45.279
And then that action reactivated the legacy code across all eight servers.
00:05:46.560 --> 00:05:59.920
What happened then was what they thought was the fix actually served to multiply the damage because the old server and the old code was now activated, running on the wrong information.
00:06:00.160 --> 00:06:07.920
The result was in the time of forty five minutes, over four hundred million dollars in losses were accrued by the company.
00:06:08.079 --> 00:06:11.680
The deployment wasn't the error that killed this company.
00:06:11.920 --> 00:06:14.240
That wasn't the critical impact.
00:06:14.399 --> 00:06:22.399
The critical impact was a corrective action that was built and deployed on a misdiagnosis of what the actual problem was.
00:06:23.120 --> 00:06:30.879
The actual problem was that the trading software had not deployed on all of the servers, it had missed one.
00:06:32.079 --> 00:06:39.360
But that was not actually noticed until after what they thought was wrong was addressed in the wrong way.
00:06:40.240 --> 00:06:45.360
So the team was competent, the team did what they were supposed to, but the picture was wrong.
00:06:45.600 --> 00:06:47.040
Here's another case.
00:06:47.279 --> 00:06:48.800
This one involves Target.
00:06:48.879 --> 00:06:51.279
It's actually Target in Canada.
00:06:51.519 --> 00:06:53.360
They're called Target Canada.
00:06:53.759 --> 00:07:03.360
They acquired Zellers, which is another department type store in Canada, and they acquired the leases for one point eight billion dollars.
00:07:03.519 --> 00:07:12.720
And they committed to opening one hundred and twenty four stores simultaneously in Canada because the real estate costs demanded that they do so.
00:07:12.879 --> 00:07:14.319
So what happened?
00:07:14.959 --> 00:07:19.120
They stood up a new ERP system and it got populated manually.
00:07:19.360 --> 00:07:24.480
This happened under an extreme time pressure due to the deadline to get everything open.
00:07:24.959 --> 00:07:29.759
At launch, up to seventy percent of the data was found to be corrupted.
00:07:29.920 --> 00:07:35.680
There were wrong dimensions, wrong barcodes, wrong case pack counts.
00:07:36.000 --> 00:07:44.639
That happened because it had to be populated manually due to the time restraints and so some of the errors were made by the manual entry of the data.
00:07:45.120 --> 00:07:53.360
The automated replenishment system didn't know that that data was not accurate, and so it treated it as it was accurate.
00:07:54.319 --> 00:08:01.680
What happened next was the distribution centers overflowed with the wrong product while the store shelves sat empty.
00:08:02.000 --> 00:08:07.600
Everything that they needed to order was automated and sent to the distribution centers.
00:08:07.759 --> 00:08:14.319
However, once it arrived at the distribution centers, it never made it out of there and actually hit the store shelves.
00:08:14.800 --> 00:08:24.160
So what happened next was the leadership decided that they needed to disable the auto replenishment software as outlined in their plan because something was obviously wrong.
00:08:24.319 --> 00:08:31.199
And they decided that they would go manual, try to get the supplies out to the stores and make the deliveries.
00:08:31.439 --> 00:08:38.960
What that resulted in was the manual corrections were now feeding new bad data back into the same corrupted system.
00:08:39.200 --> 00:08:48.240
So they were still using their distribution system, but it was still relying on the bad data that had come from the improper system that was originally activated.
00:08:48.399 --> 00:08:52.480
They still had the dashboard that showed one reality, everything was fine.
00:08:52.720 --> 00:08:56.240
But the people in the warehouse were reporting something entirely different.
00:08:56.399 --> 00:09:00.480
The warehouses were overstocked, nothing on the shelf in the store.
00:09:01.200 --> 00:09:08.000
According to the filing on january fifteenth, twenty fifteen, one hundred and thirty three of the stores were closed.
00:09:08.159 --> 00:09:17.600
This resulted in seventeen thousand six hundred people who were now out of work and over five billion dollars in pre-tax losses for the company.
00:09:18.159 --> 00:09:27.039
Every corrective action, both automated and manual, was operating on a foundation that was wrong from the start.
00:09:29.360 --> 00:09:44.159
One other highly publicized incident, I just watched a movie on this as a matter of fact recently, and it really shed a lot of light on it, really a good movie, but it involved Deepwater Horizon and the disaster that resulted from that oil spill.
00:09:44.559 --> 00:09:52.960
This was a third organization, an entirely different industry, this time the oil and gas industry, not technology and not retail.
00:09:53.200 --> 00:10:01.600
In this incident, as you recall, the crew of the Deepwater Horizon misread a negative pressure test on the offshore oil rig.
00:10:02.399 --> 00:10:09.840
They interpreted the results through the lens of their last coherent mental model, and that assumed that the well was fine.
00:10:10.080 --> 00:10:13.919
The test made sense and they thought that they could proceed without any problem.
00:10:14.480 --> 00:10:17.200
In reality, the well wasn't sealed.
00:10:17.519 --> 00:10:23.759
The CSB documented this as a diagnostic failure interacting with a physical safety system failure.
00:10:24.080 --> 00:10:27.840
The information was actually there, and the information existent.
00:10:28.080 --> 00:10:31.519
The problem was the process was not followed.
00:10:32.000 --> 00:10:39.679
But what arrived at each decision point had not been verified against an independent read before the action was taken.
00:10:41.759 --> 00:10:44.320
You can find out more about this information.
00:10:44.480 --> 00:10:58.159
There's a lot of public information, a lot of movies and other research, after action reviews, a lot of things on the internet that are available that go into the details, but we're focused mainly here on the processes.
00:10:58.399 --> 00:11:12.480
In fact, I have a colleague that I work closely with who is a direct responder to this disaster in Deepwater Horizon, worked on the Gulf Coast at the time and was responsible for a portion of the response and the recovery after it happened.
00:11:12.639 --> 00:11:19.759
I hope to have him on and interview him about this and what the processes that failed as far as the response went.
00:11:20.000 --> 00:11:29.039
There were other issues that were involved there too, so hopefully I can get him on the podcast on a future episode and talk a little bit more in detail about that.
00:11:29.360 --> 00:11:32.720
But we'll also look at that more later in resilience.
00:11:32.799 --> 00:11:38.559
We're going to look at exactly where the gap lives and why it's so hard to see until it's too late.
00:11:39.120 --> 00:11:45.600
So both of these organizations had smart and capable people, and they were running hard at a real problem.
00:11:46.159 --> 00:11:48.000
But in the end both had made it worse.
00:11:48.159 --> 00:11:51.440
The question isn't whether they were competent because they were.
00:11:52.080 --> 00:12:00.320
The question is actually what information were they actually using to operate on and to base their response?
00:12:00.960 --> 00:12:02.879
Sit on that for a minute.
00:12:04.000 --> 00:12:05.679
That's the readiness picture.
00:12:06.000 --> 00:12:10.639
That's what happened and what it looked like from inside both of the organizations.
00:12:11.840 --> 00:12:13.759
Now let's move into resilience.
00:12:14.000 --> 00:12:21.360
Resilience is where we're going to stop looking at the events themselves and start looking at the mechanisms that made the events occur.
00:12:22.240 --> 00:12:27.360
Because these two failures aren't parallel stories that just happen to resemble each other.
00:12:27.600 --> 00:12:34.960
They share a similar architecture, and once you see it, you can't really unsee it inside your own organization.
00:12:36.639 --> 00:12:48.559
In both cases, both Night Capital and Target Canada, the information systems and the operational systems were separated completely before any corrective action was taken.
00:12:48.720 --> 00:12:55.360
This didn't happen gradually over time, and it wasn't because of negligence that had built up across months.
00:12:55.600 --> 00:13:02.720
This was a structural separation, and it was present before the first decision in the crisis was even made.
00:13:03.039 --> 00:13:10.799
And then every decision after that that was made downstream was built or based on a corrupted picture based on bad information.
00:13:11.279 --> 00:13:16.000
In the case of the Knight team, they had corrected for the wrong failure.
00:13:16.320 --> 00:13:21.600
Targets leadership had fed new bad data into an already broken system.
00:13:22.159 --> 00:13:26.080
These were competent people working in a wrong reality.
00:13:26.399 --> 00:13:28.480
And that's not a technology problem.
00:13:28.639 --> 00:13:30.480
That's not even a training problem.
00:13:30.720 --> 00:13:37.279
What that actually shows is an architectural problem, something in the structure that just wasn't working right.
00:13:37.519 --> 00:13:42.080
I spent thirty years in emergency management, and that teaches you a lot.
00:13:42.240 --> 00:13:46.080
It teaches you something about how systems can fail under pressure.
00:13:46.320 --> 00:13:51.279
And it rarely results from steps that were taken in the process that broke.
00:13:51.759 --> 00:13:57.679
People are always following the process, following the steps, taking the steps that are needed and doing their best.
00:13:57.919 --> 00:14:02.159
The steps are documented and they're trained and they're exercised.
00:14:02.559 --> 00:14:06.720
But what breaks is what happens in the space between those steps.
00:14:07.039 --> 00:14:14.799
This is where the information is moving from one part of the system to the next, and it's critical to making things work as they should.
00:14:15.279 --> 00:14:19.120
In both of these cases, that's exactly where the failures lived.
00:14:19.360 --> 00:14:27.120
For Night Capital, the team followed a legitimate, bad deployment response, as they were supposed to and as was outlined in the plan.
00:14:27.519 --> 00:14:39.440
The steps were correct, but the information that arrived at each step, which was about the servers that were affected, and that's what the root cause was, had not been verified before it was acted on.
00:14:40.240 --> 00:14:49.919
In Target Canada, the leadership followed the legitimate crisis response for the incident they were dealing with, the problem with the stock shelves in the stores.
00:14:50.159 --> 00:14:54.559
They identified the problem and disabled the automation system.
00:14:55.360 --> 00:15:01.759
They went on manual and the steps made sense for what they were dealing with according to what they thought was occurring.
00:15:02.399 --> 00:15:07.120
The problem was that the information that those steps were built on was corrupted from the start.
00:15:07.360 --> 00:15:13.840
In emergency management, one of the doctrine standards is real-time, accurate situational awareness.
00:15:14.080 --> 00:15:22.879
Not just having the information, but maintaining a verified, current picture of what is actually true as the conditions are always changing.
00:15:23.200 --> 00:15:26.639
The emphasis here is on verified, not reported.
00:15:26.960 --> 00:15:29.279
Not assumed, but confirmed.
00:15:29.919 --> 00:15:36.399
I had someone tell me one time there's basically two types of information, confirmed and unconfirmed.
00:15:36.639 --> 00:15:40.080
If the information's confirmed, then it's safe to act.
00:15:40.240 --> 00:15:42.960
If it's unconfirmed, then you're assuming a risk.
00:15:43.039 --> 00:15:56.399
So depending on how you've obtained the information and how sure you are the information is accurate, you're assuming a certain level of risk by taking that as unconfirmed information and making a decision based on that information.
00:15:57.039 --> 00:16:08.000
Because in a fast-moving situation, the most dangerous thing in the room is a confident action that's built and taken on unverified or unconfirmed information.
00:16:08.240 --> 00:16:14.320
So before I apply a framework here, I want to name it because it's going to be foundational to everything we're going to talk about.
00:16:14.480 --> 00:16:26.320
And especially if you're new to this show or you are an occasional listener, I want to give you a 30-second background version to discuss and point out exactly what we're talking about here.
00:16:26.559 --> 00:16:35.600
What I've come up with I call business lifelines, and these are the critical functions that every organization depends on to keep themselves operating under stress.
00:16:35.840 --> 00:16:46.080
These aren't theoretical categories, they came out of the same discipline that asks when an organization is under serious pressure, what's going to break first and in what order, and how do we know that?
00:16:46.879 --> 00:16:51.519
One way to think of this is as the load-bearing walls of your operation.
00:16:51.759 --> 00:16:55.039
That's really what these business lifelines are.
00:16:55.840 --> 00:17:04.400
When a lifeline degrades or starts to break down, one of those walls starts to give way and it doesn't stay contained, it still can't support its weight.
00:17:04.640 --> 00:17:08.000
It puts stress on itself and everything that's connected to it.
00:17:08.319 --> 00:17:18.720
Business Lifeline gives us a common language for identifying where organizational capacities are and where they're actual failing, as opposed to where the dashboard says that it's failing.
00:17:18.880 --> 00:17:30.319
If you can monitor the lifelines and monitoring them accurately, and compare that to what your dashboard's saying, it's a much more clear picture of what's actually going on so it can be dealt with properly.
00:17:30.720 --> 00:17:33.519
In this episode, there are two lifelines that are relevant.
00:17:33.759 --> 00:17:35.599
Keep that in mind as we go forward.
00:17:35.759 --> 00:17:46.480
The next thing we need to discuss here is the information hygiene concept, and this becomes a central concept in this incident and why it matters more than most organizations realize.
00:17:47.039 --> 00:17:52.160
Information hygiene isn't about whether you have data, both of these organizations had data.
00:17:52.319 --> 00:17:59.839
It's about whether the information moving through your system has been verified, vetted, and is confirmed as accurate before someone takes action.
00:18:00.319 --> 00:18:18.480
The gap in both of these cases was not at the decision points, it was in the handoffs, or the spaces in between the steps where the information moved from one part of the system into the next, and where nobody checked on whether that information that arrived was actually true and accurate.
00:18:20.240 --> 00:18:26.079
In the case of Night Capital, if you remember, I said that there were ninety seven emails sent at eight oh one AM.
00:18:26.880 --> 00:18:35.759
These were automatically generated by the system, so the information existed and the pathway to act on it was taken in the form of an email.
00:18:36.079 --> 00:18:48.640
However, that pathway was broken, and no one verified that the deployment was complete across all eight servers before the market opened because those emails weren't identified as high priority.
00:18:48.799 --> 00:18:50.480
That's a pathway failure.
00:18:50.720 --> 00:18:54.640
The signal was present, but the verification step didn't exist.
00:18:54.880 --> 00:19:00.319
Neither did the notification that the information was there so the verification could take place.
00:19:00.640 --> 00:19:06.960
In the case of Target Canada, corrupted data entered the ERP at the foundation at the very beginning.
00:19:07.279 --> 00:19:23.440
The system processed that information as though it was accurate, so every automated decision downstream, and every manual correction that was attempted by the staff was built and based on information that had never been confirmed against physical reality.
00:19:24.000 --> 00:19:26.000
That is a foundational failure.
00:19:26.160 --> 00:19:32.240
The data was there again, but nobody had verified that the data was true and correct before taking action.
00:19:32.480 --> 00:19:36.799
These are different mechanisms, but they have the same missing step.
00:19:37.440 --> 00:19:44.559
The missing step is verification at the handoff point before the information becomes a basis for action.
00:19:44.880 --> 00:19:49.119
So let's look at this from the perspective of how it matters to your own organization.
00:19:49.519 --> 00:19:55.920
When information hygiene fails, it doesn't produce an information problem that stays in the information system.
00:19:56.319 --> 00:19:58.799
It actually produces a lifeline problem.
00:19:59.119 --> 00:20:12.799
In the case of Night Capital, operational execution, which is lifeline number three, was degraded because the corrective actions were based on a misdiagnosis of which server was actually causing the problem.
00:20:13.359 --> 00:20:17.440
The team was executing correctly against the wrong target.
00:20:17.839 --> 00:20:25.440
And in Target Canada, Lifeline one, or the supply chain and resource flow lifeline, had collapsed entirely.
00:20:26.000 --> 00:20:33.759
But the mechanism wasn't a supply chain problem, it was an information system feeding false instructions to the operational system.
00:20:34.559 --> 00:20:37.119
This happened from the moment that it went live.
00:20:38.000 --> 00:20:40.000
These two failures aren't parallel.
00:20:40.240 --> 00:20:44.000
One caused the other, had a direct impact on the other.
00:20:44.240 --> 00:20:49.599
The information hygiene failure created the conditions for the lifeline failure.
00:20:50.319 --> 00:21:06.000
This illustrates the direct link and the value and importance of clean information, valid information, because that's what's going to feed the status of the business lifelines to give you a clear picture of what the conditions actually are.
00:21:06.799 --> 00:21:16.720
Once the information layer was compromised in these cases, every operational decision downstream was built on that bad foundation that had already given way.
00:21:17.279 --> 00:21:26.160
Something else that I'd like to discuss and that we've mentioned in the previous episode and we'll mention again because it's key is what I call the delta of decay.
00:21:26.480 --> 00:21:30.720
This is something else that I'd like to discuss here is the delta of decay.
00:21:31.039 --> 00:21:43.759
In Target Canada, which is an extreme case, the corruption at launch was visible almost immediately, but the mechanism it exposes operates in most organizations on a much slower clock.
00:21:44.160 --> 00:21:56.559
Data that was accurate when someone entered it, but it hasn't been touched in a while, or processes that worked when they were written, but the team no longer exists, are both things that can linger for a long period of time.
00:21:56.960 --> 00:21:59.119
This expands that delta of decay.
00:21:59.200 --> 00:22:12.880
The delta of the Of decay is an indicator that illustrates the time between when something actually went wrong and when something is actually understood as an issue or a problem and addressed.
00:22:14.160 --> 00:22:20.640
Things like vendor relationships that functioned well until the person who managed them left and is no longer at the company.
00:22:21.039 --> 00:22:26.640
The gap between what the system reports and what actually is true widens gradually.
00:22:27.119 --> 00:22:30.079
But nobody flags it because nobody's watching for it.
00:22:30.319 --> 00:22:35.279
It shows up on the day when someone tries to take action on it under pressure and it doesn't work.
00:22:35.680 --> 00:22:38.160
That's an example of the delta of decay.
00:22:38.240 --> 00:22:44.000
It's the distance between your last verified reality and the one that you're operating on right now.
00:22:44.319 --> 00:22:52.400
In most organizations, nobody knows how wide that gap really is, and nobody even really knows to watch for it or to look for it.
00:22:52.559 --> 00:23:03.759
The organizations that have navigated comparable failures that have caught these kind of problems before it became a catastrophe didn't necessarily have better technology or better plans.
00:23:04.000 --> 00:23:07.119
They didn't have faster people or a bigger budget.
00:23:07.440 --> 00:23:12.640
But what they did have was a different type of relationship with their own data.
00:23:13.119 --> 00:23:21.279
At some point someone in those organizations asked the question is what this system is telling us actually true and accurate information?
00:23:21.599 --> 00:23:23.920
Not just does the system work.
00:23:24.160 --> 00:23:28.240
Systems can work perfectly even if they're operating on false information.
00:23:28.400 --> 00:23:31.200
But is what it's telling us actually accurate?
00:23:31.759 --> 00:23:39.680
That question is the differentiating variable between success and failure, not technology and not resources.
00:23:40.000 --> 00:23:42.559
Actual verification of information.
00:23:42.799 --> 00:23:52.160
That's resilience, the mechanism underneath both failures and why it shows up in organizations that look nothing like Knight and Capital or Target Canada.
00:23:53.119 --> 00:23:54.960
Now let's move into advantage.
00:23:55.119 --> 00:23:58.000
How can this be an advantage to you and your organization?
00:23:58.240 --> 00:24:01.279
This is where I'm going to give you one thing to take into this week.
00:24:01.440 --> 00:24:05.440
And it's not a framework or a model or a reading list or anything like that.
00:24:05.599 --> 00:24:07.200
This is just a simple question.
00:24:07.359 --> 00:24:13.920
And it's something that's applicable today and that neither of these organizations ever took the time to ask.
00:24:14.160 --> 00:24:16.079
So here's the one thing.
00:24:16.400 --> 00:24:20.480
Identify one critical operations process in your organization.
00:24:21.039 --> 00:24:33.680
Find one where the information moves from one system or one team or one stage to the next, and where a downstream decision gets made based on what that information is and where it arrives.
00:24:34.319 --> 00:24:36.000
Now ask one question.
00:24:36.240 --> 00:24:45.839
At each handoff point in that process, is there anyone that's verifying that the information that arrived is accurate before action is actually taken?
00:24:46.240 --> 00:24:47.839
Not does the system work.
00:24:48.000 --> 00:24:53.920
Both of these systems in Knight Capital and Target Canada had functioning systems that appeared to be working.
00:24:54.240 --> 00:25:00.000
Not did the process get followed, because both these organizations followed their processes.
00:25:00.720 --> 00:25:09.519
The question is, is the information that lands at each step verified, vetted, and confirmed as accurate before it becomes the basis for that action?
00:25:09.759 --> 00:25:13.599
That is the gap that neither of these organizations found in time.
00:25:14.240 --> 00:25:25.200
So pick one process, map the handoffs out, and ask where that verification lives or doesn't and figure out if it needs to be addressed.
00:25:25.920 --> 00:25:28.480
One more thing, and this matters practically.
00:25:29.359 --> 00:25:35.279
The verification mechanism that you find has to match the nature of where the risk lives.
00:25:36.559 --> 00:25:37.759
Let me say that again.
00:25:38.000 --> 00:25:44.960
The verification mechanism must match the nature of the location or where the risk lives.
00:25:45.519 --> 00:25:47.279
How do we find that out?
00:25:47.519 --> 00:25:53.440
Not through a tabletop exercise, a tabletop exercise would not have surfaced the night capital problem.
00:25:53.680 --> 00:25:59.359
The legacy code on that one server was invisible to any process level review.
00:25:59.519 --> 00:26:04.240
It only became visible when live transactions were actually executed.
00:26:04.480 --> 00:26:17.920
The verification mechanism that could have caught it was actually a controlled deployment test in a non production environment, a simulation that could have shown live marketing conditions before the market opened and tested it to see if it was working properly.
00:26:18.079 --> 00:26:25.759
In the example of Target Canada's data corruption, the data corruption would not have been visible in a process review either.
00:26:26.000 --> 00:26:29.119
It was actually in the foundation of the data itself.
00:26:29.279 --> 00:26:44.240
The verification mechanism that could have caught this would have been a live parallel run, something that populated the ERP with real inventory data and testing replenishment to see if it was working and chucked against the actual physical stock before going live.
00:26:45.440 --> 00:27:02.160
Things like stress testing, what we call in emergency management red teaming, having parallel runs or live fire type exercises, these things aren't bureaucratic compliance activities that we have to jump through the hoops or must do to check the box.
00:27:02.400 --> 00:27:11.680
These are mechanisms that can help find the gaps in the space between your steps before your live system does or before a crisis shows that gap.
00:27:11.920 --> 00:27:21.039
The question is whether the practice is designed to surface what's actually there or to confirm that the documented processes look correct on paper.
00:27:21.200 --> 00:27:22.960
That's the true difference.
00:27:23.440 --> 00:27:33.200
In emergency management training, we use a concept called the red slice to explain why the verification matters and why the absence of it is so dangerous.
00:27:33.440 --> 00:27:36.480
Think of your organization's information picture as a pie.
00:27:37.119 --> 00:27:39.839
The green slice is what you know that you know.
00:27:40.079 --> 00:27:41.599
This is confirmed information.
00:27:41.680 --> 00:27:44.720
It's been verified, vetted, and it's actionable.
00:27:44.960 --> 00:27:47.680
You can build your decisions on the green slice.
00:27:48.880 --> 00:27:52.160
The yellow slice is what you know that you don't know.
00:27:52.799 --> 00:27:58.400
You've identified a gap or you have a question, there's something that's still unconfirmed, but you're aware of it.
00:27:58.799 --> 00:28:01.839
This yellow slice is manageable because you can see it.
00:28:02.000 --> 00:28:06.720
You can vet it and address it and resolve it before it causes damage downstream.
00:28:08.319 --> 00:28:10.000
Then there's the red slice.
00:28:10.240 --> 00:28:13.279
This is what you don't know that you don't know.
00:28:13.599 --> 00:28:16.640
This is the most dangerous ground in any organization.
00:28:17.039 --> 00:28:19.920
You're not looking for it because you don't know it exists.
00:28:20.319 --> 00:28:29.839
Decisions get made and actions get taken, systems are activated, all without any awareness that a critical piece of the picture may be missing entirely.
00:28:30.400 --> 00:28:37.279
In the case of Night Capital, their red slice was a single server that ran on a legacy code that nobody knew was still active.
00:28:37.519 --> 00:28:47.519
In the case of Target Canada, their red slice was an entire ERP foundation that was built on data that had never been confirmed against physical reality.
00:28:48.000 --> 00:28:56.880
So the goal, both in emergency management and in your organization, is to keep that red slice as small as possible at all time.
00:28:57.279 --> 00:29:04.079
Stress testing, parallel runs, and red teamings are the mechanisms that pull things out of the red and into the yellow.
00:29:04.160 --> 00:29:08.400
They identify these things that you don't know that you don't know.
00:29:08.799 --> 00:29:15.680
For you can see them now, you can address them, and you can take action to resolve them before they go live or before the crisis hits.
00:29:15.839 --> 00:29:19.759
That's information hygiene and information management in operational terms.
00:29:19.920 --> 00:29:24.480
And that's the discipline that neither of these organizations had in place when it mattered the most.
00:29:24.640 --> 00:29:26.640
So what happens if you do find a gap?
00:29:26.799 --> 00:29:32.559
If you find a gap and there's a real chance that you will, here's how to think about what to do with it.
00:29:33.200 --> 00:29:35.599
First thing to do is recognize the gap.
00:29:35.839 --> 00:29:39.759
Name that information and operational reality that have separated.
00:29:40.000 --> 00:29:45.440
Don't minimize it and don't escalate it before you understand it, but recognize it for what it is.
00:29:47.119 --> 00:29:49.759
Then identify which system is the source.
00:29:50.160 --> 00:29:51.839
Is this a pathway failure?
00:29:52.079 --> 00:29:55.920
The information exists, but it can't reach the people who need it.
00:29:56.079 --> 00:29:59.519
Or is it a foundation failure where the data itself is wrong?
00:30:00.799 --> 00:30:05.279
If you've recognized it and you've identified the problem, now stabilize it.
00:30:05.440 --> 00:30:14.400
Before you make an operational decision downstream, establish what's actually true and verify your actions before you take action, not after.
00:30:15.920 --> 00:30:17.599
Then evaluate what happened.
00:30:17.839 --> 00:30:24.720
Was the corrective action that you took or were about to take based on the real failure or a misdiagnosis of it?
00:30:25.039 --> 00:30:27.039
Night Capital's team failed here.
00:30:27.200 --> 00:30:28.640
Make sure that you don't.
00:30:28.960 --> 00:30:46.640
So if this episode surfaced a question that you don't have a quick answer to, about your data accuracy or about whether your dashboards are reflecting operational realities, or about whether a corrective action that your team runs regularly is actually solving the right problem, that's the right discomfort.
00:30:46.799 --> 00:30:49.279
It means you're looking at the right gap.
00:30:49.680 --> 00:30:54.880
The signal integrity assessment is a structured place to take that question.
00:30:55.039 --> 00:31:05.759
It's not an audit, it's a diagnostic, and it's built around whether your critical information pathways are actually functioning that the way that your organization assumes that they are.
00:31:06.000 --> 00:31:09.359
You can find the details at Southwindplanning dot com.
00:31:09.519 --> 00:31:15.440
That's advantageous one question, one conversation before the end of this week.
00:31:15.759 --> 00:31:17.839
Now let's bring this episode home.
00:31:18.160 --> 00:31:23.920
We had two organizations, two industries, both of them followed their processes, but both failed.
00:31:24.160 --> 00:31:28.079
Not at the decision points, but at the spaces in between those points.
00:31:28.400 --> 00:31:36.640
Where information was moved from one part of the system to the next, and where nobody verified that what arrived was accurate before it was acted on.
00:31:37.359 --> 00:31:46.799
The corrective actions made it worse because they were built on a picture that hadn't been confirmed, and the differentiating variable isn't better technology or faster response.
00:31:47.039 --> 00:31:52.799
It's verification of the information at the handoff before the information becomes basis for your actions.
00:31:53.039 --> 00:32:02.559
And the only way to find that gap is before your live system does, and to do that, you need to find a design, a practice that's built to surface what's actually there.
00:32:02.880 --> 00:32:13.839
In the next episode, we'll talk about another critical failure or critical incident that occurred that we can hopefully avoid in the future by taking the proper action.
00:32:14.319 --> 00:32:20.240
The blue to gray deep dive is paired to this episode and it goes deeper on both of these cases.
00:32:20.480 --> 00:32:31.279
The emergency management doctrine, how it's connected, the information hygiene framework and how that's connected, and what the red slice looks like when you map it against your own organization.
00:32:31.839 --> 00:32:37.839
If you're already a subscriber to the Blue to Gray newsletter, then it's already in your mailbox.
00:32:38.079 --> 00:32:44.240
If you're not a subscriber, you can subscribe for free at blue to gray.beehive.com.
00:32:45.519 --> 00:32:50.559
What the hell happened is produced by Southwind Planning Solutions LLC.
00:32:51.119 --> 00:32:58.000
If this episode was useful, the Blue to Gray newsletter goes deeper into this and many other topics each week.
00:32:58.240 --> 00:33:01.200
You can subscribe for free on Beehive.
00:33:01.359 --> 00:33:03.200
The link is in the show notes.
00:33:03.839 --> 00:33:08.319
If you're sitting on a subject that you've not verified anymore that you'd like to do.
00:33:09.200 --> 00:33:13.359
Or maybe there are other challenges where reality is interrupted in the planet.
00:33:14.240 --> 00:33:16.480
The starting point for that in the show notes.
00:33:18.319 --> 00:33:23.839
You can find us on Apple Podcasts, Spotify, or anywhere else to listen to the podcast.
00:33:24.720 --> 00:33:32.480
You can also find me on LinkedIn and on our website at www.selfwindplanning.com.
00:33:33.440 --> 00:33:34.720
I'm Michael Cracker.
00:33:35.359 --> 00:33:38.000
Thanks for listening, and I'll see you next time.
00:00:03.680 --> 00:00:06.000
Sometimes it's a policy shift.
00:00:06.240 --> 00:00:08.480
Sometimes it's a system that fails.
00:00:08.720 --> 00:00:10.960
Or it could be that a market moves.
00:00:11.519 --> 00:00:14.080
Often leaders find out too late.
00:00:14.320 --> 00:00:19.679
The signal was there, but nobody turned it into something they could act on before the moment passed.
00:00:20.320 --> 00:00:25.039
What the hell happened is a weekly briefing for leaders who would rather be ready.
00:00:25.440 --> 00:00:32.399
I'm Michael Crackett, and I've spent over 30 years working in situations where planning gets interrupted by reality.
00:00:32.799 --> 00:00:38.399
This program looks at not just what the hell happened, but also what can be done about it.
00:00:40.159 --> 00:00:42.960
Welcome back, everybody, to another episode of the podcast.
00:00:43.119 --> 00:00:49.679
I'm truly excited to have you here with me again, and I appreciate your interest and taking the time to listen to these episodes.
00:00:49.759 --> 00:01:09.760
It means a lot to me, and I really appreciate any comments that you'd like to leave, and it'll help drive us to future episodes, any topics that you may be interested in hearing about, and also give you a chance to leave a review, let other folks know what you think, and give me a thumbs up if you like what you're hearing, or let me know some things that I can do to improve.
00:01:09.840 --> 00:01:11.120
It all helps us get better.
00:01:11.200 --> 00:01:13.040
So anyway, I'm glad that you're here.
00:01:13.200 --> 00:01:14.560
Let's jump into this episode.
00:01:14.719 --> 00:01:19.760
The title for this episode is When Crisis Don't Wait for Their Turn.
00:01:20.640 --> 00:01:23.040
What exactly does that mean?
00:01:23.439 --> 00:01:31.359
What it means is a lot of plans, most plans in fact, are written around focusing on one crisis at a time.
00:01:31.439 --> 00:01:38.640
When that crisis turns up, they activate the plan or the plan goes into action and the crisis is dealt with.
00:01:39.040 --> 00:01:52.079
But what a lot of plans, in fact, most plans don't address is how to deal with a situation that comes up, you're working on it in the plan, and all of a sudden something else goes wrong, and now you're facing two crises at the same time.
00:01:52.159 --> 00:02:01.040
They may or may not be related and they may or may not be connected, but they will have an impact on each other and definitely have an impact on your organization.
00:02:01.120 --> 00:02:03.439
So that's what we're going to talk about today.
00:02:03.840 --> 00:02:07.920
So with each episode, I try to open with a single moment.
00:02:08.159 --> 00:02:10.639
A blue to gray illusion is what I call it.
00:02:10.800 --> 00:02:19.599
It's a blue sky moment where the status says that everything was fine in the organization, but in reality the skies were already turning gray.
00:02:19.759 --> 00:02:23.360
Things were getting dark, and things were happening behind the scenes.
00:02:23.599 --> 00:02:26.719
So here's this episode's blue to gray illusion.
00:02:27.919 --> 00:02:32.080
The leadership dashboards showed the inventory was moving normally.
00:02:32.560 --> 00:02:40.400
The distribution centers were packed with the product, but it couldn't be shelved, and the store shelves were actually empty.
00:02:41.280 --> 00:02:46.719
The system reported one reality, but the warehouse showed an obvious different reality.
00:02:47.680 --> 00:02:53.039
The status and all the indicators said that everything was okay, but the reality was much different.
00:02:53.520 --> 00:02:55.360
So what the hell happened?
00:02:55.680 --> 00:02:58.000
Well that's what we're going to talk about today.
00:02:58.240 --> 00:02:59.840
First let's look at readiness.
00:02:59.919 --> 00:03:02.319
Here's the readiness portion of this episode.
00:03:02.479 --> 00:03:05.439
Readiness is where we're going to look at what actually happened.
00:03:05.680 --> 00:03:13.680
We'll talk about the sequence of the events and the decisions that were made, and then we'll look at some of the moment that things went wrong.
00:03:14.159 --> 00:03:19.759
Today in readiness we're going to look at two organizations and three different incidents actually.
00:03:20.000 --> 00:03:25.840
They affect different industries, they happened during different decades, and they impacted different people.
00:03:26.240 --> 00:03:29.840
But they made the same catastrophic mistake in a similar way.
00:03:30.000 --> 00:03:31.680
So let's start first with a question.
00:03:31.919 --> 00:03:34.159
You have an organization and they hit a problem.
00:03:34.400 --> 00:03:38.400
A competent team makes the appropriate response according to the plan.
00:03:38.479 --> 00:03:42.719
You have confident people and they work fast to try and solve the problem.
00:03:43.199 --> 00:03:48.240
They run all of their standard corrective procedures and the procedures only make it worse.
00:03:48.639 --> 00:03:58.560
This doesn't happen because the procedures were slow or the response was delayed, and it didn't happen because the responders or the folks managing the situation didn't care.
00:03:59.039 --> 00:04:11.919
They were operating on a picture that was based on information that they thought was actually true, when in fact the information that they were operating on wasn't the picture that was actually true in reality.
00:04:12.400 --> 00:04:17.360
So what happens when the correction is based on the wrong diagnosis?
00:04:17.600 --> 00:04:23.519
A diagnosis that's based on information that's incorrect, inaccurate, or not timely.
00:04:24.319 --> 00:04:26.639
Let's look at these two case studies.
00:04:27.199 --> 00:04:29.360
The first one is Knight Capital.
00:04:29.839 --> 00:04:35.839
It's an investment group, and the problem happened on October first, 2012.
00:04:36.480 --> 00:04:44.959
On this date, Knight Capital deployed a new trading software and it went out to eight servers, but one server got missed.
00:04:45.120 --> 00:04:56.959
It had a legacy code on it called PowerPeg, and that stayed active on that one individual server while the new trading software deployed as it was supposed to to the rest of the servers.
00:04:58.240 --> 00:05:04.399
At eight oh one AM there were ninety-seven emails that were generated that showed an error.
00:05:04.560 --> 00:05:08.240
They were automated emails, and they identified the problem.
00:05:08.800 --> 00:05:17.199
The problem here was, though, that they were not configured as high priority alerts, and so nobody actually read the emails before the market opened.
00:05:17.920 --> 00:05:20.319
The market opened at 9 30 AM.
00:05:20.800 --> 00:05:26.879
The unpatched server began to execute millions of erroneous trades automatically.
00:05:27.279 --> 00:05:34.079
The team seized the problem and they ran their bad deployment response as it was outlined in their plan.
00:05:35.040 --> 00:05:39.920
It called for them to uninstall the new code from the functioning servers, so that's what they did.
00:05:40.560 --> 00:05:45.279
And then that action reactivated the legacy code across all eight servers.
00:05:46.560 --> 00:05:59.920
What happened then was what they thought was the fix actually served to multiply the damage because the old server and the old code was now activated, running on the wrong information.
00:06:00.160 --> 00:06:07.920
The result was in the time of forty five minutes, over four hundred million dollars in losses were accrued by the company.
00:06:08.079 --> 00:06:11.680
The deployment wasn't the error that killed this company.
00:06:11.920 --> 00:06:14.240
That wasn't the critical impact.
00:06:14.399 --> 00:06:22.399
The critical impact was a corrective action that was built and deployed on a misdiagnosis of what the actual problem was.
00:06:23.120 --> 00:06:30.879
The actual problem was that the trading software had not deployed on all of the servers, it had missed one.
00:06:32.079 --> 00:06:39.360
But that was not actually noticed until after what they thought was wrong was addressed in the wrong way.
00:06:40.240 --> 00:06:45.360
So the team was competent, the team did what they were supposed to, but the picture was wrong.
00:06:45.600 --> 00:06:47.040
Here's another case.
00:06:47.279 --> 00:06:48.800
This one involves Target.
00:06:48.879 --> 00:06:51.279
It's actually Target in Canada.
00:06:51.519 --> 00:06:53.360
They're called Target Canada.
00:06:53.759 --> 00:07:03.360
They acquired Zellers, which is another department type store in Canada, and they acquired the leases for one point eight billion dollars.
00:07:03.519 --> 00:07:12.720
And they committed to opening one hundred and twenty four stores simultaneously in Canada because the real estate costs demanded that they do so.
00:07:12.879 --> 00:07:14.319
So what happened?
00:07:14.959 --> 00:07:19.120
They stood up a new ERP system and it got populated manually.
00:07:19.360 --> 00:07:24.480
This happened under an extreme time pressure due to the deadline to get everything open.
00:07:24.959 --> 00:07:29.759
At launch, up to seventy percent of the data was found to be corrupted.
00:07:29.920 --> 00:07:35.680
There were wrong dimensions, wrong barcodes, wrong case pack counts.
00:07:36.000 --> 00:07:44.639
That happened because it had to be populated manually due to the time restraints and so some of the errors were made by the manual entry of the data.
00:07:45.120 --> 00:07:53.360
The automated replenishment system didn't know that that data was not accurate, and so it treated it as it was accurate.
00:07:54.319 --> 00:08:01.680
What happened next was the distribution centers overflowed with the wrong product while the store shelves sat empty.
00:08:02.000 --> 00:08:07.600
Everything that they needed to order was automated and sent to the distribution centers.
00:08:07.759 --> 00:08:14.319
However, once it arrived at the distribution centers, it never made it out of there and actually hit the store shelves.
00:08:14.800 --> 00:08:24.160
So what happened next was the leadership decided that they needed to disable the auto replenishment software as outlined in their plan because something was obviously wrong.
00:08:24.319 --> 00:08:31.199
And they decided that they would go manual, try to get the supplies out to the stores and make the deliveries.
00:08:31.439 --> 00:08:38.960
What that resulted in was the manual corrections were now feeding new bad data back into the same corrupted system.
00:08:39.200 --> 00:08:48.240
So they were still using their distribution system, but it was still relying on the bad data that had come from the improper system that was originally activated.
00:08:48.399 --> 00:08:52.480
They still had the dashboard that showed one reality, everything was fine.
00:08:52.720 --> 00:08:56.240
But the people in the warehouse were reporting something entirely different.
00:08:56.399 --> 00:09:00.480
The warehouses were overstocked, nothing on the shelf in the store.
00:09:01.200 --> 00:09:08.000
According to the filing on january fifteenth, twenty fifteen, one hundred and thirty three of the stores were closed.
00:09:08.159 --> 00:09:17.600
This resulted in seventeen thousand six hundred people who were now out of work and over five billion dollars in pre-tax losses for the company.
00:09:18.159 --> 00:09:27.039
Every corrective action, both automated and manual, was operating on a foundation that was wrong from the start.
00:09:29.360 --> 00:09:44.159
One other highly publicized incident, I just watched a movie on this as a matter of fact recently, and it really shed a lot of light on it, really a good movie, but it involved Deepwater Horizon and the disaster that resulted from that oil spill.
00:09:44.559 --> 00:09:52.960
This was a third organization, an entirely different industry, this time the oil and gas industry, not technology and not retail.
00:09:53.200 --> 00:10:01.600
In this incident, as you recall, the crew of the Deepwater Horizon misread a negative pressure test on the offshore oil rig.
00:10:02.399 --> 00:10:09.840
They interpreted the results through the lens of their last coherent mental model, and that assumed that the well was fine.
00:10:10.080 --> 00:10:13.919
The test made sense and they thought that they could proceed without any problem.
00:10:14.480 --> 00:10:17.200
In reality, the well wasn't sealed.
00:10:17.519 --> 00:10:23.759
The CSB documented this as a diagnostic failure interacting with a physical safety system failure.
00:10:24.080 --> 00:10:27.840
The information was actually there, and the information existent.
00:10:28.080 --> 00:10:31.519
The problem was the process was not followed.
00:10:32.000 --> 00:10:39.679
But what arrived at each decision point had not been verified against an independent read before the action was taken.
00:10:41.759 --> 00:10:44.320
You can find out more about this information.
00:10:44.480 --> 00:10:58.159
There's a lot of public information, a lot of movies and other research, after action reviews, a lot of things on the internet that are available that go into the details, but we're focused mainly here on the processes.
00:10:58.399 --> 00:11:12.480
In fact, I have a colleague that I work closely with who is a direct responder to this disaster in Deepwater Horizon, worked on the Gulf Coast at the time and was responsible for a portion of the response and the recovery after it happened.
00:11:12.639 --> 00:11:19.759
I hope to have him on and interview him about this and what the processes that failed as far as the response went.
00:11:20.000 --> 00:11:29.039
There were other issues that were involved there too, so hopefully I can get him on the podcast on a future episode and talk a little bit more in detail about that.
00:11:29.360 --> 00:11:32.720
But we'll also look at that more later in resilience.
00:11:32.799 --> 00:11:38.559
We're going to look at exactly where the gap lives and why it's so hard to see until it's too late.
00:11:39.120 --> 00:11:45.600
So both of these organizations had smart and capable people, and they were running hard at a real problem.
00:11:46.159 --> 00:11:48.000
But in the end both had made it worse.
00:11:48.159 --> 00:11:51.440
The question isn't whether they were competent because they were.
00:11:52.080 --> 00:12:00.320
The question is actually what information were they actually using to operate on and to base their response?
00:12:00.960 --> 00:12:02.879
Sit on that for a minute.
00:12:04.000 --> 00:12:05.679
That's the readiness picture.
00:12:06.000 --> 00:12:10.639
That's what happened and what it looked like from inside both of the organizations.
00:12:11.840 --> 00:12:13.759
Now let's move into resilience.
00:12:14.000 --> 00:12:21.360
Resilience is where we're going to stop looking at the events themselves and start looking at the mechanisms that made the events occur.
00:12:22.240 --> 00:12:27.360
Because these two failures aren't parallel stories that just happen to resemble each other.
00:12:27.600 --> 00:12:34.960
They share a similar architecture, and once you see it, you can't really unsee it inside your own organization.
00:12:36.639 --> 00:12:48.559
In both cases, both Night Capital and Target Canada, the information systems and the operational systems were separated completely before any corrective action was taken.
00:12:48.720 --> 00:12:55.360
This didn't happen gradually over time, and it wasn't because of negligence that had built up across months.
00:12:55.600 --> 00:13:02.720
This was a structural separation, and it was present before the first decision in the crisis was even made.
00:13:03.039 --> 00:13:10.799
And then every decision after that that was made downstream was built or based on a corrupted picture based on bad information.
00:13:11.279 --> 00:13:16.000
In the case of the Knight team, they had corrected for the wrong failure.
00:13:16.320 --> 00:13:21.600
Targets leadership had fed new bad data into an already broken system.
00:13:22.159 --> 00:13:26.080
These were competent people working in a wrong reality.
00:13:26.399 --> 00:13:28.480
And that's not a technology problem.
00:13:28.639 --> 00:13:30.480
That's not even a training problem.
00:13:30.720 --> 00:13:37.279
What that actually shows is an architectural problem, something in the structure that just wasn't working right.
00:13:37.519 --> 00:13:42.080
I spent thirty years in emergency management, and that teaches you a lot.
00:13:42.240 --> 00:13:46.080
It teaches you something about how systems can fail under pressure.
00:13:46.320 --> 00:13:51.279
And it rarely results from steps that were taken in the process that broke.
00:13:51.759 --> 00:13:57.679
People are always following the process, following the steps, taking the steps that are needed and doing their best.
00:13:57.919 --> 00:14:02.159
The steps are documented and they're trained and they're exercised.
00:14:02.559 --> 00:14:06.720
But what breaks is what happens in the space between those steps.
00:14:07.039 --> 00:14:14.799
This is where the information is moving from one part of the system to the next, and it's critical to making things work as they should.
00:14:15.279 --> 00:14:19.120
In both of these cases, that's exactly where the failures lived.
00:14:19.360 --> 00:14:27.120
For Night Capital, the team followed a legitimate, bad deployment response, as they were supposed to and as was outlined in the plan.
00:14:27.519 --> 00:14:39.440
The steps were correct, but the information that arrived at each step, which was about the servers that were affected, and that's what the root cause was, had not been verified before it was acted on.
00:14:40.240 --> 00:14:49.919
In Target Canada, the leadership followed the legitimate crisis response for the incident they were dealing with, the problem with the stock shelves in the stores.
00:14:50.159 --> 00:14:54.559
They identified the problem and disabled the automation system.
00:14:55.360 --> 00:15:01.759
They went on manual and the steps made sense for what they were dealing with according to what they thought was occurring.
00:15:02.399 --> 00:15:07.120
The problem was that the information that those steps were built on was corrupted from the start.
00:15:07.360 --> 00:15:13.840
In emergency management, one of the doctrine standards is real-time, accurate situational awareness.
00:15:14.080 --> 00:15:22.879
Not just having the information, but maintaining a verified, current picture of what is actually true as the conditions are always changing.
00:15:23.200 --> 00:15:26.639
The emphasis here is on verified, not reported.
00:15:26.960 --> 00:15:29.279
Not assumed, but confirmed.
00:15:29.919 --> 00:15:36.399
I had someone tell me one time there's basically two types of information, confirmed and unconfirmed.
00:15:36.639 --> 00:15:40.080
If the information's confirmed, then it's safe to act.
00:15:40.240 --> 00:15:42.960
If it's unconfirmed, then you're assuming a risk.
00:15:43.039 --> 00:15:56.399
So depending on how you've obtained the information and how sure you are the information is accurate, you're assuming a certain level of risk by taking that as unconfirmed information and making a decision based on that information.
00:15:57.039 --> 00:16:08.000
Because in a fast-moving situation, the most dangerous thing in the room is a confident action that's built and taken on unverified or unconfirmed information.
00:16:08.240 --> 00:16:14.320
So before I apply a framework here, I want to name it because it's going to be foundational to everything we're going to talk about.
00:16:14.480 --> 00:16:26.320
And especially if you're new to this show or you are an occasional listener, I want to give you a 30-second background version to discuss and point out exactly what we're talking about here.
00:16:26.559 --> 00:16:35.600
What I've come up with I call business lifelines, and these are the critical functions that every organization depends on to keep themselves operating under stress.
00:16:35.840 --> 00:16:46.080
These aren't theoretical categories, they came out of the same discipline that asks when an organization is under serious pressure, what's going to break first and in what order, and how do we know that?
00:16:46.879 --> 00:16:51.519
One way to think of this is as the load-bearing walls of your operation.
00:16:51.759 --> 00:16:55.039
That's really what these business lifelines are.
00:16:55.840 --> 00:17:04.400
When a lifeline degrades or starts to break down, one of those walls starts to give way and it doesn't stay contained, it still can't support its weight.
00:17:04.640 --> 00:17:08.000
It puts stress on itself and everything that's connected to it.
00:17:08.319 --> 00:17:18.720
Business Lifeline gives us a common language for identifying where organizational capacities are and where they're actual failing, as opposed to where the dashboard says that it's failing.
00:17:18.880 --> 00:17:30.319
If you can monitor the lifelines and monitoring them accurately, and compare that to what your dashboard's saying, it's a much more clear picture of what's actually going on so it can be dealt with properly.
00:17:30.720 --> 00:17:33.519
In this episode, there are two lifelines that are relevant.
00:17:33.759 --> 00:17:35.599
Keep that in mind as we go forward.
00:17:35.759 --> 00:17:46.480
The next thing we need to discuss here is the information hygiene concept, and this becomes a central concept in this incident and why it matters more than most organizations realize.
00:17:47.039 --> 00:17:52.160
Information hygiene isn't about whether you have data, both of these organizations had data.
00:17:52.319 --> 00:17:59.839
It's about whether the information moving through your system has been verified, vetted, and is confirmed as accurate before someone takes action.
00:18:00.319 --> 00:18:18.480
The gap in both of these cases was not at the decision points, it was in the handoffs, or the spaces in between the steps where the information moved from one part of the system into the next, and where nobody checked on whether that information that arrived was actually true and accurate.
00:18:20.240 --> 00:18:26.079
In the case of Night Capital, if you remember, I said that there were ninety seven emails sent at eight oh one AM.
00:18:26.880 --> 00:18:35.759
These were automatically generated by the system, so the information existed and the pathway to act on it was taken in the form of an email.
00:18:36.079 --> 00:18:48.640
However, that pathway was broken, and no one verified that the deployment was complete across all eight servers before the market opened because those emails weren't identified as high priority.
00:18:48.799 --> 00:18:50.480
That's a pathway failure.
00:18:50.720 --> 00:18:54.640
The signal was present, but the verification step didn't exist.
00:18:54.880 --> 00:19:00.319
Neither did the notification that the information was there so the verification could take place.
00:19:00.640 --> 00:19:06.960
In the case of Target Canada, corrupted data entered the ERP at the foundation at the very beginning.
00:19:07.279 --> 00:19:23.440
The system processed that information as though it was accurate, so every automated decision downstream, and every manual correction that was attempted by the staff was built and based on information that had never been confirmed against physical reality.
00:19:24.000 --> 00:19:26.000
That is a foundational failure.
00:19:26.160 --> 00:19:32.240
The data was there again, but nobody had verified that the data was true and correct before taking action.
00:19:32.480 --> 00:19:36.799
These are different mechanisms, but they have the same missing step.
00:19:37.440 --> 00:19:44.559
The missing step is verification at the handoff point before the information becomes a basis for action.
00:19:44.880 --> 00:19:49.119
So let's look at this from the perspective of how it matters to your own organization.
00:19:49.519 --> 00:19:55.920
When information hygiene fails, it doesn't produce an information problem that stays in the information system.
00:19:56.319 --> 00:19:58.799
It actually produces a lifeline problem.
00:19:59.119 --> 00:20:12.799
In the case of Night Capital, operational execution, which is lifeline number three, was degraded because the corrective actions were based on a misdiagnosis of which server was actually causing the problem.
00:20:13.359 --> 00:20:17.440
The team was executing correctly against the wrong target.
00:20:17.839 --> 00:20:25.440
And in Target Canada, Lifeline one, or the supply chain and resource flow lifeline, had collapsed entirely.
00:20:26.000 --> 00:20:33.759
But the mechanism wasn't a supply chain problem, it was an information system feeding false instructions to the operational system.
00:20:34.559 --> 00:20:37.119
This happened from the moment that it went live.
00:20:38.000 --> 00:20:40.000
These two failures aren't parallel.
00:20:40.240 --> 00:20:44.000
One caused the other, had a direct impact on the other.
00:20:44.240 --> 00:20:49.599
The information hygiene failure created the conditions for the lifeline failure.
00:20:50.319 --> 00:21:06.000
This illustrates the direct link and the value and importance of clean information, valid information, because that's what's going to feed the status of the business lifelines to give you a clear picture of what the conditions actually are.
00:21:06.799 --> 00:21:16.720
Once the information layer was compromised in these cases, every operational decision downstream was built on that bad foundation that had already given way.
00:21:17.279 --> 00:21:26.160
Something else that I'd like to discuss and that we've mentioned in the previous episode and we'll mention again because it's key is what I call the delta of decay.
00:21:26.480 --> 00:21:30.720
This is something else that I'd like to discuss here is the delta of decay.
00:21:31.039 --> 00:21:43.759
In Target Canada, which is an extreme case, the corruption at launch was visible almost immediately, but the mechanism it exposes operates in most organizations on a much slower clock.
00:21:44.160 --> 00:21:56.559
Data that was accurate when someone entered it, but it hasn't been touched in a while, or processes that worked when they were written, but the team no longer exists, are both things that can linger for a long period of time.
00:21:56.960 --> 00:21:59.119
This expands that delta of decay.
00:21:59.200 --> 00:22:12.880
The delta of the Of decay is an indicator that illustrates the time between when something actually went wrong and when something is actually understood as an issue or a problem and addressed.
00:22:14.160 --> 00:22:20.640
Things like vendor relationships that functioned well until the person who managed them left and is no longer at the company.
00:22:21.039 --> 00:22:26.640
The gap between what the system reports and what actually is true widens gradually.
00:22:27.119 --> 00:22:30.079
But nobody flags it because nobody's watching for it.
00:22:30.319 --> 00:22:35.279
It shows up on the day when someone tries to take action on it under pressure and it doesn't work.
00:22:35.680 --> 00:22:38.160
That's an example of the delta of decay.
00:22:38.240 --> 00:22:44.000
It's the distance between your last verified reality and the one that you're operating on right now.
00:22:44.319 --> 00:22:52.400
In most organizations, nobody knows how wide that gap really is, and nobody even really knows to watch for it or to look for it.
00:22:52.559 --> 00:23:03.759
The organizations that have navigated comparable failures that have caught these kind of problems before it became a catastrophe didn't necessarily have better technology or better plans.
00:23:04.000 --> 00:23:07.119
They didn't have faster people or a bigger budget.
00:23:07.440 --> 00:23:12.640
But what they did have was a different type of relationship with their own data.
00:23:13.119 --> 00:23:21.279
At some point someone in those organizations asked the question is what this system is telling us actually true and accurate information?
00:23:21.599 --> 00:23:23.920
Not just does the system work.
00:23:24.160 --> 00:23:28.240
Systems can work perfectly even if they're operating on false information.
00:23:28.400 --> 00:23:31.200
But is what it's telling us actually accurate?
00:23:31.759 --> 00:23:39.680
That question is the differentiating variable between success and failure, not technology and not resources.
00:23:40.000 --> 00:23:42.559
Actual verification of information.
00:23:42.799 --> 00:23:52.160
That's resilience, the mechanism underneath both failures and why it shows up in organizations that look nothing like Knight and Capital or Target Canada.
00:23:53.119 --> 00:23:54.960
Now let's move into advantage.
00:23:55.119 --> 00:23:58.000
How can this be an advantage to you and your organization?
00:23:58.240 --> 00:24:01.279
This is where I'm going to give you one thing to take into this week.
00:24:01.440 --> 00:24:05.440
And it's not a framework or a model or a reading list or anything like that.
00:24:05.599 --> 00:24:07.200
This is just a simple question.
00:24:07.359 --> 00:24:13.920
And it's something that's applicable today and that neither of these organizations ever took the time to ask.
00:24:14.160 --> 00:24:16.079
So here's the one thing.
00:24:16.400 --> 00:24:20.480
Identify one critical operations process in your organization.
00:24:21.039 --> 00:24:33.680
Find one where the information moves from one system or one team or one stage to the next, and where a downstream decision gets made based on what that information is and where it arrives.
00:24:34.319 --> 00:24:36.000
Now ask one question.
00:24:36.240 --> 00:24:45.839
At each handoff point in that process, is there anyone that's verifying that the information that arrived is accurate before action is actually taken?
00:24:46.240 --> 00:24:47.839
Not does the system work.
00:24:48.000 --> 00:24:53.920
Both of these systems in Knight Capital and Target Canada had functioning systems that appeared to be working.
00:24:54.240 --> 00:25:00.000
Not did the process get followed, because both these organizations followed their processes.
00:25:00.720 --> 00:25:09.519
The question is, is the information that lands at each step verified, vetted, and confirmed as accurate before it becomes the basis for that action?
00:25:09.759 --> 00:25:13.599
That is the gap that neither of these organizations found in time.
00:25:14.240 --> 00:25:25.200
So pick one process, map the handoffs out, and ask where that verification lives or doesn't and figure out if it needs to be addressed.
00:25:25.920 --> 00:25:28.480
One more thing, and this matters practically.
00:25:29.359 --> 00:25:35.279
The verification mechanism that you find has to match the nature of where the risk lives.
00:25:36.559 --> 00:25:37.759
Let me say that again.
00:25:38.000 --> 00:25:44.960
The verification mechanism must match the nature of the location or where the risk lives.
00:25:45.519 --> 00:25:47.279
How do we find that out?
00:25:47.519 --> 00:25:53.440
Not through a tabletop exercise, a tabletop exercise would not have surfaced the night capital problem.
00:25:53.680 --> 00:25:59.359
The legacy code on that one server was invisible to any process level review.
00:25:59.519 --> 00:26:04.240
It only became visible when live transactions were actually executed.
00:26:04.480 --> 00:26:17.920
The verification mechanism that could have caught it was actually a controlled deployment test in a non production environment, a simulation that could have shown live marketing conditions before the market opened and tested it to see if it was working properly.
00:26:18.079 --> 00:26:25.759
In the example of Target Canada's data corruption, the data corruption would not have been visible in a process review either.
00:26:26.000 --> 00:26:29.119
It was actually in the foundation of the data itself.
00:26:29.279 --> 00:26:44.240
The verification mechanism that could have caught this would have been a live parallel run, something that populated the ERP with real inventory data and testing replenishment to see if it was working and chucked against the actual physical stock before going live.
00:26:45.440 --> 00:27:02.160
Things like stress testing, what we call in emergency management red teaming, having parallel runs or live fire type exercises, these things aren't bureaucratic compliance activities that we have to jump through the hoops or must do to check the box.
00:27:02.400 --> 00:27:11.680
These are mechanisms that can help find the gaps in the space between your steps before your live system does or before a crisis shows that gap.
00:27:11.920 --> 00:27:21.039
The question is whether the practice is designed to surface what's actually there or to confirm that the documented processes look correct on paper.
00:27:21.200 --> 00:27:22.960
That's the true difference.
00:27:23.440 --> 00:27:33.200
In emergency management training, we use a concept called the red slice to explain why the verification matters and why the absence of it is so dangerous.
00:27:33.440 --> 00:27:36.480
Think of your organization's information picture as a pie.
00:27:37.119 --> 00:27:39.839
The green slice is what you know that you know.
00:27:40.079 --> 00:27:41.599
This is confirmed information.
00:27:41.680 --> 00:27:44.720
It's been verified, vetted, and it's actionable.
00:27:44.960 --> 00:27:47.680
You can build your decisions on the green slice.
00:27:48.880 --> 00:27:52.160
The yellow slice is what you know that you don't know.
00:27:52.799 --> 00:27:58.400
You've identified a gap or you have a question, there's something that's still unconfirmed, but you're aware of it.
00:27:58.799 --> 00:28:01.839
This yellow slice is manageable because you can see it.
00:28:02.000 --> 00:28:06.720
You can vet it and address it and resolve it before it causes damage downstream.
00:28:08.319 --> 00:28:10.000
Then there's the red slice.
00:28:10.240 --> 00:28:13.279
This is what you don't know that you don't know.
00:28:13.599 --> 00:28:16.640
This is the most dangerous ground in any organization.
00:28:17.039 --> 00:28:19.920
You're not looking for it because you don't know it exists.
00:28:20.319 --> 00:28:29.839
Decisions get made and actions get taken, systems are activated, all without any awareness that a critical piece of the picture may be missing entirely.
00:28:30.400 --> 00:28:37.279
In the case of Night Capital, their red slice was a single server that ran on a legacy code that nobody knew was still active.
00:28:37.519 --> 00:28:47.519
In the case of Target Canada, their red slice was an entire ERP foundation that was built on data that had never been confirmed against physical reality.
00:28:48.000 --> 00:28:56.880
So the goal, both in emergency management and in your organization, is to keep that red slice as small as possible at all time.
00:28:57.279 --> 00:29:04.079
Stress testing, parallel runs, and red teamings are the mechanisms that pull things out of the red and into the yellow.
00:29:04.160 --> 00:29:08.400
They identify these things that you don't know that you don't know.
00:29:08.799 --> 00:29:15.680
For you can see them now, you can address them, and you can take action to resolve them before they go live or before the crisis hits.
00:29:15.839 --> 00:29:19.759
That's information hygiene and information management in operational terms.
00:29:19.920 --> 00:29:24.480
And that's the discipline that neither of these organizations had in place when it mattered the most.
00:29:24.640 --> 00:29:26.640
So what happens if you do find a gap?
00:29:26.799 --> 00:29:32.559
If you find a gap and there's a real chance that you will, here's how to think about what to do with it.
00:29:33.200 --> 00:29:35.599
First thing to do is recognize the gap.
00:29:35.839 --> 00:29:39.759
Name that information and operational reality that have separated.
00:29:40.000 --> 00:29:45.440
Don't minimize it and don't escalate it before you understand it, but recognize it for what it is.
00:29:47.119 --> 00:29:49.759
Then identify which system is the source.
00:29:50.160 --> 00:29:51.839
Is this a pathway failure?
00:29:52.079 --> 00:29:55.920
The information exists, but it can't reach the people who need it.
00:29:56.079 --> 00:29:59.519
Or is it a foundation failure where the data itself is wrong?
00:30:00.799 --> 00:30:05.279
If you've recognized it and you've identified the problem, now stabilize it.
00:30:05.440 --> 00:30:14.400
Before you make an operational decision downstream, establish what's actually true and verify your actions before you take action, not after.
00:30:15.920 --> 00:30:17.599
Then evaluate what happened.
00:30:17.839 --> 00:30:24.720
Was the corrective action that you took or were about to take based on the real failure or a misdiagnosis of it?
00:30:25.039 --> 00:30:27.039
Night Capital's team failed here.
00:30:27.200 --> 00:30:28.640
Make sure that you don't.
00:30:28.960 --> 00:30:46.640
So if this episode surfaced a question that you don't have a quick answer to, about your data accuracy or about whether your dashboards are reflecting operational realities, or about whether a corrective action that your team runs regularly is actually solving the right problem, that's the right discomfort.
00:30:46.799 --> 00:30:49.279
It means you're looking at the right gap.
00:30:49.680 --> 00:30:54.880
The signal integrity assessment is a structured place to take that question.
00:30:55.039 --> 00:31:05.759
It's not an audit, it's a diagnostic, and it's built around whether your critical information pathways are actually functioning that the way that your organization assumes that they are.
00:31:06.000 --> 00:31:09.359
You can find the details at Southwindplanning dot com.
00:31:09.519 --> 00:31:15.440
That's advantageous one question, one conversation before the end of this week.
00:31:15.759 --> 00:31:17.839
Now let's bring this episode home.
00:31:18.160 --> 00:31:23.920
We had two organizations, two industries, both of them followed their processes, but both failed.
00:31:24.160 --> 00:31:28.079
Not at the decision points, but at the spaces in between those points.
00:31:28.400 --> 00:31:36.640
Where information was moved from one part of the system to the next, and where nobody verified that what arrived was accurate before it was acted on.
00:31:37.359 --> 00:31:46.799
The corrective actions made it worse because they were built on a picture that hadn't been confirmed, and the differentiating variable isn't better technology or faster response.
00:31:47.039 --> 00:31:52.799
It's verification of the information at the handoff before the information becomes basis for your actions.
00:31:53.039 --> 00:32:02.559
And the only way to find that gap is before your live system does, and to do that, you need to find a design, a practice that's built to surface what's actually there.
00:32:02.880 --> 00:32:13.839
In the next episode, we'll talk about another critical failure or critical incident that occurred that we can hopefully avoid in the future by taking the proper action.
00:32:14.319 --> 00:32:20.240
The blue to gray deep dive is paired to this episode and it goes deeper on both of these cases.
00:32:20.480 --> 00:32:31.279
The emergency management doctrine, how it's connected, the information hygiene framework and how that's connected, and what the red slice looks like when you map it against your own organization.
00:32:31.839 --> 00:32:37.839
If you're already a subscriber to the Blue to Gray newsletter, then it's already in your mailbox.
00:32:38.079 --> 00:32:44.240
If you're not a subscriber, you can subscribe for free at blue to gray.beehive.com.
00:32:45.519 --> 00:32:50.559
What the hell happened is produced by Southwind Planning Solutions LLC.
00:32:51.119 --> 00:32:58.000
If this episode was useful, the Blue to Gray newsletter goes deeper into this and many other topics each week.
00:32:58.240 --> 00:33:01.200
You can subscribe for free on Beehive.
00:33:01.359 --> 00:33:03.200
The link is in the show notes.
00:33:03.839 --> 00:33:08.319
If you're sitting on a subject that you've not verified anymore that you'd like to do.
00:33:09.200 --> 00:33:13.359
Or maybe there are other challenges where reality is interrupted in the planet.
00:33:14.240 --> 00:33:16.480
The starting point for that in the show notes.
00:33:18.319 --> 00:33:23.839
You can find us on Apple Podcasts, Spotify, or anywhere else to listen to the podcast.
00:33:24.720 --> 00:33:32.480
You can also find me on LinkedIn and on our website at www.selfwindplanning.com.
00:33:33.440 --> 00:33:34.720
I'm Michael Cracker.
00:33:35.359 --> 00:33:38.000
Thanks for listening, and I'll see you next time.
