文字稿 🔗
00:00:01
Speaker 1: You're with John Pullman on 702.
00:00:03
Speaker 2: Let's walk the talk.
00:00:07
Speaker 1: Well, we had a tricky start to putting the show together in the earlier part of the day. That's because I was sending emails from a service that isn't Outlook to colleagues who use Outlook, and we were not communicating with each other. That's because key Microsoft services, Outlook teams, Copilot, and some others as well, have been down since... My understanding is the local effect began around 8 p.m. last night. I don't know what kind of day you have had and what it's made you think about all the wonders of being wired up with the rest of the world, but the downside as well. You may want to call us on that, on 011-883-0702. But let's talk about what happened and, crucially, why it's taking so long to fix. My next guest is Arthur Goldstuck. He's the founder of Worldwide Works and the author of of the Hitchhiker's Guide to AI. Arthur, very good afternoon to you, and thanks so much for giving us your time. What happened at Microsoft? Do they know?
00:01:11
Speaker 2: Good afternoon, John. They're not saying whether they know. They claim they don't know, but they are pointing in the direction of overloaded servers, authentication systems, and network routing. So they said that they had identified a recent network configuration change as a possible contributor, but that wasn't the definitive cause, and they were still investigating. So I want us to look at the clues between the lines of what possibly went wrong. And essentially, it boils down to the fact that they had too few points of recovery and a few points of failure that were able to roll out across the entire network because so much was dependent on so little in their system. Because if three systems go down and you bring down the world's communications, then clearly you're not spending enough money. And that is a central issue as well because they just announced the $ 92 billion quarterly revenue and $ 36 billion quarterly net profit. When you report net profit and your systems go down literally in the same week, then you actually have to answer some more serious questions than just what went wrong.
00:02:35
Speaker 1: And, I mean, overload is not an event. I mean, you get overload of all sorts of things. There's a shortage of Uber vehicles, and the algorithm puts the price up outside Newlands on the day of a big cricket match for the sake of argument. But, I mean, surely you grow your systems as you sell your services.
00:02:55
Speaker 2: Well, exactly. What makes this outage so remarkable is its sheer scale and that it's spread across email, business communications, cybersecurity services even. And while Microsoft says it's found no evidence of a cyber attack, the real worrying aspect is that when they initially try to bring their servers back online, and if you were following it already last night, they were saying they bring the servers online one by one. People must be patient because they'll first focus on emergency services, for example. But as they brought them back online, the systems failed again. And as traffic returned, more and more traffic was coming back and basically overwhelmed them with almost exponential growth in traffic. So it was a massive technical failure, but then the recovery was part of the failure as well because they couldn't cope with the deluge of demand that came in with that recovery.
00:03:55
Speaker 1: Yeah, so your ambulance services work, but when people arrive at the hospital there, Not enough doctors and beds. Arthur, I want to ask you this. I mean, has this happened to Microsoft before? Because invariably, I would think there will be some sort of inquiry. And inquiries often say, well, did you get a substantial warning or even a small one before? What does the back catalog of Microsoft mishaps look like?
00:04:26
Speaker 2: What I can certainly point to is that it's happened to some of the big global service providers, cloud computing providers, like, for example, Amazon Web Services had a major outage which brought half the world's websites down. But, yes, Microsoft has had quite a few outages before, and some of them are fairly similar to the ones we're having now because they exposed the underlying infrastructure or the weakness of the underlying infrastructure. And you won't remember because even what's happened today won't be remembered a week from now as people catch up on all the email. But back in beginning of 2023, and I know because it was as my birthday was arriving, I won't give you the exact date. But there was global disruption to Outlook and Teams and Azure and various other services that lasted for about five hours. And that was the result of, guess what, a network configuration change, which is exactly the cause of this outage. And then in 2024, almost exactly two years ago, there was a widespread Outlook failure as well. And again, it was a change that they made in their system that and it's resulted in excessive requests to their system, which overwhelmed their back end. It's not the first time their back end has been overwhelmed as they made a change to their system. And then, of course.
00:06:00
Speaker 1: No, do finish the thought, please.
00:06:02
Speaker 2: So what's especially revealing this time around is that they tried restarting the servers, but then as they had to redirect traffic to infrastructure that was up and running, they brought down that infrastructure as well. So it's not like it hasn't happened before, but this time it really happened on steroids. Yeah.
00:06:25
Speaker 1: I wonder, you know, eventually things will come back to normal. I'm making that assumption, and perhaps I shouldn't. But might there be vulnerabilities? Because some people would say, oh, thank goodness I'm back online. But, you know, in your haste, you may not necessarily be working in an environment as secure as the one that preceded this particular collapse. Are there potential risks there?
00:06:49
Speaker 2: Yeah.
00:06:50
Speaker 1: Because I don't know to what extent people are dependent on the integrity of Microsoft systems, but if they, in their haste to bring things back, might have let some levels of that integrity slip, is there anything people ought to be careful of?
00:07:07
Speaker 2: One thing I wouldn't accuse him of is haste because it's taken him an extraordinary amount of time to fix this up. But yeah, people have to be aware that this can happen at any time. The fact that it's happened three times in four years tells you that it's an accident regularly waiting to happen again. And you've got to have seatbelts to survive an accident. So people have to have backups in place. They have to have fail-safes or alternatives in place. In our case, we had Gmail, which makes up a lot of the slack for us. We don't use it as a primary email. But when the main email went down, we were able to continue getting mails via Gmail. Other people use services like Mimecast or Proton and the like. There are many alternatives, but you cannot rely on a single provider, especially one that has a global system that can go down from a small floor at a head office.
00:08:07
Speaker 1: Thanks so much, Arthur Goldstack, founder of Worldwide Works, author of An excellent book actually called The Hitchhiker's Guide to AI. Now, if you want to tell us about how all this has affected you, please do get in touch on 072702-1702.