WEBVTT
00:00:00.542 --> 00:00:05.852
Greetings, fellow incident nerds and welcome to season two of the Void Podcast.
00:00:06.331 --> 00:00:10.801
The main new thing for this new season is we're now available in video.
00:00:11.221 --> 00:00:17.071
So if you're listening to this and prefer watching me make odd faces and not a lot, you can find us on YouTube.
00:00:17.071 --> 00:00:18.481
The link is in the show notes.
00:00:18.961 --> 00:00:21.722
The other new thing is we now have sponsors.
00:00:22.292 --> 00:00:27.961
Don't worry, these folks help make the podcast possible, but they don't have any say over who joins us or what we talk about.
00:00:28.172 --> 00:00:29.341
So fear not.
00:00:29.942 --> 00:00:32.515
This episode's sponsor is Uptime Labs.
00:00:32.966 --> 00:00:37.975
Uptime Labs is a pioneering platform specializing in immersive incident response training.
00:00:38.246 --> 00:00:46.405
Their solution helps technical teams build confidence and expertise through realistic simulations that mirror real world outages and security incidents.
00:00:46.975 --> 00:00:54.865
When most of investment these days in the incident space goes to technology and process, Uptime Labs focuses on sharpening the human element of incident response.
00:00:55.585 --> 00:00:55.915
All right.
00:00:56.636 --> 00:01:01.856
In this episode, we talked to Simon Newton, head of platforms at Canva.
00:01:02.456 --> 00:01:05.816
They just published their first public incident report.
00:01:05.876 --> 00:01:14.816
It's not their first incident, but it's the first time we get to delve into the details of what happened, and we really appreciate them sharing that with us and the world.
00:01:14.820 --> 00:01:17.006
So let's get into it and talk to Simon.
00:01:17.055 --> 00:01:20.355
Simon, thank you so much for joining me on The Void Podcast.
00:01:20.566 --> 00:01:25.756
Will you do our listeners a favor and introduce yourself and tell me where you work and what you do.
00:01:26.941 --> 00:01:27.271
Hi.
00:01:27.271 --> 00:01:28.501
I'm very happy to be here.
00:01:29.131 --> 00:01:30.210
I'm Simon Newton.
00:01:30.210 --> 00:01:30.271
I.
00:01:31.230 --> 00:01:32.551
of platforms at Canva.
00:01:33.391 --> 00:01:41.191
I've been there about three years or so, and, platforms at Canva consists of a bunch of different areas, I guess most interesting today.
00:01:41.400 --> 00:01:50.340
that includes the teams that run our edge and gateway, the, front door into Canva, as the teams that, manage all of our cloud resources.
00:01:50.599 --> 00:01:54.920
So yeah, Canva can be, in the case you haven't heard of it, is a visual communication company.
00:01:55.489 --> 00:02:02.629
what we do is make it really easy for people to express their creative ideas, to others, in a way using visual content.
00:02:03.319 --> 00:02:07.519
we have a bunch of different products, built into what we call the, editor.
00:02:08.300 --> 00:02:12.620
so we have whiteboards and presentations and responsive docs and websites.
00:02:13.189 --> 00:02:15.949
and drawing tools for making social media posts.
00:02:16.639 --> 00:02:18.560
so yeah, there's a very, there's a huge amount you can do.
00:02:20.344 --> 00:02:20.555
Yep.
00:02:20.585 --> 00:02:23.405
That's how I'm gonna make the social media posts for this podcast.
00:02:23.405 --> 00:02:25.594
So, thank you all very much at Canva.
00:02:28.115 --> 00:02:33.094
that's why I was very excited not for you really, but when I saw this incident report, I.
00:02:33.810 --> 00:02:34.560
go by.
00:02:35.159 --> 00:02:40.919
I'm in a Slack group, this resilience and software group, and we have a channel that's just called cases.
00:02:41.490 --> 00:02:44.189
and that's, honestly, I just like, I could sit and watch that all day.
00:02:44.189 --> 00:02:47.280
Like, and so whenever, you know, sometimes we know because things happen to us.
00:02:47.280 --> 00:02:50.699
But I found out about this, this incident report and the outage.
00:02:50.729 --> 00:02:52.409
I did not experience the outage personally.
00:02:53.069 --> 00:02:55.949
but I did find out about the incident report from the Slack group that I'm in.
00:02:56.250 --> 00:03:03.270
And I think also I noticed, That it's the first one you've ever done, right?
00:03:03.300 --> 00:03:07.259
I don't recall ever reading a Canva incident report before.
00:03:08.729 --> 00:03:09.330
So.
00:03:09.810 --> 00:03:33.090
So you've had, and in the report, this report, I think you say you've had an internal process since we're gonna get into the incident, but you've had an incident process since 2017, and I'm always so curious about what it is that pushes an organization over the edge or, I mean, I don't want it to sound so negative, but like what brought you to the point where you said, okay, we're gonna go to the effort of publishing our first public incident report.
00:03:33.389 --> 00:03:33.689
Yeah.
00:03:33.689 --> 00:03:35.550
And it is a bit more effort, right?
00:03:35.759 --> 00:03:40.050
we've been publishing, incident reports internally, like you said, for many, years.
00:03:40.889 --> 00:03:46.379
but there is a difference between, a report for internal consumption and then a polished report for external consumption.
00:03:47.189 --> 00:03:52.409
so in terms of the why behind it, It's really a reflection of how we're evolving as a company.
00:03:52.800 --> 00:03:58.590
so Canva, stuck or got its start, with a bunch of, individuals and small businesses.
00:03:58.830 --> 00:04:01.560
that was typically the, user groups.
00:04:01.949 --> 00:04:07.770
and they'd, be creating social media content and marketing, for their small businesses, and that, that sort of material.
00:04:08.340 --> 00:04:15.000
but in the last couple of years, we've really seen increasing adoption within enterprises, and enterprises bring their own.
00:04:15.479 --> 00:04:19.439
Like new and different set of customer requirements, And customer expectations.
00:04:20.310 --> 00:04:26.490
and so it is part of us wanting to show our commitment to transparency, to those enterprise customers.
00:04:27.220 --> 00:04:34.379
and, also because we believe, and I very much believe, that sort of being open about incident reports does benefit the broader industry.
00:04:35.540 --> 00:04:35.839
and I think
00:04:36.529 --> 00:04:37.074
I agree.
00:04:37.100 --> 00:04:37.699
important.
00:04:37.819 --> 00:04:38.120
Yeah.
00:04:38.509 --> 00:04:41.329
and I, love, I, love what, the VOID is doing here.
00:04:42.589 --> 00:04:49.250
my, my take is that I think software engineering generally as a field does a fairly poor job at learning from previous failures.
00:04:49.790 --> 00:04:51.379
if you look at other engineering fields.
00:04:52.160 --> 00:04:59.959
you often they're like more highly regulated and there's like processes around this where, where, failures are like understood and investigated.
00:05:00.740 --> 00:05:05.779
but in software engineering, we, seem to make the same mistakes, many times, right?
00:05:05.779 --> 00:05:09.980
And there's not that, that not, that's not, that's learning, as a broader field.
00:05:10.699 --> 00:05:17.329
and so yeah, very keen on, on like voids efforts to see, improved education around this, right?
00:05:17.329 --> 00:05:19.790
So that we can all, uplift the industry as a whole.
00:05:21.589 --> 00:05:22.910
Thank you for the shameless plug.
00:05:22.910 --> 00:05:23.480
I appreciate it.
00:05:24.319 --> 00:05:29.209
I wanna ask you the same question I ask everyone at the start of this podcast, which I think is cruel, and yet I do it every time.
00:05:29.209 --> 00:05:31.240
Can you give a brief summary of what happened?
00:05:33.199 --> 00:05:34.550
I will, try and keep it brief.
00:05:35.750 --> 00:05:38.449
the a as sure you are aware, right?
00:05:39.889 --> 00:05:45.259
The, large and interesting, are, because of a confluence of factors, right?
00:05:45.259 --> 00:05:51.949
So in this particular case, they were, there were multiple contributing factors that sort of lined up in a way that caused this much, much, larger incident.
00:05:52.699 --> 00:05:54.019
But maybe a little bit of background.
00:05:54.439 --> 00:05:56.779
so the Canva editor is a single page app.
00:05:57.545 --> 00:05:59.134
we deploy that multiple times a day.
00:05:59.915 --> 00:06:07.685
and amongst other things, as part of that deploy, we, build the JavaScript assets, and then we publish those into an S3 bucket.
00:06:08.314 --> 00:06:17.644
and so then when clients, reload the page, or the editor self updates, those clients then go and, download the new assets, via our edge provider.
00:06:18.605 --> 00:06:26.105
and once those assets are loaded, and the editor is like functional and running, then it starts making API calls to, to load in all the content.
00:06:27.875 --> 00:06:32.225
API calls also go via our edge provider, to what we call our Canva API Gateways.
00:06:32.855 --> 00:06:39.845
and then those gateways route those requests to the various services, within, our systems, which then will, handle them.
00:06:40.444 --> 00:06:46.985
and I guess the, other key bit of information is those gateways run as auto scale groups, within our cloud provider, AWS.
00:06:48.470 --> 00:07:01.250
so in this incident, the first, the first like factor that contributed towards it was we did a deployment of the editor, at the same time that they were network issues that were occurring within our Edge provider.
00:07:01.790 --> 00:07:04.250
and now normally, these network happen.
00:07:04.310 --> 00:07:05.660
Issues happen all the time, right?
00:07:05.660 --> 00:07:07.160
Like the internet is a messy place.
00:07:07.790 --> 00:07:08.899
this is happening day in
00:07:09.589 --> 00:07:09.680
Yep.
00:07:10.475 --> 00:07:17.884
And so normally what happens is our Edge provider has automated software, that can detect these issues and can mitigate them and route around them.
00:07:18.694 --> 00:07:28.685
unfortunately in this particular case, and again, we only learned this later, they had a bit of manual stale configuration in place that have prevented that automation from running.
00:07:29.464 --> 00:07:34.834
so first contributing factor is there's a network issue, and it hasn't been mitigated, automatically.
00:07:37.285 --> 00:07:40.430
As, as often happens with those things, unfortunately.
00:07:40.430 --> 00:07:40.790
Yeah.
00:07:41.470 --> 00:07:41.689
Yep.
00:07:42.319 --> 00:07:48.889
And so then maybe the next bit to, to discuss is if you consider how the Edge proxies work, right?
00:07:48.889 --> 00:07:52.069
You've got a lot of clients, making requests for resources.
00:07:52.699 --> 00:07:57.290
and if that resource isn't in the local case, if the proxy, it all has to go and fetch it from the origin server.
00:07:58.399 --> 00:08:02.959
and it's very likely that there's multiple clients requesting the same bit of content, right?
00:08:03.620 --> 00:08:09.649
and pretty inefficient to have the, or the proxy server then go and turn around and request multiple times from the origin.
00:08:09.740 --> 00:08:16.579
So what you wanna do is you want to coalesce all those requests, do a single origin fetch, and then send that content back out to all the clients.
00:08:17.839 --> 00:08:23.600
and you can probably see where this is going, but, with a bunch of clients requesting the same JavaScript asset,
00:08:25.100 --> 00:08:29.389
Is that the sound, is that like the sound of a thundering herd I hear in the distance?
00:08:29.870 --> 00:08:31.519
yeah, that's, yeah.
00:08:31.564 --> 00:08:42.695
but the interesting bit here was if everything behaved as it was, as, it would normally, latency would go up because the origin fest fetches are taking a bit longer.
00:08:44.014 --> 00:08:45.034
and you know that, that's fine.
00:08:45.365 --> 00:08:47.495
it's just a bit of increased latency for the clients.
00:08:48.514 --> 00:08:56.894
but where we got really unlucky was one of those origin fetches didn't just go out from measured in milliseconds to seconds.
00:08:56.985 --> 00:08:59.445
The origin fetch actually took 20 minutes to occur.
00:09:00.254 --> 00:09:03.735
And, as I'm sure 20 minutes in computing time is an
00:09:04.539 --> 00:09:06.365
It is like geological time.
00:09:06.394 --> 00:09:08.914
Yeah, it's, yeah, it's geologic time, so yeah.
00:09:08.914 --> 00:09:09.664
That's rough.
00:09:09.774 --> 00:09:18.085
And then when we got particularly lucky again, was that this one, one particular asset, was the JavaScript that loads, what we call the object panel in Canva.
00:09:18.774 --> 00:09:23.575
so this is the sort of main place where you interact with content, in your designs, right?
00:09:23.575 --> 00:09:28.075
And so without the object panel loading, the editor is, essentially, frozen and dead.
00:09:28.855 --> 00:09:34.764
and what it also means is that once that one bit of JavaScript loads, it then triggers a bunch of API calls because
00:09:35.539 --> 00:09:35.870
Yeah.
00:09:36.595 --> 00:09:37.570
yes, if you've got.
00:09:38.044 --> 00:09:39.455
A fetch taking 20 minutes.
00:09:40.205 --> 00:09:42.875
got all these clients like coalescing and waiting for it.
00:09:43.294 --> 00:09:44.945
and then suddenly it becomes available.
00:09:45.394 --> 00:09:50.554
that turns into a large number of a API requests, hitting the gateways.
00:09:52.235 --> 00:09:52.745
Yeah.
00:09:53.825 --> 00:09:54.754
it is pretty ugly.
00:09:55.044 --> 00:09:58.764
At that point things fell over in terms of like the autoscaling and whatnot, right?
00:09:58.764 --> 00:10:02.004
At some point, it all just like collapsed on top of itself there, right?
00:10:02.695 --> 00:10:03.054
Yeah.
00:10:03.054 --> 00:10:10.254
And so that turned into, about, I think it was about one and a half million, API requests in a very short amount of time.
00:10:11.335 --> 00:10:13.705
and that was enough to, yeah, overwhelm the gateways.
00:10:13.705 --> 00:10:15.654
Auto scalers can't react fast enough to that.
00:10:16.284 --> 00:10:20.695
gateways went out of memory, and then that started the cascading failure due to overload.
00:10:22.690 --> 00:10:23.139
Okay.
00:10:23.980 --> 00:10:27.429
That's the shortest version of this incident, which is hard.
00:10:28.690 --> 00:10:31.870
and so I think, we'll, we'll talk a little bit more about some of these details.
00:10:32.139 --> 00:10:34.720
but how did you all find out about this?
00:10:35.899 --> 00:10:36.815
I can run you through that.
00:10:37.595 --> 00:10:47.254
so the, first signs that something was wrong, was about 10 minutes after the deploy, because we started noting, noticing a drop in search traffic.
00:10:47.945 --> 00:10:51.365
and so our search on call team was paged, for a drop in traffic.
00:10:51.365 --> 00:10:53.105
I think they went down around 20% or so.
00:10:54.080 --> 00:10:55.639
and so they started investigating.
00:10:56.539 --> 00:11:01.429
and at first they thought oh, maybe this is something like very isolated to search.
00:11:03.455 --> 00:11:18.004
as they started, communicating in an incident room, it was a sev one, which is our, second highest tier of, incident At that point, as they started communicating, other service owners started getting involved and saying oh, actually, like we've also noticed a drop in traffic.
00:11:19.254 --> 00:11:23.754
And as they were doing that, that was about the time that the fetch itself completed.
00:11:24.325 --> 00:11:28.794
and then our gateway and edge teams got paged for, massive gateway failures.
00:11:29.514 --> 00:11:31.764
at that point, the incident was upgraded to a sev zero.
00:11:32.004 --> 00:11:33.445
The gateway teams were pulled in.
00:11:33.774 --> 00:11:38.605
I was pulled in and we, the incident, incident coordinator was, activated.
00:11:40.284 --> 00:11:41.215
and then, we went from there.
00:11:41.519 --> 00:11:41.639
Yeah.
00:11:41.639 --> 00:11:44.669
What is your incident response look like?
00:11:44.669 --> 00:11:51.480
How, you know, do you all have sort of some, you know, protocols or what does that look like and how, how many people ended up being involved?
00:11:51.480 --> 00:11:55.230
If you can tell me, because it sounds like it was a doozy, and a lot of people got called in.
00:11:55.230 --> 00:11:57.690
I'm always really curious, like how many hands are on deck?
00:11:58.514 --> 00:12:03.059
Yeah, I would guess probably at least 20 or so.
00:12:04.169 --> 00:12:06.269
so let me, describe the process.
00:12:07.754 --> 00:12:11.355
typically, alerts will page, service on call teams.
00:12:11.985 --> 00:12:13.845
I think it's a, very typical process, right?
00:12:13.894 --> 00:12:30.544
Those service teams will triage, an open incident, if it's a SEV zero or SEV one, that'll activate the incident coordinator rotation, which is a, it's a small group of people, that have a lot of training and skills around managing, and coordinating these large scale incidents.
00:12:31.355 --> 00:12:33.634
the Sev Zeros will page myself.
00:12:35.134 --> 00:12:40.565
and then it, really depends on the incident as to the structure of it, right?
00:12:40.565 --> 00:12:46.654
If it's a smaller incident, you might have only a handful of people, and they'll be fulfilling all the roles, right?
00:12:47.014 --> 00:12:53.710
if it's a large incident, like in this one, we'll have, we'll, set it up so that there's dedicated people running each particular role.
00:12:54.544 --> 00:12:56.914
I think maybe one, one thing that's important as well.
00:12:57.605 --> 00:13:06.845
Is that, the larger or the highest severity incidents, will automatically assign a representative from our customer support organization.
00:13:08.355 --> 00:13:08.894
Oh, okay.
00:13:09.605 --> 00:13:15.875
all of the user reports, and, looking for signal there to not overwhelm the, sort of the responding team.
00:13:16.235 --> 00:13:16.355
and
00:13:17.125 --> 00:13:17.544
Mm-hmm.
00:13:17.735 --> 00:13:21.815
the user support, so that they can feedback look, this is the state of things.
00:13:21.875 --> 00:13:25.625
If people are, contacting or reaching out to Canva, this is what we should be telling them.
00:13:25.625 --> 00:13:26.855
If there's anything they can do.
00:13:27.960 --> 00:13:31.049
to mitigate it from a user perspective, that as well.
00:13:32.580 --> 00:13:32.789
yeah.
00:13:32.789 --> 00:13:36.509
And then, post incident, like I mentioned, we'll write those incident reports.
00:13:36.559 --> 00:13:37.460
and then we also.
00:13:38.394 --> 00:13:49.105
we've just, recently started using AI to try and extract common themes, across all of those reports as well, to, look for areas where we can, where we can improve.
00:13:49.306 --> 00:13:50.475
I have so many questions.
00:13:51.196 --> 00:13:53.770
so the IC is getting pulled.
00:13:54.316 --> 00:13:58.725
Sort of automatically, if it's like sev one, sev zero, how is that team?
00:13:59.416 --> 00:14:04.096
Are they, are they, also engineers on, on different teams?
00:14:04.096 --> 00:14:08.115
Are they kind of a, a dedicated SRE esque function?
00:14:08.466 --> 00:14:10.745
And when they're not ic, what are, what are they doing?
00:14:10.745 --> 00:14:12.275
I'm kind curious what the shape of that looks like.
00:14:12.701 --> 00:14:12.931
Yeah.
00:14:12.931 --> 00:14:21.240
So early on, say our, ICS were just a bunch of very experienced, battle hardened engineers at Canva.
00:14:22.951 --> 00:14:28.921
and then over time, of course, as we get bigger, we need to set up a dedicated function.
00:14:29.610 --> 00:14:32.880
and those, because those, folks are also, they've got their own projects, right?
00:14:32.880 --> 00:14:35.490
They their own deliverables, that they're working on.
00:14:35.671 --> 00:14:38.071
So yeah, IC is a dedicated function at Canva.
00:14:39.030 --> 00:14:42.421
when they're not on call, they're doing a bunch of other different things.
00:14:42.916 --> 00:14:48.525
some of those is improving, the incident process itself, looking for those patterns in incidents.
00:14:49.275 --> 00:14:53.235
plus also there's a bunch of work that goes into our large, launch events.
00:14:53.265 --> 00:14:56.716
So we do two, two main launches a year, and we've got one coming up now in April.
00:14:57.525 --> 00:15:00.285
and so they're, like doing a bunch of planning for that.
00:15:00.765 --> 00:15:09.826
there's a bunch of capacity planning involved, like risk, un understanding the risk and mitigating, the risks that are unique to each one of those launches.
00:15:10.535 --> 00:15:13.385
And so yeah, that's a, that's like a dedicated reliability function.
00:15:13.633 --> 00:15:22.783
When you have like a pretty big incident like this one, do you have execs in the, in the channels, in the, active incident channels?
00:15:22.783 --> 00:15:27.403
You, or like, or do you have a dedicated role who's talking to folks and kind of coordinating back or forth?
00:15:27.793 --> 00:15:28.452
What does that look like?
00:15:29.123 --> 00:15:29.423
Yeah.
00:15:29.423 --> 00:15:33.442
So in this particular case, the founders, our founders, did join the incident channel.
00:15:33.442 --> 00:15:33.472
I.
00:15:34.618 --> 00:15:39.177
typically what will happen, sometimes they're there, sometimes they're, they're busy with other things.
00:15:40.138 --> 00:15:49.977
typically like myself or one of the other senior leaders will act as that channel, to the, founders, to give them updates and, help'em understand what's going on.
00:15:50.327 --> 00:16:00.227
So you mentioned you found out later, so was was the CloudFlare wrinkle of their traffic, and you didn't mention this explicitly, but I think this is what you're getting at with the network configuration.
00:16:00.227 --> 00:16:03.467
So their traffic was going over the public internet, not over their private backbone.
00:16:03.738 --> 00:16:05.508
Is that a piece of it that you found out later?
00:16:05.567 --> 00:16:09.107
And so that was sort of like, you didn't know why that was contributing to the problem.
00:16:11.207 --> 00:16:11.508
yes.
00:16:12.138 --> 00:16:15.977
we don't have visibility into that traffic between, cloud CloudFlare and
00:16:16.727 --> 00:16:17.028
Yeah.
00:16:17.207 --> 00:16:18.467
in A-W-A-W-S.
00:16:19.278 --> 00:16:26.837
so yes, that was a detail that we, didn't find out until, I think maybe one or two days after the incident as we were working with CloudFlare to understand what happened.
00:16:27.034 --> 00:16:31.865
At, at some point, did anyone have the intuition that this was gonna get worse in the way that it did?
00:16:32.735 --> 00:16:40.115
Like once the files were downloaded from the origin server and, and there was, did you see that, hear the thundering heard coming?
00:16:40.115 --> 00:16:49.804
Like, I'm always a little curious what the, what the existential dread might be like, like in an incident or was that, did that catch you by surprise as well?
00:16:50.134 --> 00:16:53.855
yeah, it all happened, like fairly quickly in terms
00:16:54.700 --> 00:16:54.990
Okay.
00:16:55.595 --> 00:16:56.705
the unfolding incident.
00:16:58.414 --> 00:17:06.394
yeah, by the time that folks were in the room saying oh, this is, broader than search, that was about the time that origin fetch completed.
00:17:06.815 --> 00:17:08.990
and then the gateway started starting, falling, over.
00:17:10.131 --> 00:17:16.280
and so that, so even so the, like the, all the clients hitting the API gateway, you've got a known performance issue.
00:17:16.280 --> 00:17:18.260
So that all happened really quickly.
00:17:18.560 --> 00:17:22.701
And the the, or were piece of those only aware after the fact as well.
00:17:23.826 --> 00:17:24.365
Yes.
00:17:24.365 --> 00:17:26.375
I didn't actually touch on the performance side.
00:17:28.550 --> 00:17:29.060
get into that.
00:17:29.060 --> 00:17:30.921
Then I jumped ahead.
00:17:32.405 --> 00:17:34.326
Gateways are now, now crashing.
00:17:35.421 --> 00:17:40.611
we have a bunch of people on the call and that's where we started breaking up into different work streams, right?
00:17:41.391 --> 00:17:46.790
there were a set of people that were responsible for, contact taking our vendors, and getting them involved.
00:17:47.240 --> 00:17:49.221
That process differs across vendors, right?
00:17:49.221 --> 00:17:54.621
Like some of them, will be like, we'll go to our portal and sevmit like a, a P zero request.
00:17:54.621 --> 00:17:59.810
Others will be like, email this address and it will page sort of your on-call account managers.
00:18:00.411 --> 00:18:04.280
and so that process there, needs to be well practiced, right?
00:18:04.280 --> 00:18:08.151
And I think this is actually something that, we improved afterwards as well.
00:18:08.570 --> 00:18:15.260
we have internal docs on how to escalate to each of our vendors, but we tied it up that documentation and just made it a bit clearer because when you are.
00:18:15.996 --> 00:18:19.086
you're doing that, you are already in a high stress environment, right?
00:18:19.086 --> 00:18:19.865
And so you don't want to
00:18:20.780 --> 00:18:21.411
Absolutely.
00:18:21.455 --> 00:18:23.736
and have four pages of workflow to go through.
00:18:23.736 --> 00:18:26.976
It needs to be a very clear, do these four things, right?
00:18:27.951 --> 00:18:28.340
Yeah.
00:18:28.445 --> 00:18:41.810
had a set of people contacting vendors, We had one of the engineers who was not on call at the time, but was, saw the incident and jumped on, as, many people at Canva do.
00:18:42.080 --> 00:18:48.711
We, there's a very good, culture of, banding together, especially for these large incidents.
00:18:49.340 --> 00:18:53.990
so we had one, one engineer who went off and started profiling the gateway.
00:18:55.411 --> 00:19:02.405
this is he was reporting back in saying oh look, actually, profile looks different.
00:19:03.336 --> 00:19:08.705
I think, if I remember correctly, actually had, he'd done this before, other times.
00:19:08.855 --> 00:19:11.435
And so he had past profiles to compare it to.
00:19:12.096 --> 00:19:17.316
and he was like, look, when I did this two weeks ago, this, one looks different, right?
00:19:17.316 --> 00:19:18.695
And I'm seeing lock contention here.
00:19:19.560 --> 00:19:24.631
And so that was, like one, one avenue that we thought, might have been the contributing factor.
00:19:25.141 --> 00:19:32.131
and so we had a bunch of people exploring that, that turned out to be, a change in our metrics library.
00:19:32.730 --> 00:19:36.270
what we were doing was changing, making some changes to the way that metrics were collected.
00:19:37.111 --> 00:19:40.681
and so we were integrating a, like a third party library there.
00:19:41.671 --> 00:19:45.990
that had done is, inadvertently put metric registration behind a lock.
00:19:46.500 --> 00:19:48.961
and so the capacity of our gateways were reduced.
00:19:49.730 --> 00:19:56.510
which would then also contribute to the sort of the overload situation, and what we saw with the cascading failure.
00:19:56.961 --> 00:20:00.830
but that was all emerging in parallel to the, to the,
00:20:00.891 --> 00:20:01.181
Yeah.
00:20:01.221 --> 00:20:02.631
activity that was happening on the call.
00:20:02.771 --> 00:20:06.967
So you got the performance issue, which must have been.
00:20:07.646 --> 00:20:10.676
And if I'm not wrong, there was this really juicy detail in there.
00:20:11.037 --> 00:20:21.446
these are the kinds of things that I love it when companies share, where it was a known issue and there was a fix for it in like the deployment pipeline for that day.
00:20:21.686 --> 00:20:23.636
That had not gone out yet.
00:20:23.817 --> 00:20:30.957
And in many organizations there could be a lot of blame and like post hoc, like, I can't believe we didn't do that.
00:20:30.957 --> 00:20:40.961
We should have, like what was the general sense when you all realized that that also was contributing to this incident?
00:20:41.811 --> 00:20:42.021
Yeah.
00:20:42.862 --> 00:20:45.231
definitely, there, there was never any blame.
00:20:46.372 --> 00:20:53.241
and it was just very unfortunate that yes, the metrics team had already identified this, they'd already fixed it, it was already merged.
00:20:54.412 --> 00:20:58.521
and, I guess what would that have been in, in 12 hours later?
00:20:59.092 --> 00:21:03.112
the new gateway would've been deployed and, this have very different.
00:21:04.406 --> 00:21:04.696
yeah.
00:21:04.821 --> 00:21:05.811
just getting unlucky.
00:21:07.102 --> 00:21:08.751
we do have a project underway.
00:21:10.106 --> 00:21:12.896
to, move to incremental deploys.
00:21:13.346 --> 00:21:25.467
right now I'd say most backend components, are deployed on a daily cycle, and then, and then the components individually can opt into incremental deploys, and deploy on merge.
00:21:25.856 --> 00:21:35.007
but we do want to move more and more of our components to like over to that incremental model, which would've, ideally caught this, in this case.
00:21:35.183 --> 00:21:48.443
And so then the last, but not least, certainly I believe, again, if I've got the timeline right here, so you've got that going on, and then basically you had like the load balancers on your ECS containers, just they couldn't.
00:21:48.959 --> 00:21:51.239
Like they were just getting hammered at this point as well.
00:21:51.239 --> 00:21:51.568
Right.
00:21:51.568 --> 00:22:01.288
And was that also, a matter of sort of, this was reaching out and trying to figure out what was happening with AWS or like what, what was that last piece of the puzzle like?
00:22:01.288 --> 00:22:01.348
I.
00:22:02.659 --> 00:22:07.068
Yeah, so we did, we had a set of people, escalating CloudFlare.
00:22:07.308 --> 00:22:16.068
we had a set of people reaching out to AWS because, we were at that point, this was just after that Origin Fetch had completed.
00:22:17.669 --> 00:22:27.148
we were concerned that there might be something going on the AWS side, but thankfully, we have, I would say like reasonably good observability into all of this, right?
00:22:28.048 --> 00:22:34.919
and a cascading failure due to overload is a very distinct pattern in a graph, right?
00:22:34.919 --> 00:22:37.648
Where you get this, sawtooth pattern, like if you're looking at
00:22:38.364 --> 00:22:38.653
Yeah.
00:22:38.788 --> 00:22:40.439
uptime, you get this sawtooth.
00:22:40.598 --> 00:22:42.429
That sort of offset with every instance.
00:22:43.088 --> 00:22:48.249
and I've certainly seen that before, a number of other people on the call had seen that before.
00:22:48.729 --> 00:22:50.108
and so it was very clear at that point.
00:22:50.108 --> 00:22:53.078
It's oh no, this is, this is a, an overload situation.
00:22:53.888 --> 00:22:58.118
and then, a again, we've, faced this before, right?
00:22:58.989 --> 00:23:00.219
I've, seen, quite a few of these.
00:23:00.969 --> 00:23:08.199
and there's, really only one option, which is you've got to, bring the load down and get the system back into a stable state, right?
00:23:08.828 --> 00:23:10.209
You can, try and add capacity.
00:23:10.433 --> 00:23:19.193
The problem is un unless you can do that in a, an atomic flip fashion, any new additional capacity that you bring up is just gonna get hammered into the ground.
00:23:19.644 --> 00:23:21.203
and so you've gotta cut that load back.
00:23:22.534 --> 00:23:22.798
In this
00:23:22.834 --> 00:23:23.124
Yeah.
00:23:23.219 --> 00:23:33.959
case, you know what would be really nice if is if we had a sort of a, lever where we could just say oh, okay, great, like dial the load to 20% of demand.
00:23:34.679 --> 00:23:38.398
we didn't have that sort of control within the cloud providers systems.
00:23:38.848 --> 00:23:44.219
so instead what we did was reach for, the, sort of country level controls, right?
00:23:44.219 --> 00:23:53.969
And so we put in a block that said, For any country in the world, want to display this status message instead of forwarding the requests onto the, backend.
00:23:54.148 --> 00:23:56.969
So that was a, that, that was the lever that we had at the time.
00:23:58.169 --> 00:23:58.288
and
00:23:58.288 --> 00:23:58.739
Yeah.
00:23:59.038 --> 00:24:02.278
doing was, once, once we got that block in place, I.
00:24:02.443 --> 00:24:06.193
We saw the traffic drop to zero, we saw all the gateways come up and stabilize.
00:24:06.763 --> 00:24:08.534
and then we started admitting more load.
00:24:08.983 --> 00:24:17.354
I think the, maybe the interesting bit here is that, we added Europe back in first because that was where the peak load of the time was, right?
00:24:17.354 --> 00:24:23.114
This happened in Australian, like early to mid evening time, would've been European daytime.
00:24:23.144 --> 00:24:25.304
The US was mostly asleep at that point.
00:24:25.844 --> 00:24:31.663
and so we Europe to be admitted back first, and then, rolled out, out to the rest of the world.
00:24:33.479 --> 00:24:48.719
Yeah, I mean that was an interesting piece of what I like to think of as hard earned expertise where instead of just turning everything back on again and literally recreating potentially the same problem, which I've definitely seen incidents where they're like, oh no.
00:24:48.719 --> 00:24:50.519
And it like literally happens again.
00:24:51.282 --> 00:24:57.163
the only other piece I was curious about is in terms of what that incident response looks like, is like, are people remote?
00:24:57.432 --> 00:24:59.202
Is everything in a Slack channel?
00:24:59.718 --> 00:25:00.377
How does that work?
00:25:00.377 --> 00:25:01.758
Is everybody in the same time zone?
00:25:02.688 --> 00:25:04.127
How complicated does that look for you all?
00:25:04.738 --> 00:25:04.968
Yeah.
00:25:04.968 --> 00:25:08.018
Canva is a hybrid work setup.
00:25:08.798 --> 00:25:14.798
so I'd say the bulk of our engineering is in the New Zealand to West coast of Australia.
00:25:15.188 --> 00:25:15.728
Time, zone.
00:25:17.738 --> 00:25:20.948
and so yeah, this was like early to mid evening for them.
00:25:22.327 --> 00:25:23.557
but yes, as part of.
00:25:24.597 --> 00:25:25.347
The automation.
00:25:25.347 --> 00:25:28.917
When an incident is triggered, it automatically creates a slack room.
00:25:29.218 --> 00:25:36.268
it ties that slack room to the main, incident slack room, so people, can go in there and find it, and it sets up the Zoom call as well.
00:25:37.077 --> 00:25:40.617
and so then I'd say the majority of the discussion occurs on Zoom.
00:25:41.008 --> 00:25:47.998
maybe another interesting point is that we, have deliberately set up that zoom in a way that, that Zoom chat is disabled.
00:25:48.448 --> 00:25:50.488
so it funnels all of the chat through the slack room.
00:25:50.817 --> 00:25:50.998
so
00:25:51.798 --> 00:25:52.367
Ah, okay.
00:25:52.488 --> 00:25:52.758
Yeah.
00:25:52.768 --> 00:25:54.478
I think that you don't have two, two different,
00:25:55.877 --> 00:25:57.768
Two sources of, yeah.
00:25:57.978 --> 00:25:58.307
Yeah.
00:25:58.768 --> 00:26:05.367
and so typically people will share, will be sharing links to dashboards or logs or et cetera in the Slack channel.
00:26:06.357 --> 00:26:20.337
and then we have a way of annotating within the Slack channel sort of key moments, in a way that it's very easy to, when you're going back later trying to write that incident report, you can pull out the key events and say okay, this is what we knew at this particular time.
00:26:20.798 --> 00:26:26.917
homegrown tool that you all use, or is that a, that's a third party tool for the, for the incident sort of.
00:26:28.042 --> 00:26:33.667
yeah, I think it's mostly a bunch of, a bunch of tools developed by that same set of, incident coordinators.
00:26:33.972 --> 00:26:39.518
So some companies say, you know, for, for, I.
00:26:39.837 --> 00:26:46.228
Sev whatever number, we're gonna always have an incident review and we're gonna try to do it within this amount of time.
00:26:46.768 --> 00:26:56.577
and do you have like a dedicated incident analysis team and with an analysts, and do they have, processes around that and, and timeframes for when you want things to happen?
00:26:56.577 --> 00:26:57.867
Or is that more ad hoc?
00:26:58.553 --> 00:27:02.063
Yeah, so we don't have a dedicated team of people analyzing these.
00:27:02.722 --> 00:27:05.063
the incident coordinators will work with the.
00:27:05.353 --> 00:27:08.833
The sort of service teams that were contributing towards the incident.
00:27:09.702 --> 00:27:13.692
and then we have a, a template for, the, PIRs, what we call them.
00:27:15.522 --> 00:27:19.093
I don't, I, don't think there is a sort of a set deadline.
00:27:19.813 --> 00:27:20.442
but certainly I.
00:27:21.292 --> 00:27:26.482
myself, I'll be, because I'll be involved in the Sev Zeros I'll be looking for that report.
00:27:27.442 --> 00:27:33.772
and if it hasn't shown up in maybe two, two or three weeks, I'll be, ping people saying where, is it at that point?
00:27:34.792 --> 00:27:38.272
no sort of system that, you
00:27:39.063 --> 00:27:39.353
Yeah.
00:27:39.357 --> 00:27:39.597
Yeah.
00:27:40.073 --> 00:27:43.673
it is all linked in the, in that automation, in that slack room as well.
00:27:44.692 --> 00:27:45.502
as the state.
00:27:45.548 --> 00:27:48.218
Moves through from, like responding.
00:27:48.488 --> 00:27:53.798
so I guess like triage, responding to mitigated, to resolved, that's all as
00:27:53.817 --> 00:27:54.147
Yeah.
00:27:54.788 --> 00:27:55.988
and, fed into that timeline.
00:27:57.557 --> 00:28:01.248
Is there like a PIR meeting anyone can come to?
00:28:01.278 --> 00:28:03.208
Like how are the reports shared internally?
00:28:04.218 --> 00:28:07.968
Yeah, so it's, it's then published when it is published.
00:28:08.208 --> 00:28:10.998
it's done typically is like the last message in that Slack room.
00:28:11.807 --> 00:28:13.157
that, that then gets archived.
00:28:13.728 --> 00:28:24.528
we have a weekly meeting as well where we review, the Sev Zeros plus, like of the, other, lesser, severity, but maybe like interesting, failure modes.
00:28:25.488 --> 00:28:30.048
and then we publish, in our sort of monthly engineering leads meeting.
00:28:31.307 --> 00:28:35.117
we, take the, list of incidents, and we go through them briefly there.
00:28:35.867 --> 00:28:42.107
that's so that all the engineering leads have the same level of visibility, and that we're identifying common themes.
00:28:42.747 --> 00:28:50.362
Do you find that the, that those reports get referenced in like planning or are they, are they used.
00:28:50.916 --> 00:28:52.866
Do you see them being used after the fact?
00:28:53.166 --> 00:29:02.106
you know, as sort of learning pieces in terms of either yes, for planning or for architecture reviews or like any of those kinds of things.
00:29:02.406 --> 00:29:10.176
Yeah, so the action items that come out of those reports, get created in Jira, which will then be sitting on those teams backlogs.
00:29:10.866 --> 00:29:17.527
and so then, next, next planning iteration, the team will be looking at, looking at their sort of like reliability backlog.
00:29:18.172 --> 00:29:27.501
and then like mixing that with the other sort of various categories of work on the backlog, and then using that to prioritize, what, what, work occurs.
00:29:27.846 --> 00:29:32.676
And is there, is there anything outside of action items in terms of like distributing what you learned?
00:29:33.307 --> 00:29:36.866
or is there any other activity around the incidents like that?
00:29:38.017 --> 00:29:42.547
I'd say no, other than the, sort of internal Slack channels where,
00:29:43.386 --> 00:29:43.596
Yeah.
00:29:43.626 --> 00:29:43.836
right?
00:29:43.836 --> 00:29:48.682
And so anyone in the company is free to and look at those, and, read through the reports.
00:29:48.957 --> 00:29:51.457
What was the most surprising thing to you about this incident?
00:29:52.734 --> 00:29:59.365
yeah, probably the biggest one is that our edge provider would coalesce requests indefinitely.
00:30:01.375 --> 00:30:06.505
I can, understand, I can understand why that was the outcome.
00:30:07.075 --> 00:30:11.335
And I can imagine, I can put myself in the person writing that code.
00:30:11.994 --> 00:30:17.875
and it's oh, like if, fetch in flight, wait for it to complete and then respond, right?
00:30:19.795 --> 00:30:22.404
that was what allowed that thundering herd to develop.
00:30:23.664 --> 00:30:24.234
and I.
00:30:25.210 --> 00:30:31.660
all of our internal RPC traffic has timeouts, which are propagated and respected at each hop in that service graph.
00:30:33.519 --> 00:30:48.309
in this particular case, there wasn't, like any, timeout within that, within that edge providers, layer, which makes it, somewhat difficult from a client code perspective.
00:30:48.684 --> 00:30:51.924
To understand, is this an error?
00:30:52.734 --> 00:30:58.194
is this just like a slow fetch, or is this I part of a thundering herd?
00:30:58.404 --> 00:30:58.704
In which
00:30:59.440 --> 00:31:00.339
Hey, there's
00:31:00.744 --> 00:31:02.845
would wanna do something differently, add
00:31:03.549 --> 00:31:04.180
right.
00:31:04.704 --> 00:31:05.005
Yeah.
00:31:05.457 --> 00:31:16.977
Yeah, it definitely seemed like one of the themes from this incident as well was, and I, I mean I see this a lot and I write about it a lot, was how automation actually made it worse.
00:31:17.156 --> 00:31:17.576
Right.
00:31:18.626 --> 00:31:24.057
and not because you're like, we want the automation to make things worse, but because it, it can't always.
00:31:24.461 --> 00:31:31.001
React to these kinds of situations and you know, like it's model of the world is not as flexible as ours is.
00:31:31.721 --> 00:31:44.112
and so, you know, I, I definitely saw like action items out of that and everything in the report, but was there, what, was there any reaction to that in terms of like where you had these.
00:31:44.727 --> 00:31:58.247
Automation surprises of, of, and not just like, what was your response to the sort of automation, making it harder and, and how are you all adapting to that after the fact for the future?
00:31:58.967 --> 00:32:10.811
Yeah, so maybe a good example is, again, I didn't, mention this, but Part of what happened on that call, was there was a person responsible for freezing the auto scaling.
00:32:11.757 --> 00:32:15.146
because what can happen here is, your traffic go goes to zero.
00:32:15.656 --> 00:32:20.757
your auto, especially if that's, like over a, sort of like long amount of time.
00:32:21.176 --> 00:32:22.527
the auto scalers are like, oh, great.
00:32:22.527 --> 00:32:26.037
Like we can like down downscale all of the groups, and
00:32:26.731 --> 00:32:27.152
Yeah.
00:32:27.176 --> 00:32:27.656
course they can't
00:32:28.352 --> 00:32:29.612
And like, oh no again.
00:32:30.031 --> 00:32:30.362
Yeah.
00:32:30.541 --> 00:32:42.122
So we've got processes in place where, you know, for those particular types of incidents, we have tooling that we can run, again, developed, by those reliability folks where we can say hey, freeze all the auto scalers.
00:32:42.811 --> 00:32:50.701
and what we can do also is, go back and say oh, actually, set them, reset them to where they were like prior to the incident occurring, right?
00:32:50.701 --> 00:32:53.912
So if, any have down downscaled, we can get them back.
00:32:54.362 --> 00:32:57.152
and so that's a, process that we've adapted.
00:32:58.396 --> 00:33:09.946
given past experience and thinking through what's, like bad outcomes that could happen in these sort of event events that, turns the automation off for a second and humans, take control.
00:33:09.946 --> 00:33:10.007
Yeah.
00:33:11.747 --> 00:33:19.461
Yeah, I mean that's definitely a theme of, of obviously my work, but like in, instead of just saying, well, we'll add more automation to make the automation.
00:33:19.636 --> 00:33:20.057
Better.
00:33:20.267 --> 00:33:26.656
Y'all were like, well, let's give the humans better controls over what that might look like.
00:33:26.656 --> 00:33:29.777
I like, I like seeing those kinds of, of adaptations, so
00:33:30.011 --> 00:33:36.461
like I'm, I'm a very big fan of modes within automated software.
00:33:36.836 --> 00:33:44.217
Where it just throws up its hands and it says, this, does not match my, view, like my, what I've been modeled for and
00:33:44.951 --> 00:33:45.241
yeah.
00:33:45.717 --> 00:33:45.987
right?
00:33:46.586 --> 00:33:51.926
when I've been building systems in the past, I've built in these, lockdown modes, right?
00:33:51.926 --> 00:33:54.567
Where it's hey, like the inputs don't match anymore.
00:33:54.567 --> 00:33:55.886
I'm gonna go into lockdown mode.
00:33:55.946 --> 00:33:57.537
I'm gonna flag it in my telemetry.
00:33:58.596 --> 00:34:00.576
human come and get me out of this, right?
00:34:00.576 --> 00:34:09.456
Because rather than just like continuing down a path that sort of, no one has prepared, the software for, and, possibly making things much, much worse.
00:34:09.817 --> 00:34:15.146
And what do you think was the biggest learning for the team out of this incident?
00:34:15.483 --> 00:34:21.302
I think definitely practicing, those controls, on the Edge Providers network.
00:34:22.413 --> 00:34:23.827
when we were putting that block in place.
00:34:25.487 --> 00:34:28.097
we, added it with a very simple status message.
00:34:28.427 --> 00:34:32.538
I can't actually remember what it said, but it was definitely not a user friendly status message.
00:34:33.708 --> 00:34:34.577
because at the,
00:34:35.507 --> 00:34:35.967
All bad.
00:34:35.967 --> 00:34:36.487
Good luck.
00:34:36.902 --> 00:34:37.202
yeah.
00:34:37.853 --> 00:34:39.257
at the, wasn't quite that bad.
00:34:40.097 --> 00:34:42.378
but because at the time, at the
00:34:43.307 --> 00:34:46.907
I could come up with a lot of bad error messages for you if you'd like, but probably not.
00:34:47.628 --> 00:34:48.168
Yeah.
00:34:49.217 --> 00:35:00.168
so, we, because at the time the priority was like, get the load down rather than, debate the error message or, like how it was, gonna be visually presented, et cetera.
00:35:01.532 --> 00:35:01.822
Yeah.
00:35:02.327 --> 00:35:03.047
we've learned from that.
00:35:03.047 --> 00:35:06.498
We now have canned response ready to go, in a much more.
00:35:06.978 --> 00:35:09.168
user friendly and visually appealing style.
00:35:10.128 --> 00:35:12.978
and so that, yeah, that's, like one, one thing that's come outta that.
00:35:13.697 --> 00:35:19.938
also our, ability to practice using those controls and getting more familiar with them.
00:35:20.447 --> 00:35:25.967
cause you, really don't want to be using something for the first time, in a high pressure environment, right?
00:35:25.967 --> 00:35:30.483
Like you want to have this, know, built up almost in your muscle memory.
00:35:30.932 --> 00:35:43.293
one of the other, like general principles that I try and describe to when, when building these systems is that any sort of emergency or failure mode, is exercised day to day.
00:35:43.802 --> 00:35:45.782
And that way, that your
00:35:46.038 --> 00:35:46.217
Yep.
00:35:46.353 --> 00:35:46.713
for it.
00:35:46.713 --> 00:35:49.862
You've been using it, day to day or once a week.
00:35:50.012 --> 00:35:53.492
And it's not a, oh, I've gotta do this first, first.
00:35:53.853 --> 00:35:54.362
this thing.
00:35:54.393 --> 00:35:56.103
First time in a call.
00:35:57.018 --> 00:36:01.007
there's like a documentation page that is like four pages long, right?
00:36:01.007 --> 00:36:01.248
And I've
00:36:02.068 --> 00:36:02.327
Yeah,
00:36:02.538 --> 00:36:03.588
of steps perfectly,
00:36:04.427 --> 00:36:09.918
oven's on fire and I've never used the like fire extinguisher thing before and it can't get the Yeah.
00:36:09.918 --> 00:36:10.307
Basically.
00:36:10.307 --> 00:36:10.487
Yep.
00:36:10.608 --> 00:36:10.847
Yep.
00:36:11.657 --> 00:36:12.708
And do you all do.
00:36:13.458 --> 00:36:13.518
Yeah.
00:36:14.268 --> 00:36:20.628
Do you do drills like tabletop drills, chaos engineering type stuff, or what does that preparedness look like?
00:36:20.748 --> 00:36:23.507
Is how is that structured and definitely happens?
00:36:24.228 --> 00:36:27.257
I'd say there's a, variety of drills that occur, right?
00:36:28.068 --> 00:36:34.248
like teams themselves will do, will do wheel of misfortune style or like role playing incidents.
00:36:35.177 --> 00:36:44.637
but then there's also like larger drills that we do as a company, where we say oh, okay, yes, we're going to, we're gonna do like a sort of a, business continuity drill.
00:36:45.788 --> 00:36:48.978
will get involved and, do those exercises.
00:36:49.202 --> 00:36:54.873
Thank you so much for joining me and sharing your internal process with the world.
00:36:54.932 --> 00:37:02.463
And while I hope you don't have, more sev zero incidents, I do hope that you continue to share your incident reports with us.
00:37:02.463 --> 00:37:07.773
And I, I really appreciate all of the time and effort and cat herding that goes into that.
00:37:08.163 --> 00:37:13.112
but as you said, it really is so beneficial to everyone else in the industry, so thank you for doing it.
00:37:13.592 --> 00:37:14.193
Yeah, no worries.
00:37:14.342 --> 00:37:19.293
Yeah, I just, I hope people can, learn from it and we can get better, everyone together.
00:37:21.003 --> 00:37:21.543
Absolutely.
00:37:21.992 --> 00:37:22.233
Okay.
00:37:22.322 --> 00:37:23.043
Thanks so much.
00:37:23.733 --> 00:37:24.123
very much.
00:00:00.542 --> 00:00:05.852
Greetings, fellow incident nerds and welcome to season two of the Void Podcast.
00:00:06.331 --> 00:00:10.801
The main new thing for this new season is we're now available in video.
00:00:11.221 --> 00:00:17.071
So if you're listening to this and prefer watching me make odd faces and not a lot, you can find us on YouTube.
00:00:17.071 --> 00:00:18.481
The link is in the show notes.
00:00:18.961 --> 00:00:21.722
The other new thing is we now have sponsors.
00:00:22.292 --> 00:00:27.961
Don't worry, these folks help make the podcast possible, but they don't have any say over who joins us or what we talk about.
00:00:28.172 --> 00:00:29.341
So fear not.
00:00:29.942 --> 00:00:32.515
This episode's sponsor is Uptime Labs.
00:00:32.966 --> 00:00:37.975
Uptime Labs is a pioneering platform specializing in immersive incident response training.
00:00:38.246 --> 00:00:46.405
Their solution helps technical teams build confidence and expertise through realistic simulations that mirror real world outages and security incidents.
00:00:46.975 --> 00:00:54.865
When most of investment these days in the incident space goes to technology and process, Uptime Labs focuses on sharpening the human element of incident response.
00:00:55.585 --> 00:00:55.915
All right.
00:00:56.636 --> 00:01:01.856
In this episode, we talked to Simon Newton, head of platforms at Canva.
00:01:02.456 --> 00:01:05.816
They just published their first public incident report.
00:01:05.876 --> 00:01:14.816
It's not their first incident, but it's the first time we get to delve into the details of what happened, and we really appreciate them sharing that with us and the world.
00:01:14.820 --> 00:01:17.006
So let's get into it and talk to Simon.
00:01:17.055 --> 00:01:20.355
Simon, thank you so much for joining me on The Void Podcast.
00:01:20.566 --> 00:01:25.756
Will you do our listeners a favor and introduce yourself and tell me where you work and what you do.
00:01:26.941 --> 00:01:27.271
Hi.
00:01:27.271 --> 00:01:28.501
I'm very happy to be here.
00:01:29.131 --> 00:01:30.210
I'm Simon Newton.
00:01:30.210 --> 00:01:30.271
I.
00:01:31.230 --> 00:01:32.551
of platforms at Canva.
00:01:33.391 --> 00:01:41.191
I've been there about three years or so, and, platforms at Canva consists of a bunch of different areas, I guess most interesting today.
00:01:41.400 --> 00:01:50.340
that includes the teams that run our edge and gateway, the, front door into Canva, as the teams that, manage all of our cloud resources.
00:01:50.599 --> 00:01:54.920
So yeah, Canva can be, in the case you haven't heard of it, is a visual communication company.
00:01:55.489 --> 00:02:02.629
what we do is make it really easy for people to express their creative ideas, to others, in a way using visual content.
00:02:03.319 --> 00:02:07.519
we have a bunch of different products, built into what we call the, editor.
00:02:08.300 --> 00:02:12.620
so we have whiteboards and presentations and responsive docs and websites.
00:02:13.189 --> 00:02:15.949
and drawing tools for making social media posts.
00:02:16.639 --> 00:02:18.560
so yeah, there's a very, there's a huge amount you can do.
00:02:20.344 --> 00:02:20.555
Yep.
00:02:20.585 --> 00:02:23.405
That's how I'm gonna make the social media posts for this podcast.
00:02:23.405 --> 00:02:25.594
So, thank you all very much at Canva.
00:02:28.115 --> 00:02:33.094
that's why I was very excited not for you really, but when I saw this incident report, I.
00:02:33.810 --> 00:02:34.560
go by.
00:02:35.159 --> 00:02:40.919
I'm in a Slack group, this resilience and software group, and we have a channel that's just called cases.
00:02:41.490 --> 00:02:44.189
and that's, honestly, I just like, I could sit and watch that all day.
00:02:44.189 --> 00:02:47.280
Like, and so whenever, you know, sometimes we know because things happen to us.
00:02:47.280 --> 00:02:50.699
But I found out about this, this incident report and the outage.
00:02:50.729 --> 00:02:52.409
I did not experience the outage personally.
00:02:53.069 --> 00:02:55.949
but I did find out about the incident report from the Slack group that I'm in.
00:02:56.250 --> 00:03:03.270
And I think also I noticed, That it's the first one you've ever done, right?
00:03:03.300 --> 00:03:07.259
I don't recall ever reading a Canva incident report before.
00:03:08.729 --> 00:03:09.330
So.
00:03:09.810 --> 00:03:33.090
So you've had, and in the report, this report, I think you say you've had an internal process since we're gonna get into the incident, but you've had an incident process since 2017, and I'm always so curious about what it is that pushes an organization over the edge or, I mean, I don't want it to sound so negative, but like what brought you to the point where you said, okay, we're gonna go to the effort of publishing our first public incident report.
00:03:33.389 --> 00:03:33.689
Yeah.
00:03:33.689 --> 00:03:35.550
And it is a bit more effort, right?
00:03:35.759 --> 00:03:40.050
we've been publishing, incident reports internally, like you said, for many, years.
00:03:40.889 --> 00:03:46.379
but there is a difference between, a report for internal consumption and then a polished report for external consumption.
00:03:47.189 --> 00:03:52.409
so in terms of the why behind it, It's really a reflection of how we're evolving as a company.
00:03:52.800 --> 00:03:58.590
so Canva, stuck or got its start, with a bunch of, individuals and small businesses.
00:03:58.830 --> 00:04:01.560
that was typically the, user groups.
00:04:01.949 --> 00:04:07.770
and they'd, be creating social media content and marketing, for their small businesses, and that, that sort of material.
00:04:08.340 --> 00:04:15.000
but in the last couple of years, we've really seen increasing adoption within enterprises, and enterprises bring their own.
00:04:15.479 --> 00:04:19.439
Like new and different set of customer requirements, And customer expectations.
00:04:20.310 --> 00:04:26.490
and so it is part of us wanting to show our commitment to transparency, to those enterprise customers.
00:04:27.220 --> 00:04:34.379
and, also because we believe, and I very much believe, that sort of being open about incident reports does benefit the broader industry.
00:04:35.540 --> 00:04:35.839
and I think
00:04:36.529 --> 00:04:37.074
I agree.
00:04:37.100 --> 00:04:37.699
important.
00:04:37.819 --> 00:04:38.120
Yeah.
00:04:38.509 --> 00:04:41.329
and I, love, I, love what, the VOID is doing here.
00:04:42.589 --> 00:04:49.250
my, my take is that I think software engineering generally as a field does a fairly poor job at learning from previous failures.
00:04:49.790 --> 00:04:51.379
if you look at other engineering fields.
00:04:52.160 --> 00:04:59.959
you often they're like more highly regulated and there's like processes around this where, where, failures are like understood and investigated.
00:05:00.740 --> 00:05:05.779
but in software engineering, we, seem to make the same mistakes, many times, right?
00:05:05.779 --> 00:05:09.980
And there's not that, that not, that's not, that's learning, as a broader field.
00:05:10.699 --> 00:05:17.329
and so yeah, very keen on, on like voids efforts to see, improved education around this, right?
00:05:17.329 --> 00:05:19.790
So that we can all, uplift the industry as a whole.
00:05:21.589 --> 00:05:22.910
Thank you for the shameless plug.
00:05:22.910 --> 00:05:23.480
I appreciate it.
00:05:24.319 --> 00:05:29.209
I wanna ask you the same question I ask everyone at the start of this podcast, which I think is cruel, and yet I do it every time.
00:05:29.209 --> 00:05:31.240
Can you give a brief summary of what happened?
00:05:33.199 --> 00:05:34.550
I will, try and keep it brief.
00:05:35.750 --> 00:05:38.449
the a as sure you are aware, right?
00:05:39.889 --> 00:05:45.259
The, large and interesting, are, because of a confluence of factors, right?
00:05:45.259 --> 00:05:51.949
So in this particular case, they were, there were multiple contributing factors that sort of lined up in a way that caused this much, much, larger incident.
00:05:52.699 --> 00:05:54.019
But maybe a little bit of background.
00:05:54.439 --> 00:05:56.779
so the Canva editor is a single page app.
00:05:57.545 --> 00:05:59.134
we deploy that multiple times a day.
00:05:59.915 --> 00:06:07.685
and amongst other things, as part of that deploy, we, build the JavaScript assets, and then we publish those into an S3 bucket.
00:06:08.314 --> 00:06:17.644
and so then when clients, reload the page, or the editor self updates, those clients then go and, download the new assets, via our edge provider.
00:06:18.605 --> 00:06:26.105
and once those assets are loaded, and the editor is like functional and running, then it starts making API calls to, to load in all the content.
00:06:27.875 --> 00:06:32.225
API calls also go via our edge provider, to what we call our Canva API Gateways.
00:06:32.855 --> 00:06:39.845
and then those gateways route those requests to the various services, within, our systems, which then will, handle them.
00:06:40.444 --> 00:06:46.985
and I guess the, other key bit of information is those gateways run as auto scale groups, within our cloud provider, AWS.
00:06:48.470 --> 00:07:01.250
so in this incident, the first, the first like factor that contributed towards it was we did a deployment of the editor, at the same time that they were network issues that were occurring within our Edge provider.
00:07:01.790 --> 00:07:04.250
and now normally, these network happen.
00:07:04.310 --> 00:07:05.660
Issues happen all the time, right?
00:07:05.660 --> 00:07:07.160
Like the internet is a messy place.
00:07:07.790 --> 00:07:08.899
this is happening day in
00:07:09.589 --> 00:07:09.680
Yep.
00:07:10.475 --> 00:07:17.884
And so normally what happens is our Edge provider has automated software, that can detect these issues and can mitigate them and route around them.
00:07:18.694 --> 00:07:28.685
unfortunately in this particular case, and again, we only learned this later, they had a bit of manual stale configuration in place that have prevented that automation from running.
00:07:29.464 --> 00:07:34.834
so first contributing factor is there's a network issue, and it hasn't been mitigated, automatically.
00:07:37.285 --> 00:07:40.430
As, as often happens with those things, unfortunately.
00:07:40.430 --> 00:07:40.790
Yeah.
00:07:41.470 --> 00:07:41.689
Yep.
00:07:42.319 --> 00:07:48.889
And so then maybe the next bit to, to discuss is if you consider how the Edge proxies work, right?
00:07:48.889 --> 00:07:52.069
You've got a lot of clients, making requests for resources.
00:07:52.699 --> 00:07:57.290
and if that resource isn't in the local case, if the proxy, it all has to go and fetch it from the origin server.
00:07:58.399 --> 00:08:02.959
and it's very likely that there's multiple clients requesting the same bit of content, right?
00:08:03.620 --> 00:08:09.649
and pretty inefficient to have the, or the proxy server then go and turn around and request multiple times from the origin.
00:08:09.740 --> 00:08:16.579
So what you wanna do is you want to coalesce all those requests, do a single origin fetch, and then send that content back out to all the clients.
00:08:17.839 --> 00:08:23.600
and you can probably see where this is going, but, with a bunch of clients requesting the same JavaScript asset,
00:08:25.100 --> 00:08:29.389
Is that the sound, is that like the sound of a thundering herd I hear in the distance?
00:08:29.870 --> 00:08:31.519
yeah, that's, yeah.
00:08:31.564 --> 00:08:42.695
but the interesting bit here was if everything behaved as it was, as, it would normally, latency would go up because the origin fest fetches are taking a bit longer.
00:08:44.014 --> 00:08:45.034
and you know that, that's fine.
00:08:45.365 --> 00:08:47.495
it's just a bit of increased latency for the clients.
00:08:48.514 --> 00:08:56.894
but where we got really unlucky was one of those origin fetches didn't just go out from measured in milliseconds to seconds.
00:08:56.985 --> 00:08:59.445
The origin fetch actually took 20 minutes to occur.
00:09:00.254 --> 00:09:03.735
And, as I'm sure 20 minutes in computing time is an
00:09:04.539 --> 00:09:06.365
It is like geological time.
00:09:06.394 --> 00:09:08.914
Yeah, it's, yeah, it's geologic time, so yeah.
00:09:08.914 --> 00:09:09.664
That's rough.
00:09:09.774 --> 00:09:18.085
And then when we got particularly lucky again, was that this one, one particular asset, was the JavaScript that loads, what we call the object panel in Canva.
00:09:18.774 --> 00:09:23.575
so this is the sort of main place where you interact with content, in your designs, right?
00:09:23.575 --> 00:09:28.075
And so without the object panel loading, the editor is, essentially, frozen and dead.
00:09:28.855 --> 00:09:34.764
and what it also means is that once that one bit of JavaScript loads, it then triggers a bunch of API calls because
00:09:35.539 --> 00:09:35.870
Yeah.
00:09:36.595 --> 00:09:37.570
yes, if you've got.
00:09:38.044 --> 00:09:39.455
A fetch taking 20 minutes.
00:09:40.205 --> 00:09:42.875
got all these clients like coalescing and waiting for it.
00:09:43.294 --> 00:09:44.945
and then suddenly it becomes available.
00:09:45.394 --> 00:09:50.554
that turns into a large number of a API requests, hitting the gateways.
00:09:52.235 --> 00:09:52.745
Yeah.
00:09:53.825 --> 00:09:54.754
it is pretty ugly.
00:09:55.044 --> 00:09:58.764
At that point things fell over in terms of like the autoscaling and whatnot, right?
00:09:58.764 --> 00:10:02.004
At some point, it all just like collapsed on top of itself there, right?
00:10:02.695 --> 00:10:03.054
Yeah.
00:10:03.054 --> 00:10:10.254
And so that turned into, about, I think it was about one and a half million, API requests in a very short amount of time.
00:10:11.335 --> 00:10:13.705
and that was enough to, yeah, overwhelm the gateways.
00:10:13.705 --> 00:10:15.654
Auto scalers can't react fast enough to that.
00:10:16.284 --> 00:10:20.695
gateways went out of memory, and then that started the cascading failure due to overload.
00:10:22.690 --> 00:10:23.139
Okay.
00:10:23.980 --> 00:10:27.429
That's the shortest version of this incident, which is hard.
00:10:28.690 --> 00:10:31.870
and so I think, we'll, we'll talk a little bit more about some of these details.
00:10:32.139 --> 00:10:34.720
but how did you all find out about this?
00:10:35.899 --> 00:10:36.815
I can run you through that.
00:10:37.595 --> 00:10:47.254
so the, first signs that something was wrong, was about 10 minutes after the deploy, because we started noting, noticing a drop in search traffic.
00:10:47.945 --> 00:10:51.365
and so our search on call team was paged, for a drop in traffic.
00:10:51.365 --> 00:10:53.105
I think they went down around 20% or so.
00:10:54.080 --> 00:10:55.639
and so they started investigating.
00:10:56.539 --> 00:11:01.429
and at first they thought oh, maybe this is something like very isolated to search.
00:11:03.455 --> 00:11:18.004
as they started, communicating in an incident room, it was a sev one, which is our, second highest tier of, incident At that point, as they started communicating, other service owners started getting involved and saying oh, actually, like we've also noticed a drop in traffic.
00:11:19.254 --> 00:11:23.754
And as they were doing that, that was about the time that the fetch itself completed.
00:11:24.325 --> 00:11:28.794
and then our gateway and edge teams got paged for, massive gateway failures.
00:11:29.514 --> 00:11:31.764
at that point, the incident was upgraded to a sev zero.
00:11:32.004 --> 00:11:33.445
The gateway teams were pulled in.
00:11:33.774 --> 00:11:38.605
I was pulled in and we, the incident, incident coordinator was, activated.
00:11:40.284 --> 00:11:41.215
and then, we went from there.
00:11:41.519 --> 00:11:41.639
Yeah.
00:11:41.639 --> 00:11:44.669
What is your incident response look like?
00:11:44.669 --> 00:11:51.480
How, you know, do you all have sort of some, you know, protocols or what does that look like and how, how many people ended up being involved?
00:11:51.480 --> 00:11:55.230
If you can tell me, because it sounds like it was a doozy, and a lot of people got called in.
00:11:55.230 --> 00:11:57.690
I'm always really curious, like how many hands are on deck?
00:11:58.514 --> 00:12:03.059
Yeah, I would guess probably at least 20 or so.
00:12:04.169 --> 00:12:06.269
so let me, describe the process.
00:12:07.754 --> 00:12:11.355
typically, alerts will page, service on call teams.
00:12:11.985 --> 00:12:13.845
I think it's a, very typical process, right?
00:12:13.894 --> 00:12:30.544
Those service teams will triage, an open incident, if it's a SEV zero or SEV one, that'll activate the incident coordinator rotation, which is a, it's a small group of people, that have a lot of training and skills around managing, and coordinating these large scale incidents.
00:12:31.355 --> 00:12:33.634
the Sev Zeros will page myself.
00:12:35.134 --> 00:12:40.565
and then it, really depends on the incident as to the structure of it, right?
00:12:40.565 --> 00:12:46.654
If it's a smaller incident, you might have only a handful of people, and they'll be fulfilling all the roles, right?
00:12:47.014 --> 00:12:53.710
if it's a large incident, like in this one, we'll have, we'll, set it up so that there's dedicated people running each particular role.
00:12:54.544 --> 00:12:56.914
I think maybe one, one thing that's important as well.
00:12:57.605 --> 00:13:06.845
Is that, the larger or the highest severity incidents, will automatically assign a representative from our customer support organization.
00:13:08.355 --> 00:13:08.894
Oh, okay.
00:13:09.605 --> 00:13:15.875
all of the user reports, and, looking for signal there to not overwhelm the, sort of the responding team.
00:13:16.235 --> 00:13:16.355
and
00:13:17.125 --> 00:13:17.544
Mm-hmm.
00:13:17.735 --> 00:13:21.815
the user support, so that they can feedback look, this is the state of things.
00:13:21.875 --> 00:13:25.625
If people are, contacting or reaching out to Canva, this is what we should be telling them.
00:13:25.625 --> 00:13:26.855
If there's anything they can do.
00:13:27.960 --> 00:13:31.049
to mitigate it from a user perspective, that as well.
00:13:32.580 --> 00:13:32.789
yeah.
00:13:32.789 --> 00:13:36.509
And then, post incident, like I mentioned, we'll write those incident reports.
00:13:36.559 --> 00:13:37.460
and then we also.
00:13:38.394 --> 00:13:49.105
we've just, recently started using AI to try and extract common themes, across all of those reports as well, to, look for areas where we can, where we can improve.
00:13:49.306 --> 00:13:50.475
I have so many questions.
00:13:51.196 --> 00:13:53.770
so the IC is getting pulled.
00:13:54.316 --> 00:13:58.725
Sort of automatically, if it's like sev one, sev zero, how is that team?
00:13:59.416 --> 00:14:04.096
Are they, are they, also engineers on, on different teams?
00:14:04.096 --> 00:14:08.115
Are they kind of a, a dedicated SRE esque function?
00:14:08.466 --> 00:14:10.745
And when they're not ic, what are, what are they doing?
00:14:10.745 --> 00:14:12.275
I'm kind curious what the shape of that looks like.
00:14:12.701 --> 00:14:12.931
Yeah.
00:14:12.931 --> 00:14:21.240
So early on, say our, ICS were just a bunch of very experienced, battle hardened engineers at Canva.
00:14:22.951 --> 00:14:28.921
and then over time, of course, as we get bigger, we need to set up a dedicated function.
00:14:29.610 --> 00:14:32.880
and those, because those, folks are also, they've got their own projects, right?
00:14:32.880 --> 00:14:35.490
They their own deliverables, that they're working on.
00:14:35.671 --> 00:14:38.071
So yeah, IC is a dedicated function at Canva.
00:14:39.030 --> 00:14:42.421
when they're not on call, they're doing a bunch of other different things.
00:14:42.916 --> 00:14:48.525
some of those is improving, the incident process itself, looking for those patterns in incidents.
00:14:49.275 --> 00:14:53.235
plus also there's a bunch of work that goes into our large, launch events.
00:14:53.265 --> 00:14:56.716
So we do two, two main launches a year, and we've got one coming up now in April.
00:14:57.525 --> 00:15:00.285
and so they're, like doing a bunch of planning for that.
00:15:00.765 --> 00:15:09.826
there's a bunch of capacity planning involved, like risk, un understanding the risk and mitigating, the risks that are unique to each one of those launches.
00:15:10.535 --> 00:15:13.385
And so yeah, that's a, that's like a dedicated reliability function.
00:15:13.633 --> 00:15:22.783
When you have like a pretty big incident like this one, do you have execs in the, in the channels, in the, active incident channels?
00:15:22.783 --> 00:15:27.403
You, or like, or do you have a dedicated role who's talking to folks and kind of coordinating back or forth?
00:15:27.793 --> 00:15:28.452
What does that look like?
00:15:29.123 --> 00:15:29.423
Yeah.
00:15:29.423 --> 00:15:33.442
So in this particular case, the founders, our founders, did join the incident channel.
00:15:33.442 --> 00:15:33.472
I.
00:15:34.618 --> 00:15:39.177
typically what will happen, sometimes they're there, sometimes they're, they're busy with other things.
00:15:40.138 --> 00:15:49.977
typically like myself or one of the other senior leaders will act as that channel, to the, founders, to give them updates and, help'em understand what's going on.
00:15:50.327 --> 00:16:00.227
So you mentioned you found out later, so was was the CloudFlare wrinkle of their traffic, and you didn't mention this explicitly, but I think this is what you're getting at with the network configuration.
00:16:00.227 --> 00:16:03.467
So their traffic was going over the public internet, not over their private backbone.
00:16:03.738 --> 00:16:05.508
Is that a piece of it that you found out later?
00:16:05.567 --> 00:16:09.107
And so that was sort of like, you didn't know why that was contributing to the problem.
00:16:11.207 --> 00:16:11.508
yes.
00:16:12.138 --> 00:16:15.977
we don't have visibility into that traffic between, cloud CloudFlare and
00:16:16.727 --> 00:16:17.028
Yeah.
00:16:17.207 --> 00:16:18.467
in A-W-A-W-S.
00:16:19.278 --> 00:16:26.837
so yes, that was a detail that we, didn't find out until, I think maybe one or two days after the incident as we were working with CloudFlare to understand what happened.
00:16:27.034 --> 00:16:31.865
At, at some point, did anyone have the intuition that this was gonna get worse in the way that it did?
00:16:32.735 --> 00:16:40.115
Like once the files were downloaded from the origin server and, and there was, did you see that, hear the thundering heard coming?
00:16:40.115 --> 00:16:49.804
Like, I'm always a little curious what the, what the existential dread might be like, like in an incident or was that, did that catch you by surprise as well?
00:16:50.134 --> 00:16:53.855
yeah, it all happened, like fairly quickly in terms
00:16:54.700 --> 00:16:54.990
Okay.
00:16:55.595 --> 00:16:56.705
the unfolding incident.
00:16:58.414 --> 00:17:06.394
yeah, by the time that folks were in the room saying oh, this is, broader than search, that was about the time that origin fetch completed.
00:17:06.815 --> 00:17:08.990
and then the gateway started starting, falling, over.
00:17:10.131 --> 00:17:16.280
and so that, so even so the, like the, all the clients hitting the API gateway, you've got a known performance issue.
00:17:16.280 --> 00:17:18.260
So that all happened really quickly.
00:17:18.560 --> 00:17:22.701
And the the, or were piece of those only aware after the fact as well.
00:17:23.826 --> 00:17:24.365
Yes.
00:17:24.365 --> 00:17:26.375
I didn't actually touch on the performance side.
00:17:28.550 --> 00:17:29.060
get into that.
00:17:29.060 --> 00:17:30.921
Then I jumped ahead.
00:17:32.405 --> 00:17:34.326
Gateways are now, now crashing.
00:17:35.421 --> 00:17:40.611
we have a bunch of people on the call and that's where we started breaking up into different work streams, right?
00:17:41.391 --> 00:17:46.790
there were a set of people that were responsible for, contact taking our vendors, and getting them involved.
00:17:47.240 --> 00:17:49.221
That process differs across vendors, right?
00:17:49.221 --> 00:17:54.621
Like some of them, will be like, we'll go to our portal and sevmit like a, a P zero request.
00:17:54.621 --> 00:17:59.810
Others will be like, email this address and it will page sort of your on-call account managers.
00:18:00.411 --> 00:18:04.280
and so that process there, needs to be well practiced, right?
00:18:04.280 --> 00:18:08.151
And I think this is actually something that, we improved afterwards as well.
00:18:08.570 --> 00:18:15.260
we have internal docs on how to escalate to each of our vendors, but we tied it up that documentation and just made it a bit clearer because when you are.
00:18:15.996 --> 00:18:19.086
you're doing that, you are already in a high stress environment, right?
00:18:19.086 --> 00:18:19.865
And so you don't want to
00:18:20.780 --> 00:18:21.411
Absolutely.
00:18:21.455 --> 00:18:23.736
and have four pages of workflow to go through.
00:18:23.736 --> 00:18:26.976
It needs to be a very clear, do these four things, right?
00:18:27.951 --> 00:18:28.340
Yeah.
00:18:28.445 --> 00:18:41.810
had a set of people contacting vendors, We had one of the engineers who was not on call at the time, but was, saw the incident and jumped on, as, many people at Canva do.
00:18:42.080 --> 00:18:48.711
We, there's a very good, culture of, banding together, especially for these large incidents.
00:18:49.340 --> 00:18:53.990
so we had one, one engineer who went off and started profiling the gateway.
00:18:55.411 --> 00:19:02.405
this is he was reporting back in saying oh look, actually, profile looks different.
00:19:03.336 --> 00:19:08.705
I think, if I remember correctly, actually had, he'd done this before, other times.
00:19:08.855 --> 00:19:11.435
And so he had past profiles to compare it to.
00:19:12.096 --> 00:19:17.316
and he was like, look, when I did this two weeks ago, this, one looks different, right?
00:19:17.316 --> 00:19:18.695
And I'm seeing lock contention here.
00:19:19.560 --> 00:19:24.631
And so that was, like one, one avenue that we thought, might have been the contributing factor.
00:19:25.141 --> 00:19:32.131
and so we had a bunch of people exploring that, that turned out to be, a change in our metrics library.
00:19:32.730 --> 00:19:36.270
what we were doing was changing, making some changes to the way that metrics were collected.
00:19:37.111 --> 00:19:40.681
and so we were integrating a, like a third party library there.
00:19:41.671 --> 00:19:45.990
that had done is, inadvertently put metric registration behind a lock.
00:19:46.500 --> 00:19:48.961
and so the capacity of our gateways were reduced.
00:19:49.730 --> 00:19:56.510
which would then also contribute to the sort of the overload situation, and what we saw with the cascading failure.
00:19:56.961 --> 00:20:00.830
but that was all emerging in parallel to the, to the,
00:20:00.891 --> 00:20:01.181
Yeah.
00:20:01.221 --> 00:20:02.631
activity that was happening on the call.
00:20:02.771 --> 00:20:06.967
So you got the performance issue, which must have been.
00:20:07.646 --> 00:20:10.676
And if I'm not wrong, there was this really juicy detail in there.
00:20:11.037 --> 00:20:21.446
these are the kinds of things that I love it when companies share, where it was a known issue and there was a fix for it in like the deployment pipeline for that day.
00:20:21.686 --> 00:20:23.636
That had not gone out yet.
00:20:23.817 --> 00:20:30.957
And in many organizations there could be a lot of blame and like post hoc, like, I can't believe we didn't do that.
00:20:30.957 --> 00:20:40.961
We should have, like what was the general sense when you all realized that that also was contributing to this incident?
00:20:41.811 --> 00:20:42.021
Yeah.
00:20:42.862 --> 00:20:45.231
definitely, there, there was never any blame.
00:20:46.372 --> 00:20:53.241
and it was just very unfortunate that yes, the metrics team had already identified this, they'd already fixed it, it was already merged.
00:20:54.412 --> 00:20:58.521
and, I guess what would that have been in, in 12 hours later?
00:20:59.092 --> 00:21:03.112
the new gateway would've been deployed and, this have very different.
00:21:04.406 --> 00:21:04.696
yeah.
00:21:04.821 --> 00:21:05.811
just getting unlucky.
00:21:07.102 --> 00:21:08.751
we do have a project underway.
00:21:10.106 --> 00:21:12.896
to, move to incremental deploys.
00:21:13.346 --> 00:21:25.467
right now I'd say most backend components, are deployed on a daily cycle, and then, and then the components individually can opt into incremental deploys, and deploy on merge.
00:21:25.856 --> 00:21:35.007
but we do want to move more and more of our components to like over to that incremental model, which would've, ideally caught this, in this case.
00:21:35.183 --> 00:21:48.443
And so then the last, but not least, certainly I believe, again, if I've got the timeline right here, so you've got that going on, and then basically you had like the load balancers on your ECS containers, just they couldn't.
00:21:48.959 --> 00:21:51.239
Like they were just getting hammered at this point as well.
00:21:51.239 --> 00:21:51.568
Right.
00:21:51.568 --> 00:22:01.288
And was that also, a matter of sort of, this was reaching out and trying to figure out what was happening with AWS or like what, what was that last piece of the puzzle like?
00:22:01.288 --> 00:22:01.348
I.
00:22:02.659 --> 00:22:07.068
Yeah, so we did, we had a set of people, escalating CloudFlare.
00:22:07.308 --> 00:22:16.068
we had a set of people reaching out to AWS because, we were at that point, this was just after that Origin Fetch had completed.
00:22:17.669 --> 00:22:27.148
we were concerned that there might be something going on the AWS side, but thankfully, we have, I would say like reasonably good observability into all of this, right?
00:22:28.048 --> 00:22:34.919
and a cascading failure due to overload is a very distinct pattern in a graph, right?
00:22:34.919 --> 00:22:37.648
Where you get this, sawtooth pattern, like if you're looking at
00:22:38.364 --> 00:22:38.653
Yeah.
00:22:38.788 --> 00:22:40.439
uptime, you get this sawtooth.
00:22:40.598 --> 00:22:42.429
That sort of offset with every instance.
00:22:43.088 --> 00:22:48.249
and I've certainly seen that before, a number of other people on the call had seen that before.
00:22:48.729 --> 00:22:50.108
and so it was very clear at that point.
00:22:50.108 --> 00:22:53.078
It's oh no, this is, this is a, an overload situation.
00:22:53.888 --> 00:22:58.118
and then, a again, we've, faced this before, right?
00:22:58.989 --> 00:23:00.219
I've, seen, quite a few of these.
00:23:00.969 --> 00:23:08.199
and there's, really only one option, which is you've got to, bring the load down and get the system back into a stable state, right?
00:23:08.828 --> 00:23:10.209
You can, try and add capacity.
00:23:10.433 --> 00:23:19.193
The problem is un unless you can do that in a, an atomic flip fashion, any new additional capacity that you bring up is just gonna get hammered into the ground.
00:23:19.644 --> 00:23:21.203
and so you've gotta cut that load back.
00:23:22.534 --> 00:23:22.798
In this
00:23:22.834 --> 00:23:23.124
Yeah.
00:23:23.219 --> 00:23:33.959
case, you know what would be really nice if is if we had a sort of a, lever where we could just say oh, okay, great, like dial the load to 20% of demand.
00:23:34.679 --> 00:23:38.398
we didn't have that sort of control within the cloud providers systems.
00:23:38.848 --> 00:23:44.219
so instead what we did was reach for, the, sort of country level controls, right?
00:23:44.219 --> 00:23:53.969
And so we put in a block that said, For any country in the world, want to display this status message instead of forwarding the requests onto the, backend.
00:23:54.148 --> 00:23:56.969
So that was a, that, that was the lever that we had at the time.
00:23:58.169 --> 00:23:58.288
and
00:23:58.288 --> 00:23:58.739
Yeah.
00:23:59.038 --> 00:24:02.278
doing was, once, once we got that block in place, I.
00:24:02.443 --> 00:24:06.193
We saw the traffic drop to zero, we saw all the gateways come up and stabilize.
00:24:06.763 --> 00:24:08.534
and then we started admitting more load.
00:24:08.983 --> 00:24:17.354
I think the, maybe the interesting bit here is that, we added Europe back in first because that was where the peak load of the time was, right?
00:24:17.354 --> 00:24:23.114
This happened in Australian, like early to mid evening time, would've been European daytime.
00:24:23.144 --> 00:24:25.304
The US was mostly asleep at that point.
00:24:25.844 --> 00:24:31.663
and so we Europe to be admitted back first, and then, rolled out, out to the rest of the world.
00:24:33.479 --> 00:24:48.719
Yeah, I mean that was an interesting piece of what I like to think of as hard earned expertise where instead of just turning everything back on again and literally recreating potentially the same problem, which I've definitely seen incidents where they're like, oh no.
00:24:48.719 --> 00:24:50.519
And it like literally happens again.
00:24:51.282 --> 00:24:57.163
the only other piece I was curious about is in terms of what that incident response looks like, is like, are people remote?
00:24:57.432 --> 00:24:59.202
Is everything in a Slack channel?
00:24:59.718 --> 00:25:00.377
How does that work?
00:25:00.377 --> 00:25:01.758
Is everybody in the same time zone?
00:25:02.688 --> 00:25:04.127
How complicated does that look for you all?
00:25:04.738 --> 00:25:04.968
Yeah.
00:25:04.968 --> 00:25:08.018
Canva is a hybrid work setup.
00:25:08.798 --> 00:25:14.798
so I'd say the bulk of our engineering is in the New Zealand to West coast of Australia.
00:25:15.188 --> 00:25:15.728
Time, zone.
00:25:17.738 --> 00:25:20.948
and so yeah, this was like early to mid evening for them.
00:25:22.327 --> 00:25:23.557
but yes, as part of.
00:25:24.597 --> 00:25:25.347
The automation.
00:25:25.347 --> 00:25:28.917
When an incident is triggered, it automatically creates a slack room.
00:25:29.218 --> 00:25:36.268
it ties that slack room to the main, incident slack room, so people, can go in there and find it, and it sets up the Zoom call as well.
00:25:37.077 --> 00:25:40.617
and so then I'd say the majority of the discussion occurs on Zoom.
00:25:41.008 --> 00:25:47.998
maybe another interesting point is that we, have deliberately set up that zoom in a way that, that Zoom chat is disabled.
00:25:48.448 --> 00:25:50.488
so it funnels all of the chat through the slack room.
00:25:50.817 --> 00:25:50.998
so
00:25:51.798 --> 00:25:52.367
Ah, okay.
00:25:52.488 --> 00:25:52.758
Yeah.
00:25:52.768 --> 00:25:54.478
I think that you don't have two, two different,
00:25:55.877 --> 00:25:57.768
Two sources of, yeah.
00:25:57.978 --> 00:25:58.307
Yeah.
00:25:58.768 --> 00:26:05.367
and so typically people will share, will be sharing links to dashboards or logs or et cetera in the Slack channel.
00:26:06.357 --> 00:26:20.337
and then we have a way of annotating within the Slack channel sort of key moments, in a way that it's very easy to, when you're going back later trying to write that incident report, you can pull out the key events and say okay, this is what we knew at this particular time.
00:26:20.798 --> 00:26:26.917
homegrown tool that you all use, or is that a, that's a third party tool for the, for the incident sort of.
00:26:28.042 --> 00:26:33.667
yeah, I think it's mostly a bunch of, a bunch of tools developed by that same set of, incident coordinators.
00:26:33.972 --> 00:26:39.518
So some companies say, you know, for, for, I.
00:26:39.837 --> 00:26:46.228
Sev whatever number, we're gonna always have an incident review and we're gonna try to do it within this amount of time.
00:26:46.768 --> 00:26:56.577
and do you have like a dedicated incident analysis team and with an analysts, and do they have, processes around that and, and timeframes for when you want things to happen?
00:26:56.577 --> 00:26:57.867
Or is that more ad hoc?
00:26:58.553 --> 00:27:02.063
Yeah, so we don't have a dedicated team of people analyzing these.
00:27:02.722 --> 00:27:05.063
the incident coordinators will work with the.
00:27:05.353 --> 00:27:08.833
The sort of service teams that were contributing towards the incident.
00:27:09.702 --> 00:27:13.692
and then we have a, a template for, the, PIRs, what we call them.
00:27:15.522 --> 00:27:19.093
I don't, I, don't think there is a sort of a set deadline.
00:27:19.813 --> 00:27:20.442
but certainly I.
00:27:21.292 --> 00:27:26.482
myself, I'll be, because I'll be involved in the Sev Zeros I'll be looking for that report.
00:27:27.442 --> 00:27:33.772
and if it hasn't shown up in maybe two, two or three weeks, I'll be, ping people saying where, is it at that point?
00:27:34.792 --> 00:27:38.272
no sort of system that, you
00:27:39.063 --> 00:27:39.353
Yeah.
00:27:39.357 --> 00:27:39.597
Yeah.
00:27:40.073 --> 00:27:43.673
it is all linked in the, in that automation, in that slack room as well.
00:27:44.692 --> 00:27:45.502
as the state.
00:27:45.548 --> 00:27:48.218
Moves through from, like responding.
00:27:48.488 --> 00:27:53.798
so I guess like triage, responding to mitigated, to resolved, that's all as
00:27:53.817 --> 00:27:54.147
Yeah.
00:27:54.788 --> 00:27:55.988
and, fed into that timeline.
00:27:57.557 --> 00:28:01.248
Is there like a PIR meeting anyone can come to?
00:28:01.278 --> 00:28:03.208
Like how are the reports shared internally?
00:28:04.218 --> 00:28:07.968
Yeah, so it's, it's then published when it is published.
00:28:08.208 --> 00:28:10.998
it's done typically is like the last message in that Slack room.
00:28:11.807 --> 00:28:13.157
that, that then gets archived.
00:28:13.728 --> 00:28:24.528
we have a weekly meeting as well where we review, the Sev Zeros plus, like of the, other, lesser, severity, but maybe like interesting, failure modes.
00:28:25.488 --> 00:28:30.048
and then we publish, in our sort of monthly engineering leads meeting.
00:28:31.307 --> 00:28:35.117
we, take the, list of incidents, and we go through them briefly there.
00:28:35.867 --> 00:28:42.107
that's so that all the engineering leads have the same level of visibility, and that we're identifying common themes.
00:28:42.747 --> 00:28:50.362
Do you find that the, that those reports get referenced in like planning or are they, are they used.
00:28:50.916 --> 00:28:52.866
Do you see them being used after the fact?
00:28:53.166 --> 00:29:02.106
you know, as sort of learning pieces in terms of either yes, for planning or for architecture reviews or like any of those kinds of things.
00:29:02.406 --> 00:29:10.176
Yeah, so the action items that come out of those reports, get created in Jira, which will then be sitting on those teams backlogs.
00:29:10.866 --> 00:29:17.527
and so then, next, next planning iteration, the team will be looking at, looking at their sort of like reliability backlog.
00:29:18.172 --> 00:29:27.501
and then like mixing that with the other sort of various categories of work on the backlog, and then using that to prioritize, what, what, work occurs.
00:29:27.846 --> 00:29:32.676
And is there, is there anything outside of action items in terms of like distributing what you learned?
00:29:33.307 --> 00:29:36.866
or is there any other activity around the incidents like that?
00:29:38.017 --> 00:29:42.547
I'd say no, other than the, sort of internal Slack channels where,
00:29:43.386 --> 00:29:43.596
Yeah.
00:29:43.626 --> 00:29:43.836
right?
00:29:43.836 --> 00:29:48.682
And so anyone in the company is free to and look at those, and, read through the reports.
00:29:48.957 --> 00:29:51.457
What was the most surprising thing to you about this incident?
00:29:52.734 --> 00:29:59.365
yeah, probably the biggest one is that our edge provider would coalesce requests indefinitely.
00:30:01.375 --> 00:30:06.505
I can, understand, I can understand why that was the outcome.
00:30:07.075 --> 00:30:11.335
And I can imagine, I can put myself in the person writing that code.
00:30:11.994 --> 00:30:17.875
and it's oh, like if, fetch in flight, wait for it to complete and then respond, right?
00:30:19.795 --> 00:30:22.404
that was what allowed that thundering herd to develop.
00:30:23.664 --> 00:30:24.234
and I.
00:30:25.210 --> 00:30:31.660
all of our internal RPC traffic has timeouts, which are propagated and respected at each hop in that service graph.
00:30:33.519 --> 00:30:48.309
in this particular case, there wasn't, like any, timeout within that, within that edge providers, layer, which makes it, somewhat difficult from a client code perspective.
00:30:48.684 --> 00:30:51.924
To understand, is this an error?
00:30:52.734 --> 00:30:58.194
is this just like a slow fetch, or is this I part of a thundering herd?
00:30:58.404 --> 00:30:58.704
In which
00:30:59.440 --> 00:31:00.339
Hey, there's
00:31:00.744 --> 00:31:02.845
would wanna do something differently, add
00:31:03.549 --> 00:31:04.180
right.
00:31:04.704 --> 00:31:05.005
Yeah.
00:31:05.457 --> 00:31:16.977
Yeah, it definitely seemed like one of the themes from this incident as well was, and I, I mean I see this a lot and I write about it a lot, was how automation actually made it worse.
00:31:17.156 --> 00:31:17.576
Right.
00:31:18.626 --> 00:31:24.057
and not because you're like, we want the automation to make things worse, but because it, it can't always.
00:31:24.461 --> 00:31:31.001
React to these kinds of situations and you know, like it's model of the world is not as flexible as ours is.
00:31:31.721 --> 00:31:44.112
and so, you know, I, I definitely saw like action items out of that and everything in the report, but was there, what, was there any reaction to that in terms of like where you had these.
00:31:44.727 --> 00:31:58.247
Automation surprises of, of, and not just like, what was your response to the sort of automation, making it harder and, and how are you all adapting to that after the fact for the future?
00:31:58.967 --> 00:32:10.811
Yeah, so maybe a good example is, again, I didn't, mention this, but Part of what happened on that call, was there was a person responsible for freezing the auto scaling.
00:32:11.757 --> 00:32:15.146
because what can happen here is, your traffic go goes to zero.
00:32:15.656 --> 00:32:20.757
your auto, especially if that's, like over a, sort of like long amount of time.
00:32:21.176 --> 00:32:22.527
the auto scalers are like, oh, great.
00:32:22.527 --> 00:32:26.037
Like we can like down downscale all of the groups, and
00:32:26.731 --> 00:32:27.152
Yeah.
00:32:27.176 --> 00:32:27.656
course they can't
00:32:28.352 --> 00:32:29.612
And like, oh no again.
00:32:30.031 --> 00:32:30.362
Yeah.
00:32:30.541 --> 00:32:42.122
So we've got processes in place where, you know, for those particular types of incidents, we have tooling that we can run, again, developed, by those reliability folks where we can say hey, freeze all the auto scalers.
00:32:42.811 --> 00:32:50.701
and what we can do also is, go back and say oh, actually, set them, reset them to where they were like prior to the incident occurring, right?
00:32:50.701 --> 00:32:53.912
So if, any have down downscaled, we can get them back.
00:32:54.362 --> 00:32:57.152
and so that's a, process that we've adapted.
00:32:58.396 --> 00:33:09.946
given past experience and thinking through what's, like bad outcomes that could happen in these sort of event events that, turns the automation off for a second and humans, take control.
00:33:09.946 --> 00:33:10.007
Yeah.
00:33:11.747 --> 00:33:19.461
Yeah, I mean that's definitely a theme of, of obviously my work, but like in, instead of just saying, well, we'll add more automation to make the automation.
00:33:19.636 --> 00:33:20.057
Better.
00:33:20.267 --> 00:33:26.656
Y'all were like, well, let's give the humans better controls over what that might look like.
00:33:26.656 --> 00:33:29.777
I like, I like seeing those kinds of, of adaptations, so
00:33:30.011 --> 00:33:36.461
like I'm, I'm a very big fan of modes within automated software.
00:33:36.836 --> 00:33:44.217
Where it just throws up its hands and it says, this, does not match my, view, like my, what I've been modeled for and
00:33:44.951 --> 00:33:45.241
yeah.
00:33:45.717 --> 00:33:45.987
right?
00:33:46.586 --> 00:33:51.926
when I've been building systems in the past, I've built in these, lockdown modes, right?
00:33:51.926 --> 00:33:54.567
Where it's hey, like the inputs don't match anymore.
00:33:54.567 --> 00:33:55.886
I'm gonna go into lockdown mode.
00:33:55.946 --> 00:33:57.537
I'm gonna flag it in my telemetry.
00:33:58.596 --> 00:34:00.576
human come and get me out of this, right?
00:34:00.576 --> 00:34:09.456
Because rather than just like continuing down a path that sort of, no one has prepared, the software for, and, possibly making things much, much worse.
00:34:09.817 --> 00:34:15.146
And what do you think was the biggest learning for the team out of this incident?
00:34:15.483 --> 00:34:21.302
I think definitely practicing, those controls, on the Edge Providers network.
00:34:22.413 --> 00:34:23.827
when we were putting that block in place.
00:34:25.487 --> 00:34:28.097
we, added it with a very simple status message.
00:34:28.427 --> 00:34:32.538
I can't actually remember what it said, but it was definitely not a user friendly status message.
00:34:33.708 --> 00:34:34.577
because at the,
00:34:35.507 --> 00:34:35.967
All bad.
00:34:35.967 --> 00:34:36.487
Good luck.
00:34:36.902 --> 00:34:37.202
yeah.
00:34:37.853 --> 00:34:39.257
at the, wasn't quite that bad.
00:34:40.097 --> 00:34:42.378
but because at the time, at the
00:34:43.307 --> 00:34:46.907
I could come up with a lot of bad error messages for you if you'd like, but probably not.
00:34:47.628 --> 00:34:48.168
Yeah.
00:34:49.217 --> 00:35:00.168
so, we, because at the time the priority was like, get the load down rather than, debate the error message or, like how it was, gonna be visually presented, et cetera.
00:35:01.532 --> 00:35:01.822
Yeah.
00:35:02.327 --> 00:35:03.047
we've learned from that.
00:35:03.047 --> 00:35:06.498
We now have canned response ready to go, in a much more.
00:35:06.978 --> 00:35:09.168
user friendly and visually appealing style.
00:35:10.128 --> 00:35:12.978
and so that, yeah, that's, like one, one thing that's come outta that.
00:35:13.697 --> 00:35:19.938
also our, ability to practice using those controls and getting more familiar with them.
00:35:20.447 --> 00:35:25.967
cause you, really don't want to be using something for the first time, in a high pressure environment, right?
00:35:25.967 --> 00:35:30.483
Like you want to have this, know, built up almost in your muscle memory.
00:35:30.932 --> 00:35:43.293
one of the other, like general principles that I try and describe to when, when building these systems is that any sort of emergency or failure mode, is exercised day to day.
00:35:43.802 --> 00:35:45.782
And that way, that your
00:35:46.038 --> 00:35:46.217
Yep.
00:35:46.353 --> 00:35:46.713
for it.
00:35:46.713 --> 00:35:49.862
You've been using it, day to day or once a week.
00:35:50.012 --> 00:35:53.492
And it's not a, oh, I've gotta do this first, first.
00:35:53.853 --> 00:35:54.362
this thing.
00:35:54.393 --> 00:35:56.103
First time in a call.
00:35:57.018 --> 00:36:01.007
there's like a documentation page that is like four pages long, right?
00:36:01.007 --> 00:36:01.248
And I've
00:36:02.068 --> 00:36:02.327
Yeah,
00:36:02.538 --> 00:36:03.588
of steps perfectly,
00:36:04.427 --> 00:36:09.918
oven's on fire and I've never used the like fire extinguisher thing before and it can't get the Yeah.
00:36:09.918 --> 00:36:10.307
Basically.
00:36:10.307 --> 00:36:10.487
Yep.
00:36:10.608 --> 00:36:10.847
Yep.
00:36:11.657 --> 00:36:12.708
And do you all do.
00:36:13.458 --> 00:36:13.518
Yeah.
00:36:14.268 --> 00:36:20.628
Do you do drills like tabletop drills, chaos engineering type stuff, or what does that preparedness look like?
00:36:20.748 --> 00:36:23.507
Is how is that structured and definitely happens?
00:36:24.228 --> 00:36:27.257
I'd say there's a, variety of drills that occur, right?
00:36:28.068 --> 00:36:34.248
like teams themselves will do, will do wheel of misfortune style or like role playing incidents.
00:36:35.177 --> 00:36:44.637
but then there's also like larger drills that we do as a company, where we say oh, okay, yes, we're going to, we're gonna do like a sort of a, business continuity drill.
00:36:45.788 --> 00:36:48.978
will get involved and, do those exercises.
00:36:49.202 --> 00:36:54.873
Thank you so much for joining me and sharing your internal process with the world.
00:36:54.932 --> 00:37:02.463
And while I hope you don't have, more sev zero incidents, I do hope that you continue to share your incident reports with us.
00:37:02.463 --> 00:37:07.773
And I, I really appreciate all of the time and effort and cat herding that goes into that.
00:37:08.163 --> 00:37:13.112
but as you said, it really is so beneficial to everyone else in the industry, so thank you for doing it.
00:37:13.592 --> 00:37:14.193
Yeah, no worries.
00:37:14.342 --> 00:37:19.293
Yeah, I just, I hope people can, learn from it and we can get better, everyone together.
00:37:21.003 --> 00:37:21.543
Absolutely.
00:37:21.992 --> 00:37:22.233
Okay.
00:37:22.322 --> 00:37:23.043
Thanks so much.
00:37:23.733 --> 00:37:24.123
very much.
