TRANSKRIP 🔗
WEBVTT
00:00:00.000 --> 00:00:02.980
Webhooks might be the simplest distributed system
00:00:02.980 --> 00:00:06.400
most teams underestimate. You give a service
00:00:06.400 --> 00:00:10.859
a URL. It sends an HTTP POST when something happens.
00:00:11.179 --> 00:00:15.560
Done. Until your endpoint is down during a deploy.
00:00:15.919 --> 00:00:19.699
Or the sender retries and you process the same
00:00:19.699 --> 00:00:23.559
event twice. Or events arrive out of order. Or
00:00:23.559 --> 00:00:26.420
a valid request triggers something it should
00:00:26.420 --> 00:00:30.079
not. Or you discover the provider stopped retrying
00:00:30.079 --> 00:00:33.820
hours ago and nobody noticed. Suddenly, that
00:00:33.820 --> 00:00:38.079
simple HTTP POST needs queues, idempotency,
00:00:38.240 --> 00:00:42.000
signature verification, replay, observability,
00:00:42.320 --> 00:00:45.780
retention, and someone willing to own it when
00:00:45.780 --> 00:00:49.039
it breaks. That is the strange thing about webhooks.
00:00:49.039 --> 00:00:52.259
The interface is simple. The operational contract
00:00:52.259 --> 00:00:55.479
is not. I'm Brian Teller from Teller's Tech.
00:00:55.799 --> 00:01:15.819
And this is Ship It Weekly. Welcome back to Ship
00:01:15.819 --> 00:01:18.280
It Weekly, where I filter the noise and focus
00:01:18.280 --> 00:01:20.680
on what matters when you are the one running
00:01:20.680 --> 00:01:23.659
infrastructure and owning reliability. Most weeks,
00:01:23.719 --> 00:01:28.000
it's a quick DevOps, SRE, platform, cloud, and
00:01:28.000 --> 00:01:31.819
security news recap. In between those, I do conversation
00:01:31.819 --> 00:01:35.000
episodes with people building and operating the
00:01:35.000 --> 00:01:38.640
systems we all depend on. Today I'm joined by
00:01:38.640 --> 00:01:42.200
Jay Lark, Principal DevOps Engineer and the founder
00:01:42.200 --> 00:01:45.760
of Hookbridge. Jay has spent years dealing with
00:01:45.760 --> 00:01:48.959
the operational problems hiding behind webhooks.
00:01:49.040 --> 00:01:53.159
Failed deliveries, duplicate events, retry behavior,
00:01:53.819 --> 00:01:57.459
signature verification, replay, payload visibility,
00:01:57.879 --> 00:02:01.659
and the awkward process of testing webhook integrations
00:02:01.659 --> 00:02:05.420
locally. We talk about what at-least-once delivery
00:02:05.420 --> 00:02:09.419
actually means, why the receiver usually carries
00:02:09.419 --> 00:02:12.860
most of the reliability burden, what teams need
00:02:12.860 --> 00:02:15.719
to log without accidentally storing sensitive
00:02:15.719 --> 00:02:19.659
data, and why replay feels optional right up
00:02:19.659 --> 00:02:23.210
until the first incident. Jay also explains why
00:02:23.210 --> 00:02:26.330
polling may be more useful than engineers give
00:02:26.330 --> 00:02:29.889
it credit for, and why every team probably does
00:02:29.889 --> 00:02:32.830
not need to build its own webhook delivery platform.
Load full transcript
00:02:33.449 --> 00:02:41.090
All right, let's jump in. Today I'm joined by Jay
00:02:41.090 --> 00:02:43.669
Lark, principal DevOps engineer and founder of
00:02:43.669 --> 00:02:46.310
Hookbridge. We're talking about webhooks, why
00:02:46.310 --> 00:02:48.610
they became the default way services talk to
00:02:48.610 --> 00:02:51.069
each other, why they're harder than they look,
00:02:51.439 --> 00:02:54.620
and what teams need to understand before relying
00:02:54.620 --> 00:02:57.439
on them in production. Jay, thank you for joining
00:02:57.439 --> 00:03:00.840
me. Thanks for having me. Glad to be here. So
00:03:00.840 --> 00:03:04.719
I guess let's start at the top. What are webhooks?
00:03:05.120 --> 00:03:08.060
Yeah, we should define it so we all know what
00:03:08.060 --> 00:03:09.900
we're talking about here. They're deceptively
00:03:09.900 --> 00:03:13.099
simple. At least they seem like they're simple
00:03:13.099 --> 00:03:16.080
until they're not. But basically, it's an HTTP
00:03:16.080 --> 00:03:20.449
POST request from one website to another website,
00:03:20.590 --> 00:03:23.569
one app to another. Just a real simple example.
00:03:23.710 --> 00:03:28.050
If you want to, say, get a notification when
00:03:28.050 --> 00:03:30.530
something happens on GitHub, like your PR merges
00:03:30.530 --> 00:03:34.849
in or something, you can give GitHub a URL. And
00:03:34.849 --> 00:03:37.889
when your PR merges in, GitHub will send an HTTP
00:03:37.889 --> 00:03:41.110
POST to the URL you gave it with a little JSON
00:03:41.110 --> 00:03:43.710
body saying PR merge or whatever. And that's
00:03:43.710 --> 00:03:47.310
it. It's real simple. HTTP POST. Why are they
00:03:47.310 --> 00:03:49.469
harder than they look? There's a whole bunch
00:03:49.469 --> 00:03:52.310
of things that can happen. If the receiver is
00:03:52.310 --> 00:03:56.349
down, what happens then? Many webhooks don't
00:03:56.349 --> 00:03:59.169
retry. They just kind of sit there. And some
00:03:59.169 --> 00:04:02.409
webhook providers give you an option to go manually
00:04:02.409 --> 00:04:05.810
hit the retry button, but many don't. A lot of
00:04:05.810 --> 00:04:10.069
the big webhook providers will send webhooks
00:04:10.069 --> 00:04:13.270
kind of out of order. So you might get an order
00:04:13.270 --> 00:04:17.620
updated before order created, and your system
00:04:17.620 --> 00:04:21.519
receiving it might not be able to update an order
00:04:21.519 --> 00:04:24.339
that it doesn't have yet. There are latency issues.
00:04:24.660 --> 00:04:28.720
If your servers are slow to say, yes, I received
00:04:28.720 --> 00:04:32.899
this webhook, the sending party may decide, hey,
00:04:33.000 --> 00:04:34.899
they didn't get that. There's an error. Let me
00:04:34.899 --> 00:04:36.300
send it again. And now all of a sudden you have
00:04:36.300 --> 00:04:38.839
duplicates. There's a whole host of things that
00:04:38.839 --> 00:04:41.360
can go wrong in the whole transaction. So on
00:04:41.360 --> 00:04:44.709
paper, a webhook is just an HTTP POST, as you
00:04:44.709 --> 00:04:48.449
said. Why is that mental model the source of so
00:04:48.449 --> 00:04:51.430
much pain? Probably because it seems so simple.
00:04:51.430 --> 00:04:55.149
So people write kind of the happy path in their
00:04:55.149 --> 00:04:56.970
code, right? It's one or two lines of code
00:04:56.970 --> 00:05:00.970
to send an HTTP POST. But all of the things that
00:05:00.970 --> 00:05:03.910
you would have to do when it fails, and, you know,
00:05:03.910 --> 00:05:05.209
you're wishing you did a little bit more than
00:05:05.209 --> 00:05:07.750
the one or two lines of code. All of the things
00:05:07.750 --> 00:05:11.060
that you end up having to do to kind of catch
00:05:11.060 --> 00:05:13.639
all those edge cases, it's pretty significant.
00:05:14.579 --> 00:05:17.319
Separating events by different IDs, deduplicating
00:05:17.319 --> 00:05:19.959
events if you get the same one, and verifying
00:05:19.959 --> 00:05:23.399
that the webhook came in from whoever you expect
00:05:23.399 --> 00:05:26.720
it to. It's a whole set of things that you really
00:05:26.720 --> 00:05:30.040
need to account for and to build for. A lot of
00:05:30.040 --> 00:05:33.500
people don't. And then there's at-least-once delivery.
00:05:34.250 --> 00:05:36.490
What does that actually mean for the person receiving
00:05:36.490 --> 00:05:38.889
the webhook? Yeah, that's good wording, isn't
00:05:38.889 --> 00:05:41.829
it? The big webhook senders, like
00:05:41.829 --> 00:05:45.329
Stripe or PayPal, say, at-least-once
00:05:45.329 --> 00:05:47.209
delivery: We will send you the event at least
00:05:47.209 --> 00:05:50.129
once. We might send it 10 times. We might send
00:05:50.129 --> 00:05:53.550
it more, yeah. So, you know, if the webhook is
00:05:53.550 --> 00:05:56.490
charging your credit card for an order, you certainly
00:05:56.490 --> 00:05:59.670
don't want to process that 10 times. So you're
00:05:59.670 --> 00:06:01.930
going to want to make sure you only process it
00:06:01.930 --> 00:06:05.720
that once. Okay, so there's an asymmetry with webhooks.
00:06:05.720 --> 00:06:08.579
The sender fires an event, but the receiver has
00:06:08.579 --> 00:06:11.660
to expose an endpoint, keep it available, verify
00:06:11.660 --> 00:06:14.180
it, process it, and not lose anything. Is that
00:06:14.180 --> 00:06:17.519
a fair burden to put on every consumer? Fair?
00:06:17.600 --> 00:06:21.040
That's a tough call. That's the way it is now.
00:06:21.459 --> 00:06:24.319
You know, before webhooks became the main thing,
00:06:24.439 --> 00:06:27.300
we used to poll for events. So if you go back
00:06:27.300 --> 00:06:29.920
to our example of a GitHub PR merge, you could
00:06:30.810 --> 00:06:33.470
change your app to check every five minutes or
00:06:33.470 --> 00:06:35.470
every 10 minutes, or maybe every minute: Has
00:06:35.470 --> 00:06:38.290
that PR merged in? Has that PR merged
00:06:38.290 --> 00:06:41.129
in? And most of the time you're going to get nothing
00:06:41.129 --> 00:06:43.949
back. The thing didn't happen, so it's pretty
00:06:43.949 --> 00:06:46.889
wasteful. And also then, on GitHub's end, they're
00:06:46.889 --> 00:06:49.649
getting hammered by you every minute and then
00:06:49.649 --> 00:06:53.269
like a million other people. And I'm not blaming
00:06:53.269 --> 00:06:55.649
GitHub; literally everybody does this. But
00:06:55.649 --> 00:06:58.329
they've switched the contract now, right?
00:06:58.329 --> 00:07:00.709
We'll just send it to you as it
00:07:00.709 --> 00:07:02.889
happens. So it's good for you. You get the event
00:07:02.889 --> 00:07:06.529
as soon as it happens. But yes, there's
00:07:06.529 --> 00:07:09.189
a huge burden on you now. You have to be up all
00:07:09.189 --> 00:07:12.250
the time. You can't go down, or like we said,
00:07:12.250 --> 00:07:14.470
they may not retry. You might never get it. You
00:07:14.470 --> 00:07:16.990
have to do all these deduplications. The
00:07:16.990 --> 00:07:19.230
events may not come in in the order you
00:07:19.230 --> 00:07:21.610
expect, so you have to maybe queue them somewhere
00:07:21.610 --> 00:07:26.370
and wait for event number one in a chain to arrive
00:07:26.370 --> 00:07:28.750
before you process events two, three,
00:07:28.769 --> 00:07:31.089
and four. Then you have to do all the verification
00:07:31.089 --> 00:07:34.310
and all those types of things. So is it fair?
00:07:34.449 --> 00:07:36.689
Maybe. I mean, it's your data you're looking
00:07:36.689 --> 00:07:41.029
for, really. Maybe. But it is just a fact. That's
00:07:41.029 --> 00:07:43.750
how the modern web works now. If you're optimizing
00:07:43.750 --> 00:07:47.269
for like six nines with like 31 seconds per year
00:07:47.269 --> 00:07:49.589
or whatever, if it happens to go down during
00:07:49.589 --> 00:07:52.170
that 31 seconds, you lose out on that webhook,
00:07:52.230 --> 00:07:54.529
right? I mean... For a lot of providers,
00:07:54.769 --> 00:07:57.629
yeah. Somebody like Stripe has a web console
00:07:57.629 --> 00:08:00.110
you can go in and you can hit retry and it'll
00:08:00.110 --> 00:08:02.709
resend it to you. But you have to know that you
00:08:02.709 --> 00:08:05.250
missed it. You don't know that somebody submitted
00:08:05.250 --> 00:08:08.069
an order and you didn't get it. So you're building
00:08:08.069 --> 00:08:09.910
it. Then you're building some checker to go on
00:08:09.910 --> 00:08:11.490
to Stripe's API and check for a missed order.
00:08:11.569 --> 00:08:14.329
Like, wow, you could have built the redundancy
00:08:14.329 --> 00:08:17.490
into your system in the first place. So I guess
00:08:17.490 --> 00:08:20.529
that leads to debugging the failed webhooks.
00:08:20.810 --> 00:08:23.490
That's been miserable for so long. Is there
00:08:23.490 --> 00:08:26.990
a way to combat that? How do we improve it?
00:08:26.990 --> 00:08:31.189
Yeah, it's very, very difficult. A lot
00:08:31.189 --> 00:08:34.190
of the time, if you've ever debugged a failed webhook
00:08:34.190 --> 00:08:36.669
that GitHub sends you, it's this big red
00:08:36.669 --> 00:08:39.029
failed message in the GitHub console, but
00:08:39.029 --> 00:08:40.690
there's not a lot of detail about what happened.
00:08:40.690 --> 00:08:43.629
You might get the HTTP error code back that your
00:08:43.629 --> 00:08:45.710
system sent, but there's really not that
00:08:45.710 --> 00:08:48.960
much there. So really, if you're building
00:08:48.960 --> 00:08:50.639
this yourself, if you're receiving
00:08:50.639 --> 00:08:52.500
webhooks, you have to build in a whole bunch
00:08:52.500 --> 00:08:55.419
of logging to find out what happened. Did the
00:08:55.419 --> 00:08:59.559
signature verification go wrong? Is there just
00:08:59.559 --> 00:09:02.360
too much of a latency? A lot of webhook senders
00:09:02.360 --> 00:09:04.759
will just fail pretty fast if you don't respond
00:09:04.759 --> 00:09:07.039
and acknowledge and process the data quickly enough.
00:09:07.299 --> 00:09:10.139
A whole host of things can go wrong. So you need
00:09:10.139 --> 00:09:13.919
that detailed logging and information to be able
00:09:13.919 --> 00:09:17.639
to debug it. And most senders don't give you
00:09:17.639 --> 00:09:20.960
what you need. How do we get around the latency
00:09:20.960 --> 00:09:23.679
requirements or the availability requirements
00:09:23.679 --> 00:09:26.519
without just throwing a ton of infrastructure
00:09:26.519 --> 00:09:29.039
at it, right? Yeah, it's tough. It's a tough
00:09:29.039 --> 00:09:32.519
call. We have a lot of customers that use n8n
00:09:32.519 --> 00:09:36.159
to receive webhooks. It's a great product, really
00:09:36.159 --> 00:09:38.840
easy to use. And a lot of the workflows get kicked
00:09:38.840 --> 00:09:41.950
off by webhooks. I think a lot of people use
00:09:41.950 --> 00:09:44.909
n8n, and it's hard for me to say, because it's
00:09:44.909 --> 00:09:47.070
cheap, and they end up putting it on pretty
00:09:47.070 --> 00:09:49.769
under-specced servers, which just take a little bit
00:09:49.769 --> 00:09:52.690
longer to respond. And a lot of providers time
00:09:52.690 --> 00:09:55.730
out really fast on that. So the answer to your
00:09:55.730 --> 00:09:58.629
question is that you have to give it enough horsepower.
00:09:58.970 --> 00:10:01.990
That's just the fact of the matter. Going down
00:10:01.990 --> 00:10:04.149
for deployments and things like that, you have
00:10:04.149 --> 00:10:06.809
to build in that redundancy. So however many
00:10:06.809 --> 00:10:10.419
listeners you have up, they have to stay up while
00:10:10.419 --> 00:10:13.200
the new ones come take their place. It's not all
00:10:13.200 --> 00:10:16.139
that different from kind of rotating on a web
00:10:16.139 --> 00:10:18.639
server underneath a load balancer, but
it's just
00:10:18.639 --> 00:10:21.500
something you do have to account for when you're
00:10:21.500 --> 00:10:23.779
receiving webhooks because you have no idea when
00:10:23.779 --> 00:10:26.139
they might come in. Whose problem is it when
00:10:26.139 --> 00:10:28.179
there is a failure? Is it on the sender? Is it
00:10:28.179 --> 00:10:30.620
on the receiver? Is it, I guess, case by case?
00:10:30.659 --> 00:10:32.580
Yeah, I mean, it can be both, right? Depending
00:10:32.580 --> 00:10:35.820
on why. Ultimately, it's kind of the receiver's
00:10:35.820 --> 00:10:38.700
problem to deal with, right? They want the
00:10:38.700 --> 00:10:41.259
data. They have to kind of figure it out, and
00:10:41.259 --> 00:10:42.899
whether that's creating a whole bunch of support
00:10:42.899 --> 00:10:45.220
tickets with the upstream provider and, you know,
00:10:45.220 --> 00:10:48.279
maybe someday getting a response, or beefing up your
00:10:48.279 --> 00:10:50.659
infrastructure and, you know, solving whatever
00:10:50.659 --> 00:10:52.700
problem is causing the delivery problem.
00:10:52.700 --> 00:10:55.820
Ultimately, I think it comes down on the person
00:10:55.820 --> 00:10:58.700
receiving the webhook. That's fair. And I guess
00:10:58.700 --> 00:11:01.779
before we even know, right, we don't know if
00:11:01.779 --> 00:11:03.799
there was an issue if the webhook didn't go through
00:11:03.799 --> 00:11:08.019
if it was a latency drop-off, if our servers just
00:11:08.019 --> 00:11:09.700
didn't receive the webhook for whatever reason
00:11:09.700 --> 00:11:13.179
during that time that it was sent. How much visibility
00:11:13.179 --> 00:11:18.039
should a provider have or service have to look
00:11:18.039 --> 00:11:21.679
into webhook calls? Like what's a standard as
00:11:21.679 --> 00:11:24.200
far as visibility? Yeah, I think you need a lot
00:11:24.200 --> 00:11:26.700
there. You need good logging, like I said, between
00:11:26.700 --> 00:11:29.340
the whole TCP handshake to see whether
00:11:29.340 --> 00:11:31.379
things are getting cut off because of latency.
00:11:31.950 --> 00:11:34.610
You need visibility into where that might be
00:11:34.610 --> 00:11:38.149
dying. You need the request and the response
00:11:38.149 --> 00:11:41.590
headers. Maybe the signature verification is
00:11:41.590 --> 00:11:43.789
just not working. A lot of people, at first,
00:11:43.909 --> 00:11:46.470
if they're actually trying to do signature verification,
00:11:46.570 --> 00:11:49.529
actually verify on the wrong part of the message.
00:11:49.889 --> 00:11:52.769
And that's going to be pretty clear if you're
00:11:52.769 --> 00:11:54.990
looking at the headers of the response, but sometimes
00:11:54.990 --> 00:11:57.169
you just don't get those. So you're going to
00:11:57.169 --> 00:12:00.080
need the full request and response headers.
00:12:00.340 --> 00:12:03.460
You really need everything except for probably
00:12:03.460 --> 00:12:05.720
the body of the message, right? There's some
00:12:05.720 --> 00:12:08.659
privacy and security issues with seeing
00:12:08.659 --> 00:12:11.139
whatever might be sent in that body. But other
00:12:11.139 --> 00:12:13.700
than that, you really need a pretty comprehensive
00:12:13.700 --> 00:12:18.159
observability view into that transaction. So
00:12:18.159 --> 00:12:19.580
that was actually going to be my next question
00:12:19.580 --> 00:12:23.879
on the privacy compliance angle. Is it just the
00:12:23.879 --> 00:12:26.169
body? Is that the only thing that we're really
00:12:26.169 --> 00:12:29.149
concerned with, I guess, with PII? I mean, well,
00:12:29.350 --> 00:12:32.529
that's a huge one. People use webhooks for anything.
00:12:32.750 --> 00:12:35.330
So there's no telling what could be in the body.
00:12:35.470 --> 00:12:37.870
Obviously, listeners out there, they'll know
00:12:37.870 --> 00:12:40.909
what their app does. Maybe there's nothing super
00:12:40.909 --> 00:12:44.649
sensitive in a PR merged message, but typically
00:12:44.649 --> 00:12:45.889
there's going to be something more sensitive
00:12:45.889 --> 00:12:48.789
in there. But it's not only the body. Tokens
00:12:48.789 --> 00:12:51.870
and signature verification data
00:12:51.870 --> 00:12:55.809
can be in the headers. Those can also be intercepted
00:12:55.809 --> 00:12:58.389
and used for nefarious purposes if somebody has
00:12:58.389 --> 00:13:01.309
access to things they shouldn't. What about replayability?
00:13:01.629 --> 00:13:04.330
Is that a core requirement, or does that feel
00:13:04.330 --> 00:13:07.549
optional until the first incident? It probably
00:13:07.549 --> 00:13:11.009
feels optional to a lot of people at first. Because
00:13:11.009 --> 00:13:13.870
going back to the idea of sending an HTTP POST,
00:13:13.909 --> 00:13:15.429
it's usually one or two lines of code, right?
00:13:15.529 --> 00:13:18.830
You just do it. The first time you have an incident,
00:13:18.950 --> 00:13:21.549
you're going to want to be able to replay things.
00:13:21.870 --> 00:13:24.070
And the fact is a lot of providers just don't
00:13:24.070 --> 00:13:27.889
allow that. But yeah, you can, especially if
00:13:27.889 --> 00:13:32.509
you are receiving a webhook and your system is
00:13:32.509 --> 00:13:34.690
sending a 200 response back, yeah, I got it.
00:13:34.730 --> 00:13:38.409
But something internal to your system broke and
00:13:38.409 --> 00:13:40.529
you didn't process the message correctly. It
00:13:40.529 --> 00:13:43.450
might be gone now from the sender. So, you know,
00:13:43.450 --> 00:13:45.429
maybe you'll fix your back end and say, oh, things
00:13:45.429 --> 00:13:47.710
work now, but you need to be able to go and replay
00:13:47.710 --> 00:13:50.350
those messages. Okay, so we have observability.
00:13:50.690 --> 00:13:53.529
We have webhooks being received. We maybe have
00:13:53.529 --> 00:13:55.909
the ability to replay them. Now from the security
00:13:55.909
--> 00:13:59.730
side of things, the trust boundary: Once you expose
00:13:59.730 --> 00:14:02.049
that public endpoint for incoming events,
00:14:02.269 --> 00:14:05.049
what are the security basics that people should
00:14:05.049 --> 00:14:07.960
get right from the get-go? Yeah, this isn't
00:14:07.960 --> 00:14:10.639
everywhere, but most providers now are going
00:14:10.639 --> 00:14:13.620
to give you a cryptographic signature that you
00:14:13.620 --> 00:14:16.820
can verify that the sender is who you expect.
00:14:16.820 --> 00:14:19.320
It's them sending it. So if the provider
00:14:19.320 --> 00:14:21.259
you're receiving webhooks from offers that,
00:14:21.379 --> 00:14:25.279
please do that. Verify it. But there's a second
00:14:25.279 --> 00:14:28.440
part that I definitely see people ignoring is
00:14:28.440 --> 00:14:32.559
just because the webhook came from
00:14:32.990 --> 00:14:36.009
a trusted source doesn't mean whatever the payload
00:14:36.009 --> 00:14:39.850
is asking you to do in your system is okay. Like
00:14:39.850 --> 00:14:42.049
you need to verify the business logic and say,
00:14:42.169 --> 00:14:45.090
okay, yeah, this order can go to this or this
00:14:45.090 --> 00:14:48.570
thing is owned by this project ID. So it's not
00:14:48.570 --> 00:14:51.710
just blind trust. It's a trusted message, but still
00:14:51.710 --> 00:14:53.570
do all the things you would normally do
00:14:53.570 --> 00:14:56.470
on an HTML form or something, you know,
00:14:56.470 --> 00:14:59.149
in your app. What does HMAC signing actually
00:14:59.149 --> 00:15:02.570
prove? Like more than just like tokens or? Yeah,
00:15:02.570 --> 00:15:06.690
so they'll give you a key and then they'll sign
00:15:06.690 --> 00:15:09.730
their message and your system can compare, make
00:15:09.730 --> 00:15:12.509
sure that the message is signed by them with
00:15:12.509 --> 00:15:16.250
the key that they give you. So if you're doing
00:15:16.250 --> 00:15:18.269
this signature verification with Stripe, the
00:15:18.269 --> 00:15:20.549
messages that are coming in, you can verify those
00:15:20.549 --> 00:15:23.129
are coming from Stripe. Okay. Like I said, you
00:15:23.129 --> 00:15:25.379
still want to make sure that, you know, the
00:15:25.379 --> 00:15:28.360
orders, or probably more importantly, refunds, are
00:15:28.360 --> 00:15:30.899
going to the right accounts and you know you're
00:15:30.899 --> 00:15:33.399
somebody's not spamming you with refunds
00:15:33.399 --> 00:15:35.600
to one credit card. But that's all it does. It
00:15:35.600 --> 00:15:38.460
verifies that the message is coming from the
00:15:38.460 --> 00:15:41.919
person you expect it to come from. And in regard
00:15:41.919 --> 00:15:46.139
to security models, how do replay attacks work
00:15:46.139 --> 00:15:50.129
in this context? Yeah, so this is a big
00:15:50.129 --> 00:15:52.190
problem. I don't know if it's a big problem.
00:15:52.250 --> 00:15:53.809
I don't know if people
are intercepting a lot
00:15:53.809 --> 00:15:56.629
of webhook messages, but it's a security precaution
00:15:56.629 --> 00:15:58.389
that you need to take. With signature verification,
00:15:58.929 --> 00:16:02.190
you need to be able to kind of dedupe messages.
00:16:03.190 --> 00:16:07.649
So if a refund is coming in, hopefully your provider
00:16:07.899 --> 00:16:10.340
has an event ID attached to it. Some providers
00:16:10.340 --> 00:16:12.720
don't do event IDs or anything like that. So,
00:16:12.860 --> 00:16:15.519
you use an event ID so you only process that
00:16:15.519 --> 00:16:18.299
message once, and probably attach timestamps
00:16:18.299 --> 00:16:21.600
to it. So a timestamp will say, okay, this, you
00:16:21.600 --> 00:16:23.639
know, maybe you have some rules that say only
00:16:23.639 --> 00:16:25.960
process this message within a certain window
00:16:25.960 --> 00:16:28.580
of the timestamp. So you can do these things
00:16:28.580 --> 00:16:32.139
to protect yourself from somebody who has
00:16:32.139 --> 00:16:35.320
access to those keys or whatever, so they can't
00:16:35.320 --> 00:16:37.220
not spamming you with webhooks you shouldn't
00:16:37.220 --> 00:16:39.779
follow. Okay. And so moving from that, I guess,
00:16:39.779 --> 00:16:42.419
into like local development. So we have security
00:16:42.419 --> 00:16:48.100
in place. I want to do local testing of my webhooks.
00:16:48.220 --> 00:16:52.720
Why is that so painful? It is a very painful
00:16:52.720 --> 00:16:56.039
thing. Because in our earlier example, if you
00:16:56.039 --> 00:16:58.000
want to monitor, if you want a message from GitHub
00:16:58.000 --> 00:17:00.139
when a PR merges, you have to go into GitHub
00:17:00.139 --> 00:17:03.409
and you have to put in, like, a URL of where
00:17:03.409 --> 00:17:05.970
you want GitHub to send that message. GitHub's
00:17:05.970 --> 00:17:07.690
not going to be able to send a message directly
00:17:07.690 --> 00:17:09.890
to your laptop, your localhost environment.
00:17:10.269 --> 00:17:12.750
Maybe you have some sort of tunneling program.
00:17:12.950 --> 00:17:14.809
There's all sorts of them out there. Set up some
00:17:14.809 --> 00:17:17.569
secure connection that pops your laptop out to
00:17:17.569 --> 00:17:21.470
the web with a unique URL. But 99% of those
00:17:21.470 --> 00:17:24.369
programs, the URL changes like literally every
00:17:24.369 --> 00:17:27.630
time. So every time you boot up your laptop,
00:17:27.789 --> 00:17:29.630
you've got to go into GitHub and change something.
00:17:29.910 --> 00:17:34.250
I mean, the other option is deploying to your
00:17:34.250 --> 00:17:35.750
staging environment or your testing environment
00:17:35.750 --> 00:17:38.309
somewhere up in the cloud where you can receive
00:17:38.309 --> 00:17:41.690
webhooks, but every single time you change,
00:17:41.890 --> 00:17:44.390
even if it's a simple spelling mistake, you've got to
00:17:44.390 --> 00:17:46.910
do this whole CI deployment pipeline, and it
00:17:46.910 --> 00:17:50.309
just takes time. So the time spent deploying up
00:17:50.309 --> 00:17:53.210
to the cloud or every single time going up and
00:17:53.210 --> 00:17:56.049
changing this URL, it's annoying. It's a lot
00:17:56.049 --> 00:17:58.029
of friction. I'm assuming you thought about this
00:17:58.029 --> 00:18:00.970
too while building Hookbridge. So how did you
00:18:00.970 --> 00:18:03.849
build that out for local development? Yeah, so
00:18:03.849 --> 00:18:06.890
even developing a product like Hookbridge, because
00:18:06.890 --> 00:18:10.309
we ran into this firsthand. So we built a little
00:18:10.309 --> 00:18:13.569
program, a little binary you can run on your
00:18:13.569 --> 00:18:16.069
computer. You go into our Hookbridge app and
00:18:16.069 --> 00:18:18.609
you create a one-time URL to hand to GitHub
00:18:18.609 --> 00:18:20.430
or whoever you want. And you put that in the
00:18:20.430 --> 00:18:23.210
app and that URL never changes. And then you
00:18:23.210 --> 00:18:25.309
run this little binary on your computer, and
00:18:25.309 --> 00:18:27.410
it'll create a secure tunnel from your laptop
00:18:27.410 --> 00:18:30.789
up to our system. And our system will just kind
00:18:30.789 --> 00:18:34.009
of forward those webhooks through to your laptop.
00:18:34.130 --> 00:18:36.049
So GitHub can send it to us, and then we'll send
00:18:36.049 --> 00:18:38.789
it down to your laptop. You set it up once, just
00:18:38.789 --> 00:18:41.730
like any other webhook, and then that URL is
00:18:41.730 --> 00:18:43.849
live forever. Interesting. So when you started
00:18:43.849 --> 00:18:47.220
building webhook infrastructure. What kind
00:18:47.220 --> 00:18:50.079
of things surprised you the most? Just how many
00:18:50.079 --> 00:18:53.880
ways it can break and how, you know, not our
00:18:53.880 --> 00:18:57.440
system, just webhooks in general. Just our whole
00:18:57.440 --> 00:18:59.720
conversation, how many things you have to account
00:18:59.720 --> 00:19:04.079
for on something that seems so simple. And it
00:19:04.079 --> 00:19:06.799
really turns into this huge distributed thing
00:19:06.799 --> 00:19:09.799
that you have to build to account for all the
00:19:09.799 --> 00:19:12.799
ways things can go wrong. And that was kind of
00:19:12.799 --> 00:19:14.839
the inspiration for building Hookbridge.
00:19:14.839 --> 00:19:16.559
I've been in the industry for quite
00:19:16.559 --> 00:19:18.759
a while now, and I've seen this play
00:19:18.799 --> 00:19:21.359
out over and over everywhere I've
00:19:21.359 --> 00:19:23.859
been that has accepted webhooks. You
00:19:23.859 --> 00:19:25.799
have to do it right. You do it right at the
00:19:25.799 --> 00:19:27.720
beginning or eventually you learn, you have to
00:19:27.720 --> 00:19:29.660
do it, but you have to eventually build this
00:19:29.660 --> 00:19:32.720
huge thing to accept something that seems so
00:19:32.720 -->
00:19:35.019
simple. Yeah, it's just a cURL command,
00:19:35.140 --> 00:19:38.480
right? It's simple. Yeah, it's deceptively simple
00:19:38.480 --> 00:19:41.680
for sure. Okay, so if a platform team is building
00:19:41.680 --> 00:19:45.039
or buying a webhook infrastructure, what would
00:19:45.039 --> 00:19:47.700
you say are the most non-negotiables that
00:19:47.700 --> 00:19:49.940
they should look for or should be table stakes?
00:19:50.240 --> 00:19:53.259
Yeah, I mean, table stakes, obviously, a lot
00:19:53.259 --> 00:19:55.480
of the security stuff that we talked about, I
00:19:55.480 --> 00:19:59.559
mean, I'm obsessed with security. So you have
00:19:59.559 --> 00:20:01.759
to be able to do the signature verification and
00:20:01.759 --> 00:20:04.200
things like that. But you need the tools to be
00:20:04.200 --> 00:20:07.400
able to troubleshoot when there are problems. All the
00:20:07.400 --> 00:20:10.119
observability and logging that we talked about, and
00:20:10.119 --> 00:20:14.940
inspecting all the request and response
00:20:14.940 --> 00:20:18.319
headers. That's
00:20:18.319 --> 00:20:19.880
what I'm trying to say. You just need to be able
00:20:19.880 --> 00:20:22.680
to see the whole chain to see where it's going
00:20:22.680 --> 00:20:24.799
wrong because it doesn't matter if you're sending
00:20:24.799 --> 00:20:27.099
webhooks or if you're receiving them. The other
00:20:27.099 --> 00:20:31.140
party is usually not very helpful in troubleshooting
00:20:31.140 --> 00:20:33.170
the problem. And then obviously all the other
00:20:33.170 --> 00:20:34.769
problems that we talked about, like you got to
00:20:34.769 --> 00:20:38.089
be up all the time, that kind of thing. So what
00:20:38.089 --> 00:20:40.670
prompted you to build Hookbridge to solve this?
00:20:40.910 --> 00:20:43.130
Yeah, because literally every place that I've
00:20:43.130 --> 00:20:45.430
been that uses webhooks runs into these exact
00:20:45.430 --> 00:20:48.009
same problems. And finally, I'm like,
00:20:48.009 --> 00:20:50.670
just going to build it. Build it once and we
00:20:50.670 --> 00:20:55.210
can use it forever. And what kind of users are
00:20:55.210 --> 00:20:58.150
you seeing use Hookbridge and what kind of problems
00:20:58.150 --> 00:21:01.390
are they solving? Yeah, a lot of it is the
00:21:01.839 --> 00:21:04.920
n8n users, like I talked about. We'll just act
00:21:04.920 --> 00:21:07.460
as an intermediary in front of their n8n box.
00:21:08.220 --> 00:21:10.539
We'll keep retrying for as long as they want,
00:21:10.640 --> 00:21:12.920
and we'll try a little bit longer, and we won't
00:21:12.920 --> 00:21:15.380
time out as fast as maybe some
00:21:15.380 --> 00:21:17.579
providers that are having issues with
00:21:17.579 --> 00:21:22.339
n8n. But an interesting newish thing is, I
00:21:22.339 --> 00:21:24.019
mean, I'm sure we've all heard about OpenClaw
00:21:24.019 --> 00:21:27.240
and all these other claws, all the home AI automation
00:21:27.950 --> 00:21:30.089
things that are out there. A lot of people are
00:21:30.089 --> 00:21:33.529
setting up systems, tools at home that are great,
00:21:33.609 --> 00:21:35.529
do a lot of different things. A lot of them are
00:21:35.529 --> 00:21:39.069
requiring to receive data through webhooks. Don't
00:21:39.069 --> 00:21:42.430
let OpenClaw open holes in your router
00:21:42.430 --> 00:21:45.210
to send them to your laptop. So one thing that
00:21:45.210 --> 00:21:47.470
one thing we've added to Hookbridge
00:21:47.470 --> 00:21:50.069
that's really caught on is we call it a pull
00:21:50.069 --> 00:21:52.990
endpoint, where you go into Hookbridge, you add
00:21:52.990 --> 00:21:55.680
an endpoint, a webhook-receiving endpoint, just
00:21:55.680 --> 00:21:57.960
like you would normally. You hand that to Stripe
00:21:57.960 --> 00:21:59.920
or whoever is sending you webhooks and they send
00:21:59.920 --> 00:22:02.220
webhooks normally. But we don't forward them
00:22:02.220 --> 00:22:05.140
onto you. We just hold them and then OpenClaw,
00:22:05.279 --> 00:22:08.259
whatever system you have that wants to receive
00:22:08.259 --> 00:22:11.519
webhooks, they can just hit our API and they
00:22:11.519 --> 00:22:14.460
can pull down the events from us when you want
00:22:14.460 --> 00:22:16.460
them. So you don't have to open up any firewall
00:22:16.460 --> 00:22:19.200
holes or anything like that. It also solves some
00:22:19.200 --> 00:22:21.680
issues with ordering and things. So you can
00:22:22.029 --> 00:22:24.509
pull down by event ID if you want all the
00:22:24.509 --> 00:22:26.609
orders first and then all the updates and things
00:22:26.609 --> 00:22:28.990
like that. So the out-of-order processing
00:22:28.990 --> 00:22:32.150
and process-once issues kind of go away when
00:22:32.150 --> 00:22:34.089
you're pulling things with an
00:22:34.089 --> 00:22:36.529
API instead of just receiving them whenever
00:22:36.529 --> 00:22:40.089
they might happen. Yeah, that's cool. Okay, so if
00:22:40.089 --> 00:22:43.190
I wasn't ready to start with a
00:22:43.190 --> 00:22:46.549
platform yet, but I want to shore up or fix
00:22:46.549 --> 00:22:49.470
my webhook issues so I don't run into these
00:22:49.470 --> 00:22:52.710
problems, what are the top three things
00:22:52.710 --> 00:22:54.589
that we should do right off the bat?
00:22:54.609 --> 00:22:56.049
We kind of talked a little bit about this, but
00:22:56.049 --> 00:22:59.690
is it HMAC? What is it? My
00:22:59.690 --> 00:23:01.289
number one thing is always going to be the HMAC
00:23:01.289 --> 00:23:03.369
signature. Like everyone should do that. Just
00:23:03.369 --> 00:23:07.549
don't accept random webhooks from whoever's pretending
00:23:07.549 --> 00:23:09.869
to be somebody else. Beef up your observability
00:23:09.869 --> 00:23:13.369
platform. Whatever you have now, just make sure
00:23:13.369 --> 00:23:17.170
that you can diagnose any problems, whether
they're
00:23:17.660 --> 00:23:20.519
on your end receiving, or as much as you can
00:23:20.519 --> 00:23:22.660
kind of gain insight from whoever's sending you,
00:23:22.720 --> 00:23:24.640
or if you're sending webhooks, you know, the
00:23:24.640 --> 00:23:26.720
responses you're getting back from the person
00:23:26.720 --> 00:23:28.440
you're trying to send to. Just make sure that
00:23:28.440 --> 00:23:31.380
your observability platform is as good as it
00:23:31.380 --> 00:23:34.180
could possibly be. Have you noticed, we talked
00:23:34.180 --> 00:23:36.599
a little bit, you talked a little bit about OpenClaw.
00:23:36.799 --> 00:23:41.079
Fable is now back online after being taken away
00:23:41.079 --> 00:23:44.420
for a couple weeks. Have you noticed any trends
00:23:44.420 --> 00:23:47.700
on the security side? There's been a lot of RCEs,
00:23:47.700 --> 00:23:49.339
I know, outside of just webhooks, but there's
00:23:49.339 --> 00:23:53.279
been a lot of CVEs out there lately. Have you
00:23:53.279 --> 00:23:55.460
noticed anything with the platform, like from
00:23:55.460 --> 00:23:59.140
your customers or from other friends in the industry
00:23:59.140 --> 00:24:03.200
around webhooks, trends related to AI or related
00:24:03.200 --> 00:24:06.200
to just infiltration in general, like trends
00:24:06.200 --> 00:24:09.240
that are happening? Maybe AI is making that easier,
00:24:09.319 --> 00:24:11.740
but maybe it's not even AI related. Yeah, a couple.
00:24:11.759 --> 00:24:14.480
Like I said, the people running OpenClaw at
00:24:14.480 --> 00:24:16.460
home that need to receive webhooks. That was
00:24:16.460 --> 00:24:18.339
kind of an unexpected surprise. It's nice.
00:24:18.519 --> 00:24:21.119
It's more people than you might think, too, that
00:24:21.119 --> 00:24:23.339
are all of a sudden like, I need to receive webhooks
00:24:23.339 --> 00:24:27.160
to my iMac. That's weird. Okay, I get it. I get
00:24:27.160 --> 00:24:30.259
a lot of, and this has always happened, even
00:24:30.259 --> 00:24:33.440
when I didn't run the business. Just being
00:24:33.440 --> 00:24:35.680
on the DevOps team, you do occasionally get these
00:24:35.680 --> 00:24:38.200
emails, but there's a lot of people now that
00:24:38.200 --> 00:24:41.700
have access to OpenClaw or AnyClaw, any AI, whatever,
00:24:41.839 --> 00:24:45.000
who are security researchers. And they send you
00:24:45.000 --> 00:24:48.279
all sorts of emails: "I found whatever vulnerability.
00:24:48.319 --> 00:24:51.539
I'll tell you about it for whatever fee," you
00:24:51.539 --> 00:24:54.579
know, so you get a lot of that. And I think that
00:24:54.579 --> 00:24:57.700
has probably increased since the dawn of AI.
00:24:57.940 --> 00:25:00.420
I don't think they're finding legitimate things.
00:25:01.079 --> 00:25:02.400
The couple that I've asked, oh, hey, can you
00:25:02.400 --> 00:25:04.099
show me some proof? And maybe I'll give you the
00:25:04.099 --> 00:25:07.000
bounty.
They're not finding real things, at least
00:25:07.000 --> 00:25:08.680
from what I've seen. Locally, personally, I'm
00:25:08.680 --> 00:25:11.440
using it. Yeah, it gives me good confidence when
00:25:11.440 --> 00:25:14.380
I can run the code through multiple AI models
00:25:14.380 --> 00:25:16.940
and say, give me a thorough security review.
00:25:17.119 --> 00:25:20.200
And we can definitely find things that way. Yeah,
00:25:20.200 --> 00:25:24.400
out on the open web, I haven't seen a whole lot
00:25:24.400 --> 00:25:27.059
coming our way yet. Fingers crossed, because
00:25:27.059 --> 00:25:29.839
I know some communities are really under fire
00:25:29.839 --> 00:25:33.549
from it. And what kind of customers primarily
00:25:33.549 --> 00:25:36.450
do you have or users do you have in the platform?
00:25:36.549 --> 00:25:39.190
And what are they using Hookbridge for? Like,
00:25:39.230 --> 00:25:41.849
what problems are they solving? Yeah, big one
00:25:41.849 --> 00:25:45.069
is the n8n users. Someday I'll be able to say
00:25:45.069 --> 00:25:47.769
that well. But a lot of them are the opposite
00:25:47.769 --> 00:25:52.150
of what we've been talking about. We've mostly
00:25:52.150 --> 00:25:54.519
been talking about how to build a system if
00:25:54.519 --> 00:25:56.640
you are going to receive webhooks. And it's a
00:25:56.640 --> 00:25:59.099
big thing, like we discussed. You've got to have
00:25:59.099 --> 00:26:00.900
a queue to hold all these messages and make sure
00:26:00.900 --> 00:26:02.680
you process them in the right way and do all
00:26:02.680 --> 00:26:04.420
this signature stuff. A lot of people don't
00:26:04.420 --> 00:26:06.779
do it. So if you are sending webhooks and the
00:26:06.779 --> 00:26:09.019
person you are sending to doesn't have a system
00:26:09.019 --> 00:26:12.019
that is reliable and can receive webhooks and
00:26:12.019 --> 00:26:14.259
things, you need to kind of build the reverse
00:26:14.259 --> 00:26:17.440
of it, be able to retry and stuff. So instead
00:26:17.440 --> 00:26:21.569
of having to do that, people just use us because
00:26:21.569 --> 00:26:23.970
we will queue the message, keep retrying
00:26:23.970 --> 00:26:26.130
it, sign it for you, and do all that kind
00:26:26.130 --> 00:26:28.309
of stuff. So yeah, everything
00:26:28.309 --> 00:26:30.009
we talked about receiving, you kind of have to
00:26:30.009 --> 00:26:33.210
do it on the reverse too. It's interesting. So
00:26:33.210 --> 00:26:36.950
okay, wrapping up, is there any webhook opinion
00:26:36.950 --> 00:26:40.369
that you have that maybe other engineers might
00:26:40.369 --> 00:26:43.299
disagree with? Well, as the owner of Hookbridge, I'll
00:26:43.299 --> 00:26:45.059
probably say, you know, most teams shouldn't
00:26:45.059 --> 00:26:47.920
build their own webhook infrastructure. I would
00:26:47.920 --> 00:26:50.480
say a controversial webhook take is to not use
00:26:50.480 --> 00:26:53.619
webhooks. I would say polling is
an underrated
00:26:53.619 --> 00:26:56.200
thing. Polling still has a place. Like, obviously,
00:26:56.359 --> 00:26:59.799
be a good citizen. Don't hammer someone. Or,
00:26:59.859 --> 00:27:02.059
you know, this wasn't my intention, but again,
00:27:02.240 --> 00:27:05.079
use the Hookbridge pull endpoint. Poll us.
00:27:05.259 --> 00:27:06.940
We don't care. Let's say someone was looking
00:27:06.940 --> 00:27:11.099
into growing their webhook journey and they
00:27:11.099 --> 00:27:13.980
don't want to do all of this themselves. What
00:27:13.980 --> 00:27:16.519
can Hookbridge offer them and what does it cost
00:27:16.519 --> 00:27:19.019
to get started? What's the barrier to entry?
00:27:19.740 --> 00:27:21.599
Yeah, I don't blame you if you don't want to
00:27:21.599 --> 00:27:24.160
build this huge thing because it's a lot. We
00:27:24.160 --> 00:27:27.119
have a free plan. You can sign up. All the features
00:27:27.119 --> 00:27:29.759
are there. There's no big thing that you can't
00:27:29.759 --> 00:27:32.180
do with the free plan. I think you can send like
00:27:32.180 --> 00:27:35.099
a thousand webhooks over a month. That should hopefully
00:27:35.099 --> 00:27:37.240
plenty to test something out. And for a lot of
00:27:37.240 --> 00:27:39.519
our users, that's enough for their home OpenClaw
00:27:39.519 --> 00:27:42.880
installs, like we talked about. A thousand
00:27:42.880 --> 00:27:44.500
a month is fine, and they can just send and
00:27:44.500 --> 00:27:46.420
receive that many for free. But then obviously
00:27:46.420 --> 00:27:50.039
we have paid plans if you need to send more messages
00:27:50.039 --> 00:27:51.900
per month or you need a whole bunch of endpoints
00:27:51.900 --> 00:27:54.140
and things like that. The paid plans have long
00:27:54.140 --> 00:27:57.359
retention periods. So if you're receiving webhooks
00:27:57.359 --> 00:27:59.359
and you're down or actually you're probably going
00:27:59.359 --> 00:28:00.859
to want to get your own site up really fast too.
00:28:00.960 --> 00:28:02.619
But you have no control over your customers,
00:28:02.740 --> 00:28:05.000
right? If the people you're sending to are down
00:28:05.000 --> 00:28:08.920
for a lengthy amount of time, we'll hold the
00:28:08.920 --> 00:28:10.539
message and we'll keep retrying it for a long
00:28:10.539 --> 00:28:12.700
time. And our paid plans have kind of lengthy
00:28:12.700 --> 00:28:16.160
retry periods, retention periods there. Very
00:28:16.160 --> 00:28:17.859
cool. So where can people find out more about
00:28:17.859 --> 00:28:21.299
you and Hookbridge? Hookbridge is at www.hookbridge.io.
00:28:21.299 --> 00:28:24.579
Send me an email, jay@hookbridge.io,
00:28:24.680 --> 00:28:27.519
J-A-Y. Happy to talk about webhooks. Happy
00:28:27.519 --> 00:28:29.500
to talk about anything. Yeah, let us know what
00:28:29.500 --> 00:28:32.200
you think. Let us know your pain points with
00:28:32.200 --> 00:28:34.710
webhooks, because I know there
are tons of
00:28:34.710 --> 00:28:36.930
them out there, and we are happy to discuss new
00:28:36.930 --> 00:28:40.970
use cases and how you might use
00:28:40.970 --> 00:28:43.670
webhooks or how Hookbridge could work for you.
00:28:43.789 --> 00:28:45.609
Very cool. Thanks so much, Jay, for coming on.
00:28:45.630 --> 00:28:47.710
Really appreciate your time. Thank you. That
00:28:47.710 --> 00:28:50.410
was my conversation with Jay Lark, Principal
00:28:50.410 --> 00:28:54.049
DevOps Engineer and founder of Hookbridge. The
00:28:54.049 --> 00:28:56.509
biggest takeaway for me is that webhooks are
00:28:56.509 --> 00:29:00.289
not unreliable because HTTP is unreliable or
00:29:00.289 --> 00:29:03.009
because the basic pattern is bad. They become
00:29:03.009 --> 00:29:06.289
unreliable when teams build only the happy path
00:29:06.289 --> 00:29:09.650
and treat delivery like a single request instead
00:29:09.650 --> 00:29:12.950
of a distributed workflow. Once a webhook matters
00:29:12.950 --> 00:29:16.029
to your business, you need to assume duplicates
00:29:16.029 --> 00:29:18.930
will happen. Events may arrive out of order.
00:29:19.309 --> 00:29:22.390
Endpoints will occasionally be unavailable. And
00:29:22.390 --> 00:29:25.890
something inside your own system may fail after
00:29:25.890 --> 00:29:29.130
you already acknowledged the request. That changes
00:29:29.130 --> 00:29:33.369
what success means. A 200 response does not necessarily
00:29:33.369 --> 00:29:37.089
mean the event was processed correctly. A valid
00:29:37.089 --> 00:29:39.789
signature does not mean the requested business
00:29:39.789 --> 00:29:43.829
action is safe. And a retry does not help much
00:29:43.829 --> 00:29:47.490
if nobody can see what failed or replay the event
00:29:47.490 --> 00:29:51.109
later. I also liked Jay's point about the burden
00:29:51.109 --> 00:29:54.490
placed on the receiver. The sender gets to fire
00:29:54.490 --> 00:29:57.319
the request and move on, while the receiving
00:29:57.319 --> 00:30:00.980
team has to expose a public endpoint, keep it
00:30:00.980 --> 00:30:04.380
available, validate the sender, protect the business
00:30:04.380 --> 00:30:08.920
logic, deduplicate events, and build enough observability
00:30:08.920 --> 00:30:12.140
to figure out what happened when something goes
00:30:12.140 --> 00:30:14.819
wrong. And sometimes the answer really might
00:30:14.819 --> 00:30:17.940
be polling, not hammering someone's API every
00:30:17.940 --> 00:30:21.200
second, but using polling intentionally when
00:30:21.200 --> 00:30:23.960
it gives you better control over timing, ordering,
00:30:24.220 --> 00:30:27.309
or infrastructure exposure. So if you are building
00:30:27.309 --> 00:30:30.150
a webhook integration, start with the failure
00:30:30.150 --> 00:30:34.690
path. Verify signatures. Use event IDs for idempotency.
00:30:34.690 --> 00:30:38.269
Acknowledge requests quickly. Retain
00:30:38.269 --> 00:30:41.289
enough information to debug failures. And make
00:30:41.289 --> 00:30:44.089
sure you have a replay story before you need
00:30:44.089 --> 00:30:46.670
one. You can learn more about Jay and Hookbridge
00:30:46.670 --> 00:30:50.029
at hookbridge.io. I'll have that and anything
00:30:50.029 --> 00:30:53.109
else we mentioned linked in the show notes. If
00:30:53.109 --> 00:30:55.730
you enjoyed this conversation, follow or subscribe
00:30:55.730 --> 00:30:58.490
to Ship It Weekly wherever you listen to podcasts.
00:30:58.890 --> 00:31:02.089
You can find the show, previous episodes, and
00:31:02.089 --> 00:31:06.130
the weekly DevOps and SRE news recaps at shipitweekly.fm.
00:31:06.130 --> 00:31:09.069
Thanks for listening, and I'll see you later
00:31:09.069 --> 00:31:09.569
this week.