Troubleshooting Kafka In Production

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 2:37:31 PM
Duration: 4483.76
Channels: 1
1
00:00:11.264 --> 00:00:15.445
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.384 --> 00:00:18.244
Introducing RudderStack Profiles.
3
00:00:18.705 --> 00:00:28.630
RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable enriched data to every downstream team.
4
00:00:29.255 --> 00:00:35.595
You specify the customer traits, then profiles runs the joints and computations for you to create complete customer profiles.
5
00:00:36.370 --> 00:00:39.510
Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack.
6
00:00:42.370 --> 00:01:04.730
You shouldn't have to throw away the database to build with fast changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades old batch computation model for an efficient incremental engine to get complex queries that are always up to date. With Materialise, you can. It's the only true SQL streaming database built from the ground up to meet the needs of modern data products.
7
00:01:05.525 --> 00:01:16.970
Whether it's real time dashboarding and analytics, personalization and segmentation, or automation and alerting, Materialise gives you the ability to work with fresh, correct, and scalable results, all in a familiar SQL interface.
8
00:01:17.270 --> 00:01:18.170
Go to dataengineeringpodcast.com/materialize
9
00:01:20.470 --> 00:01:22.170
today to get 2 weeks free.
10
00:01:24.375 --> 00:01:34.210
Your host is Tobias Macy, and today, I'm interviewing Elad El Dor about operating Kafka in production and how to keep your cluster stable and performant. So, Elad, can you start by introducing yourself?
11
00:01:35.310 --> 00:01:39.090
Yeah. Sure. So, nice to meet you, first of all,
12
00:01:39.805 --> 00:01:40.305
and
13
00:01:40.605 --> 00:01:43.265
and thank you for letting me be on the podcast.
14
00:01:43.805 --> 00:01:45.345
So my name is Elad,
15
00:01:45.645 --> 00:01:47.105
and I'm a DataOps
16
00:01:47.805 --> 00:01:48.705
group manager
17
00:01:49.409 --> 00:01:49.909
at,
18
00:01:50.530 --> 00:01:51.030
IronSource,
19
00:01:51.650 --> 00:01:53.270
which is an Israeli company
20
00:01:53.729 --> 00:01:55.189
that was merged with
21
00:01:55.490 --> 00:01:55.990
Unity.
22
00:01:57.225 --> 00:01:57.725
And
23
00:01:58.265 --> 00:01:59.485
on my job, I
24
00:02:00.185 --> 00:02:00.685
primarily
25
00:02:01.065 --> 00:02:01.565
focus
26
00:02:02.585 --> 00:02:03.165
on stability,
27
00:02:03.625 --> 00:02:05.245
deployment, and cost reduction
28
00:02:05.830 --> 00:02:06.330
of
29
00:02:06.630 --> 00:02:07.930
big data analytics
30
00:02:08.950 --> 00:02:10.490
clusters on AWS,
31
00:02:11.270 --> 00:02:12.090
such as
32
00:02:12.390 --> 00:02:12.890
Druid,
33
00:02:13.285 --> 00:02:13.785
Trino,
34
00:02:14.805 --> 00:02:15.305
Spark,
35
00:02:15.925 --> 00:02:17.625
etcetera. And I also,
36
00:02:18.565 --> 00:02:19.705
am involved with
37
00:02:20.540 --> 00:02:21.040
Kafka
38
00:02:21.580 --> 00:02:22.640
production issues
39
00:02:23.500 --> 00:02:26.080
and cost reduction issues, but mainly
40
00:02:26.380 --> 00:02:27.600
production and stability
41
00:02:28.060 --> 00:02:29.600
issues on, on Kafka.
42
00:02:30.455 --> 00:02:33.115
Prior to that, I was a SRE
43
00:02:33.975 --> 00:02:34.475
for
44
00:02:34.775 --> 00:02:36.395
a company called Cognite,
45
00:02:37.310 --> 00:02:37.810
where
46
00:02:38.190 --> 00:02:39.410
I where I was
47
00:02:39.870 --> 00:02:41.410
in charge of stability,
48
00:02:42.590 --> 00:02:44.610
of big data clusters
49
00:02:45.215 --> 00:02:47.795
on prem, for Linux on prem,
50
00:02:48.415 --> 00:02:51.394
mainly for Spark Streaming, Spark Batch,
51
00:02:51.775 --> 00:02:53.875
and Presto on HDFS
52
00:02:54.900 --> 00:02:55.640
at various,
53
00:02:56.180 --> 00:02:57.160
customer sites.
54
00:02:57.540 --> 00:02:58.920
Prior to that, I was
55
00:02:59.380 --> 00:03:00.840
a Java back end engineer
56
00:03:01.540 --> 00:03:02.040
for
57
00:03:02.420 --> 00:03:03.720
for about 10 years.
58
00:03:04.425 --> 00:03:04.905
And,
59
00:03:05.385 --> 00:03:10.285
I recently published a book called the Kafka troubleshooting in production,
60
00:03:10.985 --> 00:03:11.705
which will be the
61
00:03:12.849 --> 00:03:13.349
I
62
00:03:14.610 --> 00:03:17.349
think, the main topics of what we will discuss today.
63
00:03:17.890 --> 00:03:20.390
And it talks about how to handle
64
00:03:20.935 --> 00:03:23.435
production issues in Kafka clusters,
65
00:03:23.895 --> 00:03:27.355
both on prem and on the cloud on AWS.
66
00:03:28.250 --> 00:03:33.950
And do you remember how you first got started working in data and what it is about that space that keeps you interested?
67
00:03:34.865 --> 00:03:35.365
So
68
00:03:35.905 --> 00:03:38.165
I I started working on big data,
69
00:03:39.505 --> 00:03:42.565
about 7 years ago when I developed the
70
00:03:43.105 --> 00:03:43.845
my first
71
00:03:44.180 --> 00:03:45.640
Spark streaming application,
72
00:03:46.420 --> 00:03:47.640
consuming from Kafka
73
00:03:48.100 --> 00:03:51.880
and persisting to HDFS on Linux on prem clusters.
74
00:03:52.605 --> 00:03:55.665
Before that, I just wrote, you know, back end applications
75
00:03:55.965 --> 00:03:56.705
that were
76
00:03:57.085 --> 00:03:58.385
reading from some database,
77
00:03:58.765 --> 00:04:01.940
writing from some into some databases, but it wasn't,
78
00:04:02.660 --> 00:04:04.040
it wasn't formally
79
00:04:04.980 --> 00:04:06.360
big data. And
80
00:04:07.675 --> 00:04:08.575
5 years ago,
81
00:04:09.355 --> 00:04:09.935
I understood
82
00:04:10.275 --> 00:04:10.775
that,
83
00:04:11.275 --> 00:04:12.575
on big data clusters,
84
00:04:13.035 --> 00:04:15.775
there are production issues that I didn't see before.
85
00:04:16.850 --> 00:04:17.350
And,
86
00:04:17.730 --> 00:04:21.430
instead of so I reverted from being a developer,
87
00:04:22.050 --> 00:04:24.870
I reverted to being an SRE and solving
88
00:04:26.015 --> 00:04:26.915
stability issues
89
00:04:27.295 --> 00:04:31.155
in these clusters, which meant knowing better not my code,
90
00:04:31.695 --> 00:04:32.175
but,
91
00:04:33.055 --> 00:04:33.555
the
92
00:04:34.240 --> 00:04:35.060
the infrastructure,
93
00:04:35.760 --> 00:04:43.275
the the cluster that my code or some code, some big data code, runs on, whether these are Presto clusters,
94
00:04:43.895 --> 00:04:44.395
Spark
95
00:04:45.014 --> 00:04:46.555
clusters, Spark batches,
96
00:04:47.095 --> 00:04:48.075
Spark streaming,
97
00:04:48.455 --> 00:04:48.955
Kafka,
98
00:04:50.530 --> 00:04:51.030
HDFS,
99
00:04:52.210 --> 00:04:56.390
and this it was a whole a whole new world for me.
100
00:04:58.384 --> 00:05:01.764
And that that's what brought me specifically,
101
00:05:03.185 --> 00:05:03.845
to handle
102
00:05:04.705 --> 00:05:08.539
Kafka issues, which I which I do for the last,
103
00:05:09.500 --> 00:05:12.160
like, in production. I do it since 2018,
104
00:05:12.940 --> 00:05:16.159
but I started doing it a year before. So maybe
105
00:05:17.025 --> 00:05:21.605
the question is more what brought me to be not to be a developer anymore,
106
00:05:22.385 --> 00:05:25.310
but to become an SRE slash data ops
107
00:05:25.870 --> 00:05:27.889
of, of big data clusters,
108
00:05:28.270 --> 00:05:29.250
and specifically
109
00:05:29.790 --> 00:05:33.410
focusing on cost reduction, whether on prem or on the cloud.
110
00:05:33.975 --> 00:05:34.475
And
111
00:05:38.294 --> 00:05:39.354
and and focusing
112
00:05:40.615 --> 00:05:41.595
on on Kafka.
113
00:05:42.750 --> 00:05:43.250
So
114
00:05:43.790 --> 00:05:50.450
I I I I will elaborate on on the Kafka issue. So I understood that Kafka is, like, stuck in the middle of everything,
115
00:05:50.914 --> 00:05:53.095
everything. Like, if there is a problem with Kafka,
116
00:05:53.555 --> 00:05:55.495
your whole data pipeline is just
117
00:05:56.034 --> 00:05:56.534
stuck.
118
00:05:57.074 --> 00:05:57.574
Producers
119
00:05:57.955 --> 00:05:59.095
can't write, consumers
120
00:05:59.634 --> 00:06:00.455
can't read.
121
00:06:02.070 --> 00:06:03.530
And and that's what
122
00:06:04.310 --> 00:06:08.170
it was a combination of 2 things. 1, the understanding that Kafka,
123
00:06:09.044 --> 00:06:12.104
the best ROI for my time will be to focus on Kafka.
124
00:06:12.805 --> 00:06:14.664
And the second thing was that
125
00:06:15.044 --> 00:06:15.544
I
126
00:06:16.085 --> 00:06:20.570
stumbled into a book called the System Performance by Brenton Gregg,
127
00:06:21.750 --> 00:06:24.650
who was a lead performance engineer at Netflix,
128
00:06:25.185 --> 00:06:27.285
and now he works under the,
129
00:06:28.465 --> 00:06:29.685
Intel's CTO
130
00:06:30.385 --> 00:06:31.525
on distributed
131
00:06:32.145 --> 00:06:32.645
clusters.
132
00:06:33.470 --> 00:06:35.010
And this really
133
00:06:35.550 --> 00:06:37.330
opened me a world of
134
00:06:37.950 --> 00:06:40.130
that I wasn't aware of before,
135
00:06:40.645 --> 00:06:43.145
of monitoring and detecting bottlenecks
136
00:06:43.845 --> 00:06:44.905
in Linux,
137
00:06:45.765 --> 00:06:46.265
clusters.
138
00:06:47.045 --> 00:06:48.745
And not only understanding
139
00:06:49.480 --> 00:06:52.220
what what's your bottleneck what's the cluster bottleneck,
140
00:06:52.680 --> 00:06:55.980
but also it opened a way to reduce, costs.
141
00:06:56.440 --> 00:06:57.660
And what I saw
142
00:06:58.014 --> 00:07:01.155
when I stumbled into production issues in Kafka,
143
00:07:02.175 --> 00:07:03.155
Spark, Presto,
144
00:07:03.854 --> 00:07:04.354
Druid,
145
00:07:06.630 --> 00:07:11.530
For this matter, any service, any applicative service on running on Linux cluster
146
00:07:12.310 --> 00:07:13.190
is that,
147
00:07:13.910 --> 00:07:17.255
understanding how to diagnose issues in a Linux cluster
148
00:07:17.875 --> 00:07:19.815
can really help you detect
149
00:07:20.435 --> 00:07:24.615
what what's the problem in in many cases, and also allows you
150
00:07:25.000 --> 00:07:27.260
to reduce costs on the on the cloud.
151
00:07:28.680 --> 00:07:35.885
In terms of the Kafka focus, you mentioned that you were working as a back end developer. You were
152
00:07:36.265 --> 00:07:43.080
interested in the use cases for Kafka, the production requirements around it, how to make sure that it was stable and performant.
153
00:07:43.700 --> 00:07:52.235
And I'm wondering if you can talk to, at least at a high level, some of the different environments in which you've had experience working with Kafka and some of the
154
00:07:52.615 --> 00:07:56.555
categories of operational challenge that you've had to deal with in that process?
155
00:07:57.550 --> 00:08:00.370
Sure. So I started with a on prem cluster,
156
00:08:00.750 --> 00:08:02.050
a Kafka on prem,
157
00:08:02.590 --> 00:08:03.970
and it was
158
00:08:04.670 --> 00:08:07.575
on a Kafka distribution of Hortonworks,
159
00:08:08.915 --> 00:08:09.415
HDP,
160
00:08:10.595 --> 00:08:13.175
before they were acquired by Cloudera.
161
00:08:14.700 --> 00:08:15.520
So it was
162
00:08:15.900 --> 00:08:17.040
it was a free
163
00:08:17.420 --> 00:08:17.920
distribution
164
00:08:18.540 --> 00:08:19.440
of Kafka,
165
00:08:20.220 --> 00:08:20.720
and
166
00:08:21.995 --> 00:08:26.255
and now I work on with the Kafka Open Source
167
00:08:26.555 --> 00:08:27.935
on AWS Cloud.
168
00:08:28.315 --> 00:08:29.535
So in either cases,
169
00:08:29.840 --> 00:08:31.620
it wasn't a managed service,
170
00:08:32.160 --> 00:08:32.660
so
171
00:08:33.040 --> 00:08:36.980
we had to to diagnose the issues and solve them ourselves.
172
00:08:37.440 --> 00:08:37.940
Now
173
00:08:38.245 --> 00:08:39.785
the because both
174
00:08:40.485 --> 00:08:43.625
are practically Kafka open source, although it was
175
00:08:44.005 --> 00:08:49.380
when I was on prem, it was part of the HDP, but I don't see a lot a lot of difference here.
176
00:08:49.760 --> 00:08:50.420
The main
177
00:08:51.680 --> 00:08:55.140
issue here was on prem is on prem versus cloud.
178
00:08:56.355 --> 00:09:00.135
And this is why my book focuses mostly on cloud deployment,
179
00:09:00.515 --> 00:09:03.335
but also on on prem deployment. So I started
180
00:09:03.955 --> 00:09:04.455
from
181
00:09:04.839 --> 00:09:05.819
from on prem,
182
00:09:06.360 --> 00:09:08.220
and there's a big difference,
183
00:09:09.160 --> 00:09:09.880
in in the
184
00:09:11.695 --> 00:09:14.755
in some parts, there are big differences on on prem because,
185
00:09:16.334 --> 00:09:19.154
you are in charge of the hardware failures, for example.
186
00:09:19.709 --> 00:09:22.610
Now the the most common failure for hardware
187
00:09:23.069 --> 00:09:23.889
on prem,
188
00:09:24.269 --> 00:09:26.485
no matter which cluster it is, are disks.
189
00:09:27.125 --> 00:09:30.105
And in Kafka on prem, usually,
190
00:09:30.485 --> 00:09:32.665
you you you need a lot of storage.
191
00:09:33.204 --> 00:09:35.945
And so in order not to pay much, you use HDD
192
00:09:36.670 --> 00:09:38.130
disks and not SSD.
193
00:09:38.910 --> 00:09:41.329
So they fail for a lot of
194
00:09:41.790 --> 00:09:46.714
their ratio failure is pretty high, and you need to take care of it because it is not cloud.
195
00:09:47.015 --> 00:09:52.955
Another issue is that on on prem environments, you usually the SRE is not on-site.
196
00:09:53.460 --> 00:09:56.200
So you have a tier 1, tier 2 support,
197
00:09:56.740 --> 00:09:58.600
which are usually unaware
198
00:09:59.300 --> 00:09:59.800
of
199
00:10:01.025 --> 00:10:01.925
of signals
200
00:10:02.465 --> 00:10:06.005
that can say that something is going to get wrong,
201
00:10:06.305 --> 00:10:08.815
something is wrong in the in the in the cluster.
202
00:10:09.180 --> 00:10:14.160
So as an SRE of on prem, you reach the problems once the cluster holds
203
00:10:14.620 --> 00:10:15.120
sometime.
204
00:10:16.060 --> 00:10:17.120
But there are also
205
00:10:17.580 --> 00:10:18.640
but on the cloud,
206
00:10:19.325 --> 00:10:23.745
the the cloud provider will tell you, okay, you don't, you don't need to
207
00:10:24.605 --> 00:10:25.185
to handle,
208
00:10:26.205 --> 00:10:26.945
disk failures.
209
00:10:27.430 --> 00:10:27.930
However,
210
00:10:28.390 --> 00:10:28.890
when
211
00:10:29.670 --> 00:10:32.089
even on the cloud, a broker can halt
212
00:10:32.470 --> 00:10:33.430
because of disk,
213
00:10:34.390 --> 00:10:34.890
deterioration,
214
00:10:35.350 --> 00:10:38.515
and you wouldn't know of it until you get into legs.
215
00:10:39.295 --> 00:10:43.555
But you don't need to replace the disk. You just can replace the broker,
216
00:10:44.820 --> 00:10:46.360
which is much easier
217
00:10:47.140 --> 00:10:50.120
in the cloud versus on prem, where you need to,
218
00:10:51.620 --> 00:10:55.465
get into the drawer of disks and replace the disk.
219
00:10:56.165 --> 00:10:58.345
Another big difference is the scaling.
220
00:10:58.965 --> 00:11:00.425
If you have more traffic,
221
00:11:00.965 --> 00:11:05.439
you can scale out. On the cloud, you can just scale out or scale up, and
222
00:11:05.740 --> 00:11:08.399
by spinning in your or spin a new cluster
223
00:11:09.259 --> 00:11:10.079
pretty easily,
224
00:11:10.540 --> 00:11:11.920
while on on prem,
225
00:11:13.145 --> 00:11:14.685
it's very, very tough,
226
00:11:15.385 --> 00:11:19.005
because think of it like if you want to scale up, you need to have disks
227
00:11:19.519 --> 00:11:28.100
on-site, and sometimes you don't have these disks on-site. What happens if you ship the cluster to another when your customer is in another country?
228
00:11:28.835 --> 00:11:29.335
Other,
229
00:11:30.595 --> 00:11:31.815
another issue is,
230
00:11:32.195 --> 00:11:36.115
like, what happens when you need more RAM? On the cloud, you just,
231
00:11:36.675 --> 00:11:37.415
you just
232
00:11:37.930 --> 00:11:41.470
add more spin a cluster with instances with more RAM.
233
00:11:42.250 --> 00:11:47.625
But on prem, you need to check that you have enough slots in order to insert DIMM,
234
00:11:48.324 --> 00:11:50.425
DIMM sticks, memory sticks,
235
00:11:51.125 --> 00:11:51.945
into the
236
00:11:52.324 --> 00:11:54.264
into the class into each machine.
237
00:11:54.680 --> 00:11:56.779
And you need to do it, like, manually.
238
00:11:57.800 --> 00:11:59.339
So I think the
239
00:11:59.880 --> 00:12:00.620
the scaling,
240
00:12:01.800 --> 00:12:04.060
the 2 main difference of the scaling
241
00:12:05.455 --> 00:12:08.115
option, which is much easier on the cloud,
242
00:12:08.654 --> 00:12:10.035
and handling failures,
243
00:12:12.175 --> 00:12:13.075
hardware failures,
244
00:12:13.459 --> 00:12:14.120
which is
245
00:12:14.740 --> 00:12:15.240
also
246
00:12:15.860 --> 00:12:22.759
when you are the when you are an SAW of on prem, it's it's on you or on on on the first years
247
00:12:23.115 --> 00:12:26.575
that manage the issue once, but but while in the cloud,
248
00:12:27.195 --> 00:12:27.855
you just
249
00:12:28.395 --> 00:12:33.000
detect the issue. You need to detect the issue and then replace the the machine.
250
00:12:33.460 --> 00:12:37.080
The main so these were the 2 like, I came from on prem,
251
00:12:37.620 --> 00:12:39.320
not only from Kafka. Like,
252
00:12:40.325 --> 00:12:44.505
only for Spark application. It's it's pretty much the same issue,
253
00:12:45.045 --> 00:12:48.105
between on prem and on the cloud, these these 2 issues.
254
00:12:48.760 --> 00:12:52.380
However, the the benefit of, deploying on prem
255
00:12:53.000 --> 00:12:55.100
is the fact that you don't pay
256
00:12:55.560 --> 00:12:57.420
every for every hour.
257
00:12:58.665 --> 00:13:00.325
You just, pay once,
258
00:13:01.505 --> 00:13:04.404
there is I think today, there is a growing discussion,
259
00:13:05.800 --> 00:13:06.300
whether
260
00:13:06.920 --> 00:13:07.980
a cloud based
261
00:13:08.920 --> 00:13:09.420
companies,
262
00:13:10.360 --> 00:13:13.500
maybe they need to go on prem for some of their clusters.
263
00:13:14.865 --> 00:13:15.365
So
264
00:13:15.985 --> 00:13:18.165
it's the math the math
265
00:13:18.625 --> 00:13:19.125
behind
266
00:13:19.425 --> 00:13:20.165
this calculation,
267
00:13:21.690 --> 00:13:22.830
sometimes, I think,
268
00:13:23.290 --> 00:13:25.870
favors on prem even though
269
00:13:26.330 --> 00:13:27.230
you need to
270
00:13:27.770 --> 00:13:30.575
to handle this. The scaling is very tough,
271
00:13:32.015 --> 00:13:34.115
and it needs on you to to detect,
272
00:13:34.495 --> 00:13:37.555
hardware issues, but, maybe the cost sometimes
273
00:13:38.095 --> 00:13:39.060
justifies this.
274
00:13:39.700 --> 00:13:43.320
Yeah. The on premise versus in cloud debate is definitely
275
00:13:43.700 --> 00:13:46.120
ongoing and always very nuanced.
276
00:13:46.500 --> 00:13:48.279
And to your point, yeah, where
277
00:13:48.805 --> 00:13:49.545
on prem,
278
00:13:50.085 --> 00:13:50.585
the
279
00:13:51.045 --> 00:13:56.985
cost over time is much lower because you own the hardware, so you don't have to pay continuous upkeep for it. But
280
00:13:57.350 --> 00:14:01.910
there's the opportunity cost of having to move slower and more deliberately and,
281
00:14:02.790 --> 00:14:11.505
perhaps reducing the number of chances or experiments that you take because of the fact that it is so much lead time to bring in that hardware and scale up the cluster.
282
00:14:12.180 --> 00:14:17.000
And, also, on the point of Kafka, that's an interesting aspect as well where
283
00:14:17.779 --> 00:14:21.240
the my understanding of the way that Kafka itself is designed and
284
00:14:21.704 --> 00:14:23.325
some of the aspects of
285
00:14:24.105 --> 00:14:32.180
having to define up front the number of topics in order to accommodate a certain number of clients seems as though it lends itself more readily to that
286
00:14:32.560 --> 00:14:33.699
fixed installment
287
00:14:34.160 --> 00:14:38.500
on prem environment versus the cloud environment where you are incentivized
288
00:14:38.835 --> 00:14:41.175
to elastically scale up and down. And I'm wondering
289
00:14:41.635 --> 00:14:43.415
what you see as some of the
290
00:14:43.795 --> 00:14:48.560
challenges of bringing Kafka into the cloud because of that potential for elasticity in the clusters. I
291
00:14:56.320 --> 00:14:58.695
money like, use the cloud. Okay?
292
00:14:59.155 --> 00:15:07.700
The money is the the the money is the only reason why going on prem because it's indeed it's it's tough. But it's interesting that you mentioned,
293
00:15:08.880 --> 00:15:11.459
that that Kafka might sound
294
00:15:11.760 --> 00:15:13.860
a reasonable a good candidate
295
00:15:14.285 --> 00:15:19.505
to to be deployed on prem. And by the way, there are companies that most
296
00:15:20.045 --> 00:15:20.704
of their
297
00:15:21.750 --> 00:15:22.250
clusters,
298
00:15:23.190 --> 00:15:23.690
are
299
00:15:25.190 --> 00:15:31.345
that that host the open source analytics tool or messaging tools are deployed on the cloud, but Kafka
300
00:15:31.884 --> 00:15:32.545
is deployed
301
00:15:33.165 --> 00:15:33.985
on prem.
302
00:15:34.524 --> 00:15:36.279
I I know of
303
00:15:36.580 --> 00:15:39.640
2 not small Israeli companies that have their Kafka
304
00:15:40.019 --> 00:15:41.480
deployed on prem.
305
00:15:42.019 --> 00:15:42.519
And
306
00:15:43.334 --> 00:15:49.675
what like, from from my experience, because I was deploy I I was SRE for
307
00:15:50.190 --> 00:15:55.170
various customers when I was working for on prem. So I had I saw
308
00:15:55.950 --> 00:15:57.010
several examples.
309
00:15:59.035 --> 00:16:05.455
And the main thing that I saw in the main difficulty that I saw when provisioning a Kafka
310
00:16:06.100 --> 00:16:11.480
cluster by the way, most of the clusters were over provisioned. This is 1 problem in on prem, that you
311
00:16:11.860 --> 00:16:12.680
over provision
312
00:16:13.140 --> 00:16:14.760
because it takes time
313
00:16:15.425 --> 00:16:24.485
to purchase more disks or dings or brokers. So it's so it's hard to to say what what's going to be the the size of the cluster.
314
00:16:24.870 --> 00:16:29.290
But if I if I will take 1 1 factor that makes it hard
315
00:16:29.830 --> 00:16:31.530
to to provision
316
00:16:32.705 --> 00:16:36.325
upfront the Kafka cluster is the storage, is the retention.
317
00:16:37.025 --> 00:16:41.045
Because on on prem, you need to remember that most of, like,
318
00:16:41.460 --> 00:16:45.080
people on the cloud that were born to the cloud, let's say,
319
00:16:45.780 --> 00:16:49.800
that started working only and they work only on cloud environments,
320
00:16:50.295 --> 00:16:52.075
they tend to forget,
321
00:16:53.175 --> 00:16:55.835
that there are many on prem clusters.
322
00:16:56.535 --> 00:16:57.035
And
323
00:16:57.930 --> 00:17:01.550
much of these on prem clusters are deployed by, enterprises,
324
00:17:02.650 --> 00:17:05.870
or by companies that are not really high-tech, and they have
325
00:17:06.415 --> 00:17:07.795
big on prem clusters.
326
00:17:08.495 --> 00:17:12.575
And these customers, the the man the the owners of
327
00:17:13.455 --> 00:17:18.340
the customer, let's say, they are also the owners of the data center, and you provide them,
328
00:17:19.120 --> 00:17:20.179
the data center.
329
00:17:20.720 --> 00:17:21.220
And
330
00:17:21.520 --> 00:17:22.820
they have retention
331
00:17:23.520 --> 00:17:24.020
requirements
332
00:17:24.765 --> 00:17:29.185
that they serve not only how much time will it take you to recover from lag,
333
00:17:30.205 --> 00:17:34.305
but they have which is usually in in cloud and Internet companies.
334
00:17:34.630 --> 00:17:39.290
That's the retention, several hours, they want days or even a week.
335
00:17:40.070 --> 00:17:40.570
So
336
00:17:41.750 --> 00:17:43.930
this changes the whole picture because then
337
00:17:46.665 --> 00:17:47.565
when I started
338
00:17:48.745 --> 00:17:54.059
when I started as a necessary, I saw cluster that were just a a bunch
339
00:17:54.840 --> 00:17:56.380
of machines that cost
340
00:17:56.919 --> 00:17:58.860
sometimes tens of 1, 000 of dollars.
341
00:17:59.785 --> 00:18:05.085
And the CPU utilization is is, like, ridiculous. It's 10% user time.
342
00:18:05.705 --> 00:18:06.205
And
343
00:18:06.559 --> 00:18:09.860
the RAM usage, well, you never know what's your RAM usage, because
344
00:18:10.399 --> 00:18:15.940
we might later talk about the page cache, but it's very hard to understand what's the RAM usage in Kafka.
345
00:18:16.705 --> 00:18:19.285
But the disk storage was the utilization
346
00:18:19.665 --> 00:18:22.805
usage was really high because of this retention.
347
00:18:23.510 --> 00:18:28.970
And then you come to to a point, and I dedicated a chapter in the book only for this,
348
00:18:29.270 --> 00:18:30.809
to how how you
349
00:18:31.190 --> 00:18:32.410
whether you use RAID,
350
00:18:33.205 --> 00:18:36.105
RAID 1 plus RAID 0 or use JBOD.
351
00:18:36.645 --> 00:18:37.145
Because
352
00:18:37.445 --> 00:18:39.625
I can give a real example of a customer
353
00:18:40.005 --> 00:18:40.825
that had
354
00:18:41.179 --> 00:18:41.679
17,
355
00:18:42.059 --> 00:18:42.559
brokers.
356
00:18:43.980 --> 00:18:44.480
And
357
00:18:45.179 --> 00:18:47.280
just in order to satisfy the retention
358
00:18:47.660 --> 00:18:48.960
requirements of the customer,
359
00:18:49.340 --> 00:18:50.080
and then
360
00:18:50.545 --> 00:18:54.005
the customer decided to double the amount of retention
361
00:18:54.305 --> 00:18:58.085
because that's the order that he got. It was some guy government
362
00:18:59.289 --> 00:18:59.950
law enforcement.
363
00:19:00.730 --> 00:19:06.510
So you don't mess with them. They tell you, okay, I need to double my retention, so you need to satisfy this.
364
00:19:06.890 --> 00:19:07.390
And
365
00:19:09.125 --> 00:19:12.665
the the the first reaction was, okay, let's double the amount of brokers,
366
00:19:13.924 --> 00:19:15.580
but then I I,
367
00:19:16.520 --> 00:19:18.500
I convinced that the
368
00:19:18.920 --> 00:19:19.960
my managers that,
369
00:19:20.760 --> 00:19:22.700
you you we just need to
370
00:19:23.080 --> 00:19:24.455
switch from RAID 10,
371
00:19:24.774 --> 00:19:25.434
to JBOD.
372
00:19:25.735 --> 00:19:34.470
Now RAID 10 gives you double the amount of replication. So if you have replication factor of 3, you will get 6 copies per segment if you use rate
373
00:19:34.850 --> 00:19:36.389
10. But if you use JBOD,
374
00:19:37.409 --> 00:19:38.309
then you save
375
00:19:38.850 --> 00:19:43.575
then you save half of the disks, and then you don't need to add even 1 broker,
376
00:19:44.275 --> 00:19:44.775
just
377
00:19:46.035 --> 00:19:46.535
use
378
00:19:46.915 --> 00:19:56.190
the same number of disks. And if you want even more storage, you can just add more disks. But then you run into the issue of, okay, I don't have enough disks in my,
379
00:19:56.630 --> 00:19:57.450
in my drawer.
380
00:19:58.024 --> 00:20:02.125
So you need to add another drawer to each broker.
381
00:20:03.065 --> 00:20:09.309
These are these are things that on the cloud, you don't even think about them. People on the cloud even don't know about them. But
382
00:20:10.250 --> 00:20:14.110
if I will take 1 thing that that is really, like,
383
00:20:16.995 --> 00:20:21.335
that makes provisioning Kafka on prem to be tough is is the
384
00:20:21.715 --> 00:20:22.775
retention requirements.
385
00:20:23.315 --> 00:20:24.055
And managers
386
00:20:24.799 --> 00:20:29.220
for on prem clusters tend to be very sensitive for retention. They they want
387
00:20:30.240 --> 00:20:30.980
they want
388
00:20:31.520 --> 00:20:35.345
10 times the amount of retention or 20 times the amount of retention,
389
00:20:36.205 --> 00:20:38.465
that Internet companies have, for example.
390
00:20:39.165 --> 00:20:42.730
And bringing us around to the book that you wrote, which you mentioned,
391
00:20:43.110 --> 00:20:53.105
what was your motivation for bringing this all together in the written form, and what are the overall goals that you have for the book and the people who are reading it?
392
00:20:55.280 --> 00:20:55.780
So
393
00:20:56.720 --> 00:20:58.980
back in I remember myself as
394
00:20:59.280 --> 00:21:02.340
a as a back in engineering trying to
395
00:21:02.640 --> 00:21:04.020
deploy my application,
396
00:21:04.625 --> 00:21:06.565
my my streaming application on Kafka,
397
00:21:07.025 --> 00:21:12.805
and going to something didn't work in the Kafka on dev. So I remember going to the
398
00:21:13.340 --> 00:21:14.720
to the DevOps room,
399
00:21:15.020 --> 00:21:21.840
and, you know, everyone are afraid when you go to the DevOps room, because they're the most important the critical part
400
00:21:22.265 --> 00:21:23.725
of the of the organization.
401
00:21:24.184 --> 00:21:24.684
And
402
00:21:25.304 --> 00:21:28.845
I I asked them, okay. I don't know what's wrong with my Kafka.
403
00:21:29.600 --> 00:21:36.180
And I had no clue, and they didn't get an answer because they had no time. And then I went out to the room. I decided that I will know Kafka
404
00:21:37.525 --> 00:21:42.905
from all the angles. Like, that was my, like, really initial motivation. But
405
00:21:43.205 --> 00:21:44.345
if to be, like,
406
00:21:44.725 --> 00:21:45.785
more serious,
407
00:21:46.900 --> 00:21:49.800
when I started handling production issues in Kafka,
408
00:21:50.740 --> 00:21:52.200
and I went over to
409
00:21:52.740 --> 00:21:57.525
sites customer sites, I saw that people just didn't were clueless
410
00:21:58.545 --> 00:22:00.725
about what's wrong with Kafka. They blamed
411
00:22:01.665 --> 00:22:02.805
everyone blamed
412
00:22:03.590 --> 00:22:04.730
in the other part,
413
00:22:05.510 --> 00:22:06.010
the
414
00:22:06.310 --> 00:22:10.250
the devos blame consumers, consumer blame producer, producers blame Kafka.
415
00:22:11.030 --> 00:22:11.530
And
416
00:22:12.525 --> 00:22:14.765
so I I wanted to to
417
00:22:15.565 --> 00:22:18.865
and it it is in the middle of the data pipeline. So
418
00:22:19.350 --> 00:22:21.129
that was my motivation. Like,
419
00:22:21.750 --> 00:22:24.730
the best ROI for my time was definitely
420
00:22:25.110 --> 00:22:28.649
Kafka and Linux operating system, like understanding the metrics
421
00:22:28.975 --> 00:22:29.794
and how to
422
00:22:30.654 --> 00:22:31.154
diagnose
423
00:22:31.695 --> 00:22:34.595
using Linux metrics, diagnose problems in Kafka.
424
00:22:35.695 --> 00:22:39.140
After now this is was why I wanted to
425
00:22:40.080 --> 00:22:40.900
learn Kafka
426
00:22:41.680 --> 00:22:43.220
from real production issues.
427
00:22:43.815 --> 00:22:46.635
My motivation for the book was that these,
428
00:22:47.655 --> 00:22:52.395
support engineers and DevOps and developers that encounter issues in Kafka,
429
00:22:52.860 --> 00:22:56.800
which is not a managed service, so they manage it themselves,
430
00:22:57.660 --> 00:22:59.840
will have a a cookbook
431
00:23:00.674 --> 00:23:01.255
for understanding,
432
00:23:02.275 --> 00:23:03.735
like, like recipes,
433
00:23:04.275 --> 00:23:10.890
understanding how to how to handle production issues in Kafka. And this is why the book is split it into
434
00:23:11.270 --> 00:23:12.570
3, sections,
435
00:23:13.190 --> 00:23:16.330
3 logical sections. It's the the data section,
436
00:23:16.875 --> 00:23:20.575
the Linux OS section, and the Kafka metrics section section.
437
00:23:21.035 --> 00:23:21.775
And, also,
438
00:23:22.155 --> 00:23:26.970
for, most of it cloud based based, but but also 2 chapters dedicated
439
00:23:27.510 --> 00:23:29.530
specifically to to on prem.
440
00:23:29.910 --> 00:23:30.410
Because,
441
00:23:31.830 --> 00:23:32.070
the
442
00:23:35.965 --> 00:23:36.465
duplicated
443
00:23:36.845 --> 00:23:39.745
when I was, starting to work on the cloud.
444
00:23:40.200 --> 00:23:48.460
On the cloud, it's just easier because you have, you know, you have monitoring in front of your eyes. You don't need to get logs from from, support engineers.
445
00:23:49.405 --> 00:23:49.905
But,
446
00:23:50.765 --> 00:23:52.465
so I I saw that that
447
00:23:53.165 --> 00:23:56.865
that there there was no book. There was nothing that even resembled
448
00:23:57.330 --> 00:24:01.750
a book about real production issues in Kafka, and there is a real need,
449
00:24:03.170 --> 00:24:04.790
for for this because
450
00:24:05.235 --> 00:24:08.934
there are so many Kafka deployments out there, whether on prem or the cloud.
451
00:24:09.555 --> 00:24:10.055
And
452
00:24:10.675 --> 00:24:20.480
and people are just you know, I I talked to 1 CTO of a start, Israeli startup who told me, I must use Kafka because that's the message of us today, but it's hell.
453
00:24:21.180 --> 00:24:23.360
Okay? Managing it is so tough.
454
00:24:24.544 --> 00:24:27.924
And I heard it after he said it from different people,
455
00:24:29.825 --> 00:24:31.684
different manager ops managers,
456
00:24:32.304 --> 00:24:32.804
or
457
00:24:33.580 --> 00:24:36.560
tech tech tech leaders at some small companies,
458
00:24:37.420 --> 00:24:37.920
and
459
00:24:38.620 --> 00:24:40.400
understood that, it's just
460
00:24:41.605 --> 00:24:43.625
a common common problem.
461
00:24:44.325 --> 00:24:45.225
And then
462
00:24:45.605 --> 00:24:47.225
then the idea for the book,
463
00:24:49.149 --> 00:24:53.809
I I mean, it didn't come to me. I got an offer to to write the book, and then,
464
00:24:54.990 --> 00:24:57.570
it took me some months because it's a lot of work.
465
00:24:58.165 --> 00:25:03.625
Then I then I decided to compile to to gather all the stuff that I compiled for
466
00:25:04.725 --> 00:25:07.385
during the 6 years of me working in Kafka.
467
00:25:09.110 --> 00:25:12.170
And that was my my motivation. Like,
468
00:25:12.550 --> 00:25:18.225
I'm not saying that people shouldn't use Amazon MSK, yeah, or, yes, or Confluent or Ivan,
469
00:25:18.845 --> 00:25:19.905
but I think that
470
00:25:20.205 --> 00:25:23.505
for for for those who already manage Kafka,
471
00:25:24.125 --> 00:25:25.665
they should have a better
472
00:25:26.205 --> 00:25:26.705
guide
473
00:25:27.840 --> 00:25:29.700
than than no guide, first of all.
474
00:25:30.080 --> 00:25:32.100
And, also, for those who
475
00:25:32.480 --> 00:25:38.115
consider they they manage their own Kafka and consent, say, okay. It's tough. It's too it's too much. It's too tough.
476
00:25:38.495 --> 00:25:39.555
We need to pay
477
00:25:39.935 --> 00:25:41.955
license and move to some managed service.
478
00:25:42.975 --> 00:25:44.675
I think that if they have
479
00:25:45.169 --> 00:25:46.629
the core the right guide,
480
00:25:47.169 --> 00:25:48.789
then it can save them money.
481
00:25:49.490 --> 00:25:49.990
And,
482
00:25:50.769 --> 00:25:56.855
I work in the open source with open source for 20 years, and, like, I see this as my small contribution
483
00:25:57.715 --> 00:25:59.335
to the open source community.
484
00:26:00.515 --> 00:26:07.330
To the point of saving money and whether to run your own Kafka cluster or use a managed service,
485
00:26:07.630 --> 00:26:09.410
there are a lot of considerations
486
00:26:09.870 --> 00:26:11.809
that go into all of that
487
00:26:12.304 --> 00:26:13.764
as well as the
488
00:26:14.065 --> 00:26:14.804
use cases
489
00:26:15.105 --> 00:26:17.924
of what you're going to be building on top of Kafka
490
00:26:18.625 --> 00:26:19.845
and what are the
491
00:26:20.250 --> 00:26:22.670
sizing and scaling requirements. I'm wondering
492
00:26:23.050 --> 00:26:24.350
for people who are
493
00:26:24.730 --> 00:26:29.630
in that position of deciding, do I want to use Kafka? How do I want to use Kafka
494
00:26:29.965 --> 00:26:30.784
as far as
495
00:26:31.164 --> 00:26:33.904
self managed or managed service? And
496
00:26:34.205 --> 00:26:39.345
what are the parameters along which I need to project what my cost is going to be? What are the
497
00:26:39.980 --> 00:26:46.160
different elements that they need to be considering and planning for as they start to evaluate and do those initial deployments?
498
00:26:47.505 --> 00:26:50.405
So so first of all, you need to understand your traffic.
499
00:26:50.865 --> 00:26:51.845
Okay? So,
500
00:26:53.345 --> 00:26:55.205
let's assume you are not a small
501
00:26:55.665 --> 00:27:00.529
company and that you you decide you you you are the decision point whether
502
00:27:00.990 --> 00:27:09.125
you need to use Kafka, first of all. And then if you use Kafka, whether you're going to use cloud manage or you manage it yourself. So,
503
00:27:10.065 --> 00:27:19.630
regard regarding alternatives to Kafka, I I'm not aware of any alternatives. However, I never search for alternative because Kafka is just everywhere.
504
00:27:20.730 --> 00:27:23.870
So let's say you chose Kafka and you have enough traffic
505
00:27:24.355 --> 00:27:25.015
so that
506
00:27:26.035 --> 00:27:27.095
you can build
507
00:27:27.715 --> 00:27:29.235
even a minimal cluster of,
508
00:27:29.875 --> 00:27:31.175
33 brokers. Yeah?
509
00:27:31.760 --> 00:27:42.275
But let's assume you have more, and let's assume you have multiple Kafka clusters per each team or service or a group or whatever. So in that case, first of all, you need to know your traffic,
510
00:27:43.054 --> 00:27:44.275
the number of topics,
511
00:27:44.655 --> 00:27:47.875
the per to per cluster, the number of partitions,
512
00:27:48.810 --> 00:27:49.310
and
513
00:27:49.770 --> 00:27:51.150
who who are your consumers,
514
00:27:52.250 --> 00:27:53.870
per topic, who are your producers.
515
00:27:54.650 --> 00:27:56.825
So understanding the producing rate,
516
00:27:57.225 --> 00:27:58.524
the consuming rate,
517
00:27:58.985 --> 00:27:59.485
and
518
00:27:59.945 --> 00:28:01.725
then you need to provision
519
00:28:02.264 --> 00:28:09.730
a cluster. Let's say, even before you know whether it's cloud based or not, you need to understand how much CPU you're going to require,
520
00:28:10.190 --> 00:28:19.245
how much RAM in order to support not the Kafka process itself, but the page cache because you want data to be read from the page cache and not from the disk,
521
00:28:19.625 --> 00:28:21.245
and allocate enough disks,
522
00:28:21.785 --> 00:28:23.725
cheap disks in order to support
523
00:28:24.025 --> 00:28:25.245
as cheaper as possible,
524
00:28:26.410 --> 00:28:28.990
to handle the the retention requirement.
525
00:28:29.929 --> 00:28:30.429
And
526
00:28:31.290 --> 00:28:31.790
now
527
00:28:32.330 --> 00:28:33.790
after you have this estimation
528
00:28:34.170 --> 00:28:35.070
and you know
529
00:28:35.735 --> 00:28:39.595
which how many brokers you have and what's the size of the broker,
530
00:28:39.975 --> 00:28:41.275
now it depends
531
00:28:41.655 --> 00:28:43.115
the decision whether it's
532
00:28:43.850 --> 00:28:47.070
cloud based or not. If you are cloud based, you you will
533
00:28:47.450 --> 00:28:47.950
usually
534
00:28:48.649 --> 00:28:49.149
pick
535
00:28:49.450 --> 00:28:50.269
cloud based.
536
00:28:51.505 --> 00:28:58.005
But again, some companies decide, okay, I will put my Kafka on prem because I know it's it cannot be spot instances.
537
00:28:59.260 --> 00:29:03.440
But most of the companies which are cloud based will will deploy it on the cloud, and
538
00:29:03.820 --> 00:29:05.120
they will deploy it
539
00:29:06.385 --> 00:29:09.525
probably on on demand instances if they're on AWS.
540
00:29:10.545 --> 00:29:11.025
Some
541
00:29:11.905 --> 00:29:17.950
if if they have a strong DevOps team, they will, deploy it on a on a Kubernetes,
542
00:29:18.929 --> 00:29:19.630
maybe. But,
543
00:29:21.130 --> 00:29:21.630
but
544
00:29:21.985 --> 00:29:29.284
I I don't have an experience with Kafka and Kubernetes, so I can't say anything about it. I also didn't write about it, of course.
545
00:29:30.870 --> 00:29:31.270
But,
546
00:29:31.910 --> 00:29:42.175
so so and if you're on prem, you will deploy it on prem, of course. Now the question may of whether to deploy it on like, purchase a a license
547
00:29:42.715 --> 00:29:46.470
for, for Confluent or for Amazon MSK,
548
00:29:48.050 --> 00:29:48.950
or for Ivan.
549
00:29:49.730 --> 00:29:54.915
These are some of the many services. Or use it or or manage it yourself.
550
00:29:56.335 --> 00:29:58.355
It really depends on on the
551
00:29:59.750 --> 00:30:03.290
on how your ops team, whether it's DevOps or DataOps,
552
00:30:04.470 --> 00:30:05.850
know how to
553
00:30:06.310 --> 00:30:06.810
handle
554
00:30:07.430 --> 00:30:08.490
production issues.
555
00:30:09.965 --> 00:30:12.545
And let's say, you know, ops teams are
556
00:30:13.245 --> 00:30:16.705
are rare. Good ops teams, I mean, are are rare.
557
00:30:19.530 --> 00:30:20.890
And and I think that if you
558
00:30:22.570 --> 00:30:23.470
if, like,
559
00:30:23.985 --> 00:30:26.485
you need good mon an excellent monitoring
560
00:30:26.945 --> 00:30:30.005
of Kafka. And when I say an excellent time, I don't mean
561
00:30:30.385 --> 00:30:31.684
hundreds of metrics.
562
00:30:32.230 --> 00:30:36.809
Okay? I mean, I mean, a minimal subs a minimal set of metrics
563
00:30:37.350 --> 00:30:38.570
that will show you
564
00:30:39.029 --> 00:30:40.970
where production issues can occur.
565
00:30:41.395 --> 00:30:47.015
And in my book, I have 2 chapters, 1 on producer metrics, 1 on consumer metrics,
566
00:30:47.555 --> 00:30:48.055
and
567
00:30:49.770 --> 00:30:50.750
is 3 chapters
568
00:30:51.210 --> 00:30:51.710
on
569
00:30:52.090 --> 00:30:58.405
CPU RAM and Disks for the broker themselves. So I think, like, half of the book talks about
570
00:30:59.665 --> 00:31:01.045
how to diagnose issues,
571
00:31:01.345 --> 00:31:01.845
and
572
00:31:02.760 --> 00:31:03.660
and from this,
573
00:31:04.360 --> 00:31:05.740
the the reader can understand
574
00:31:06.360 --> 00:31:08.620
what to monitor. So you need an excellent
575
00:31:09.000 --> 00:31:11.315
monitoring, but not too large. You need,
576
00:31:12.175 --> 00:31:13.155
like, specific
577
00:31:14.015 --> 00:31:14.515
monitoring,
578
00:31:15.775 --> 00:31:19.795
in order to deploy it yourself. Because if you are blind in Kafka,
579
00:31:20.419 --> 00:31:23.000
you will pay a big price. You will have downtime,
580
00:31:23.620 --> 00:31:25.480
and you will just lose data.
581
00:31:26.179 --> 00:31:27.960
So you must have an excellent,
582
00:31:28.580 --> 00:31:29.720
excellent monitoring
583
00:31:30.535 --> 00:31:36.395
and a team who knows how to how to manage the Kafka cluster. And,
584
00:31:37.095 --> 00:31:37.595
otherwise,
585
00:31:38.270 --> 00:31:38.930
to choose,
586
00:31:40.190 --> 00:31:48.915
choose cloud based or there are again, because it's the knowledge of handling Kafka clusters is is pretty rare.
587
00:31:49.295 --> 00:31:51.715
What happens is that some
588
00:31:52.095 --> 00:31:53.395
places, some customers,
589
00:31:54.095 --> 00:31:55.715
and they saw it in on prem,
590
00:31:56.490 --> 00:31:57.310
They just
591
00:31:58.090 --> 00:32:00.910
sometimes Kafka holds, and they lose data because
592
00:32:01.210 --> 00:32:02.830
there's not enough knowledge.
593
00:32:03.975 --> 00:32:09.115
That's not, like, that's not, of course, the situation at my current company. We have an excellent,
594
00:32:10.774 --> 00:32:17.340
DevOps team, and its team leader was also the technical editor for my book. And he affected
595
00:32:17.720 --> 00:32:18.380
a lot
596
00:32:19.645 --> 00:32:22.145
on the content and also on how to,
597
00:32:22.605 --> 00:32:23.745
like, what
598
00:32:24.445 --> 00:32:27.105
some parts of what to focus. His name is Oronon,
599
00:32:27.405 --> 00:32:28.225
and I
600
00:32:29.530 --> 00:32:30.590
I am very grateful
601
00:32:30.970 --> 00:32:32.429
for him to to
602
00:32:33.049 --> 00:32:34.190
to that he
603
00:32:34.809 --> 00:32:36.030
invested the time
604
00:32:36.664 --> 00:32:39.245
to to read and to edit the book.
605
00:32:39.784 --> 00:32:45.164
But, again, it's at some companies, you have an excellent DevOps team, and some customers,
606
00:32:45.750 --> 00:32:46.410
you don't
607
00:32:46.870 --> 00:32:50.090
have. And if you're willing to lose data at some time, then
608
00:32:50.790 --> 00:32:53.210
use the open source. And if not, then
609
00:32:53.904 --> 00:32:54.565
go to
610
00:32:56.304 --> 00:32:58.965
then pay the license for Managed Kafka.
611
00:33:01.930 --> 00:33:03.150
Data lakes are notoriously
612
00:33:03.450 --> 00:33:03.950
complex.
613
00:33:04.410 --> 00:33:20.110
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte scale SQL analytics fast at a fraction of the cost of traditional methods so that you can meet all of your data needs ranging from AI to data applications to complete analytics.
614
00:33:20.570 --> 00:33:30.725
Trusted by teams of all sizes, including Comcast and DoorDash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises.
615
00:33:31.400 --> 00:33:34.059
And Starburst does all of this on an open architecture
616
00:33:34.360 --> 00:33:40.780
with first class support for Apache Iceberg, Delta Lake, and Hoody, so you always maintain ownership of your data.
617
00:33:41.495 --> 00:33:45.115
Want to see Starburst in action? Go to data engineering podcast.com/starburst
618
00:33:46.615 --> 00:33:47.515
and get $500
619
00:33:47.815 --> 00:33:53.130
in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino.
620
00:33:55.350 --> 00:33:57.290
And on that note of
621
00:33:57.590 --> 00:33:58.410
data loss
622
00:33:58.870 --> 00:34:00.090
and the
623
00:34:00.554 --> 00:34:02.015
cluster uptime and stability,
624
00:34:03.035 --> 00:34:05.695
what are some of the failure conditions
625
00:34:06.155 --> 00:34:12.170
that cluster operators need to be thinking about that might lead to data loss and some of the ways to
626
00:34:12.550 --> 00:34:14.090
mitigate that and
627
00:34:14.550 --> 00:34:18.170
plan for it in order to be able to reduce the time to resolution?
628
00:34:19.625 --> 00:34:20.925
In terms of preventing,
629
00:34:21.865 --> 00:34:23.805
data loss, let's start with prevention
630
00:34:24.185 --> 00:34:25.565
and then go to to
631
00:34:26.720 --> 00:34:28.020
to what can cause,
632
00:34:28.560 --> 00:34:31.220
data loss. First of all, you have a replication.
633
00:34:32.800 --> 00:34:36.340
Now replication goes side by side with the size of the cluster
634
00:34:36.905 --> 00:34:38.925
because the higher the replication factor,
635
00:34:39.305 --> 00:34:42.765
then each segment will have more copies on different brokers.
636
00:34:43.225 --> 00:34:45.645
However, you will need more storage, and more storage
637
00:34:47.880 --> 00:34:48.380
means,
638
00:34:49.800 --> 00:34:50.300
like,
639
00:34:50.760 --> 00:34:52.300
more storage means sometimes,
640
00:34:52.920 --> 00:34:53.740
more brokers.
641
00:34:54.315 --> 00:34:57.115
And even if not more brokers, it means more more,
642
00:34:58.075 --> 00:34:58.815
more disks.
643
00:34:59.275 --> 00:35:00.975
So it will cost you more,
644
00:35:01.995 --> 00:35:06.370
higher replication factor. Also, you can add another level of,
645
00:35:07.110 --> 00:35:07.610
assurance
646
00:35:07.910 --> 00:35:08.410
and
647
00:35:09.030 --> 00:35:17.625
deploy RAID. Let's say RAID 10, RAID 1 plus RAID 0, and this will double the amount of, of storage that you that you need.
648
00:35:17.925 --> 00:35:18.425
So,
649
00:35:18.885 --> 00:35:20.345
replication is 1 thing.
650
00:35:20.970 --> 00:35:22.829
Making sure that your your producer,
651
00:35:24.010 --> 00:35:24.970
they have a
652
00:35:25.609 --> 00:35:27.390
they receive a axe acknowledgments
653
00:35:27.930 --> 00:35:28.829
from the broker
654
00:35:29.315 --> 00:35:31.335
is another thing, although it may affect
655
00:35:32.675 --> 00:35:33.175
latency.
656
00:35:34.755 --> 00:35:35.815
Retention policy.
657
00:35:36.170 --> 00:35:38.910
Retention policy, it's pretty simple policy,
658
00:35:39.290 --> 00:35:44.990
and you wouldn't believe the number of times that that consumers lose data because of it. Because
659
00:35:45.655 --> 00:35:50.075
you can define it by time or by size or by both, and then,
660
00:35:50.695 --> 00:35:53.435
the threshold is the minimal between both. And
661
00:35:54.340 --> 00:35:57.080
and and, if you define it by size
662
00:35:57.700 --> 00:36:00.840
and suddenly the topic size the the traffic increases
663
00:36:01.674 --> 00:36:08.575
even by a small amount of percentage and you your consumer lags, then you will lose data, or some consumers will lose data.
664
00:36:09.035 --> 00:36:09.535
So
665
00:36:11.020 --> 00:36:11.520
if,
666
00:36:11.980 --> 00:36:12.480
like,
667
00:36:13.340 --> 00:36:17.360
if the audience will take 1 thing from regarding retention is that
668
00:36:18.525 --> 00:36:19.345
highly consider
669
00:36:19.885 --> 00:36:22.625
configuring retention by, by time
670
00:36:23.485 --> 00:36:25.905
and and not by by size, because
671
00:36:26.420 --> 00:36:29.560
you you really don't know the traffic at any point of time.
672
00:36:29.940 --> 00:36:33.640
And and some producer can increase the traffic by
673
00:36:34.255 --> 00:36:36.035
by multiply of 10,
674
00:36:36.894 --> 00:36:37.394
because,
675
00:36:37.855 --> 00:36:41.154
there was some filter and the developer just removed that filter.
676
00:36:41.840 --> 00:36:44.500
And and suddenly you get 10 times the traffic.
677
00:36:44.880 --> 00:36:45.380
However,
678
00:36:46.400 --> 00:36:47.380
if again,
679
00:36:47.680 --> 00:36:50.500
nothing is simple in Kafka. If you configure retention,
680
00:36:51.145 --> 00:36:51.645
by
681
00:36:52.025 --> 00:36:53.085
time, then
682
00:36:53.625 --> 00:36:56.285
you might get into storage, a 100%
683
00:36:56.585 --> 00:36:58.525
storage. So, again, this equilibrium
684
00:36:58.825 --> 00:36:59.325
between
685
00:36:59.869 --> 00:37:00.590
retention size
686
00:37:01.390 --> 00:37:05.410
retention time and and storage. Again, we reach storage.
687
00:37:07.070 --> 00:37:07.970
How do failures?
688
00:37:08.285 --> 00:37:10.944
Monitor hardware failures. Now on on prem,
689
00:37:11.484 --> 00:37:13.184
you can just check the status
690
00:37:13.565 --> 00:37:18.390
of the disks. I repeat again. Disk, disk, disk. This is the the most critical
691
00:37:18.790 --> 00:37:20.410
the cheapest part of Kafka,
692
00:37:20.950 --> 00:37:23.609
but the most critical part of Kafka, and
693
00:37:24.070 --> 00:37:26.695
and the cause for many, many problems. So
694
00:37:27.575 --> 00:37:33.275
like, on on prem, you can you there are tools that you can monitor the state of the of the disk,
695
00:37:34.215 --> 00:37:35.035
and just
696
00:37:35.575 --> 00:37:36.635
a very, like,
697
00:37:37.240 --> 00:37:38.940
recommendation that can save
698
00:37:39.640 --> 00:37:41.660
several clusterizers, I assume.
699
00:37:42.359 --> 00:37:44.700
If a disk becomes read only,
700
00:37:45.345 --> 00:37:47.605
so you can just read and not write,
701
00:37:48.305 --> 00:37:50.565
usually, it means you need to replace the disk,
702
00:37:51.425 --> 00:38:00.950
because it, even running FSEK on on on the disk will help maybe for several minutes, hours, or days. But after that, it will become again read only,
703
00:38:01.730 --> 00:38:05.164
and producers cannot write to it, and it will create a whole
704
00:38:05.545 --> 00:38:07.164
mess. So monitor the disks.
705
00:38:07.625 --> 00:38:09.085
On prem, you have the the
706
00:38:10.184 --> 00:38:11.325
tool to to check
707
00:38:11.700 --> 00:38:12.020
its,
708
00:38:12.580 --> 00:38:13.080
itself.
709
00:38:13.540 --> 00:38:16.040
And on the cloud, you can check,
710
00:38:17.380 --> 00:38:19.480
you can check the the disk utilization.
711
00:38:19.780 --> 00:38:20.760
So the IOSTAT
712
00:38:21.755 --> 00:38:23.775
tool, the Linux tool called IOSTAT,
713
00:38:24.395 --> 00:38:25.855
IOSTAT minus x,
714
00:38:26.635 --> 00:38:28.255
print every 1 second
715
00:38:28.680 --> 00:38:31.820
can show you that if if the disk utilization is 100%
716
00:38:32.360 --> 00:38:33.180
all the time,
717
00:38:33.480 --> 00:38:35.900
then something is wrong with your with your disk.
718
00:38:36.405 --> 00:38:43.385
Another metric is, what there there are a lot of metrics, but the the another 1 that happens to be around every
719
00:38:43.940 --> 00:38:51.640
production issue is the system time, the CPU system time. There are 4 main CPU metrics. You have the user time, system time,
720
00:38:52.744 --> 00:39:01.630
I wait time, which is either disk time, wait time for disk or wait time for network and context switches. If the system time goes, let's say, about
721
00:39:02.090 --> 00:39:03.310
10%, then
722
00:39:03.610 --> 00:39:08.565
you should suspect something this is not good in your cluster. If your contact switches
723
00:39:10.065 --> 00:39:12.405
time reaches, let's say, more than 3, 4%,
724
00:39:13.425 --> 00:39:13.925
then
725
00:39:14.770 --> 00:39:15.270
check
726
00:39:16.210 --> 00:39:17.589
check what happens there.
727
00:39:18.450 --> 00:39:19.030
A common
728
00:39:19.570 --> 00:39:21.750
cause, by the way, to a high
729
00:39:22.369 --> 00:39:23.430
context switch
730
00:39:24.464 --> 00:39:25.285
time percentage
731
00:39:25.825 --> 00:39:26.644
could be,
732
00:39:27.184 --> 00:39:31.690
that you just have too many disk kind of threads compared to the number of disks that you have.
733
00:39:32.490 --> 00:39:36.829
And this is if you want like, if to if you want to be, like,
734
00:39:38.089 --> 00:39:43.295
like, very cautious, you can you can regular backup regularly your data,
735
00:39:44.155 --> 00:39:51.100
either by using the feature of a tiered storage of Kafka, which I admit I I didn't use until now,
736
00:39:51.400 --> 00:39:52.460
or you can just
737
00:39:53.560 --> 00:39:55.420
read add another consumer
738
00:39:55.720 --> 00:39:56.460
and just
739
00:39:56.994 --> 00:39:59.815
take the data from the topic and persist it to some
740
00:40:00.434 --> 00:40:02.615
cheaper storage, like HDFS,
741
00:40:04.090 --> 00:40:04.750
or s
742
00:40:05.130 --> 00:40:08.670
3, where the storage is separate for the from the compute.
743
00:40:09.290 --> 00:40:10.430
So this is regarding
744
00:40:11.050 --> 00:40:13.305
some ways to prevent data loss.
745
00:40:14.425 --> 00:40:18.365
But in order to but these these are the these are the, like,
746
00:40:19.224 --> 00:40:20.444
what happens after
747
00:40:20.744 --> 00:40:22.605
after you already have a problem.
748
00:40:23.359 --> 00:40:26.099
But the question is how you prevent the problems.
749
00:40:26.720 --> 00:40:30.880
And from my experience with Kafka in production in the last,
750
00:40:31.599 --> 00:40:34.995
6 years is that Kafka cluster talk before
751
00:40:35.695 --> 00:40:38.435
they hint you before they they creep you down.
752
00:40:38.735 --> 00:40:39.875
They tell you stuff,
753
00:40:40.380 --> 00:40:42.720
and they tell you this through the disk utilization,
754
00:40:43.500 --> 00:40:48.640
the IOSTA tool, which you can see using the IOSTA tool. They tell you this using top command.
755
00:40:49.275 --> 00:40:50.815
Check the system time,
756
00:40:51.195 --> 00:40:55.535
your CPU system time. Check the CPU context switch, CPU wait time.
757
00:40:56.050 --> 00:40:58.470
If if you have GC issues in your Kafka
758
00:40:59.170 --> 00:40:59.670
process,
759
00:41:00.210 --> 00:41:02.150
then it will tell you this through
760
00:41:03.545 --> 00:41:07.245
spikes in the user time during a full GC. You can use the JSTAD
761
00:41:07.705 --> 00:41:11.405
tool, which is part of the JDK tools in order to check for
762
00:41:12.420 --> 00:41:14.760
full GC the frequency of full GC.
763
00:41:15.940 --> 00:41:21.395
Again, it's it's like, this is what half the book talks about half the chapters almost talk about
764
00:41:22.275 --> 00:41:23.415
how to, like,
765
00:41:23.955 --> 00:41:26.135
various cases that they stumbled upon
766
00:41:27.395 --> 00:41:27.895
regarding
767
00:41:28.355 --> 00:41:29.975
what can lead you to
768
00:41:30.819 --> 00:41:32.200
to to data loss.
769
00:41:32.660 --> 00:41:35.720
I had, I had a colleague asking me,
770
00:41:36.420 --> 00:41:42.555
I I gave him a copy of the book and he has he asked me, okay, but you didn't talk about the the issue of un replicated
771
00:41:43.895 --> 00:41:44.395
partitions.
772
00:41:45.575 --> 00:41:47.995
And I replied to him, well,
773
00:41:48.619 --> 00:41:49.119
every
774
00:41:49.819 --> 00:41:53.519
problem in Kafka can result in under replicated partitions
775
00:41:54.220 --> 00:42:00.865
or in in partitions that are not. They don't appear in the list of in sync replicas in the ISR.
776
00:42:01.565 --> 00:42:02.065
So
777
00:42:03.725 --> 00:42:11.440
and there are tens of problems, which might not even be on the Kafka itself. Like, it could be in the operating system because something
778
00:42:12.380 --> 00:42:12.880
is,
779
00:42:13.505 --> 00:42:14.005
like
780
00:42:14.305 --> 00:42:17.125
sometimes it's even not in the Kafka. Again, like,
781
00:42:18.065 --> 00:42:20.724
if you deploy another service, if you deploy
782
00:42:21.585 --> 00:42:22.325
a antivirus
783
00:42:23.100 --> 00:42:23.600
antivirus,
784
00:42:24.300 --> 00:42:25.200
okay, or firewall,
785
00:42:25.660 --> 00:42:26.720
and they scan
786
00:42:27.500 --> 00:42:32.845
the the segments all the time, and you will have high disk utilization, this is not the fault of Kafka,
787
00:42:34.045 --> 00:42:36.145
but it will cripple down your Kafka.
788
00:42:36.845 --> 00:42:37.345
So
789
00:42:37.805 --> 00:42:44.550
and there are some I saw on prem clusters where you shut down the firewall, and then it comes up again because of some policy.
790
00:42:45.490 --> 00:42:45.990
So
791
00:42:46.530 --> 00:42:52.655
so so you need to remember that not always you run alone on the on the broker, but asking,
792
00:42:53.915 --> 00:42:57.135
like, how like, what can cause data loss
793
00:42:57.595 --> 00:42:58.095
is,
794
00:42:59.170 --> 00:43:05.430
like, is like asking what should I monitor in Kafka and and and, how to deal with it. And that's
795
00:43:05.865 --> 00:43:07.885
that's a topic of the book in general.
796
00:43:09.065 --> 00:43:10.845
And given your experience
797
00:43:11.145 --> 00:43:13.645
both working as a back end engineer and
798
00:43:14.210 --> 00:43:16.390
operating Kafka clusters at scale.
799
00:43:17.089 --> 00:43:19.430
I'm wondering if you were to
800
00:43:19.730 --> 00:43:21.109
be in the room today
801
00:43:21.410 --> 00:43:27.855
redesigning Kafka from the ground up, what are some of the aspects of the system design that you might choose
802
00:43:28.315 --> 00:43:28.974
to revisit
803
00:43:29.355 --> 00:43:30.335
or revitalize?
804
00:43:34.300 --> 00:43:36.240
I think the design is is,
805
00:43:37.500 --> 00:43:40.035
makes a lot of sense, the log based
806
00:43:40.815 --> 00:43:41.315
approach.
807
00:43:42.655 --> 00:43:44.035
I I don't have many
808
00:43:44.575 --> 00:43:45.075
changes.
809
00:43:45.990 --> 00:43:49.290
I don't think of any change other than 1 change which is,
810
00:43:49.750 --> 00:43:52.790
for me, it's a bug. I don't know if they declare it as a feat
811
00:43:53.670 --> 00:43:54.490
as a feature
812
00:43:55.234 --> 00:43:56.775
that, they spread.
813
00:43:57.075 --> 00:43:59.894
If you have more than 1 disk, then the partitions
814
00:44:00.194 --> 00:44:01.255
are being spread,
815
00:44:02.994 --> 00:44:04.020
among the disks,
816
00:44:04.420 --> 00:44:06.440
given the number of partition per disk
817
00:44:06.740 --> 00:44:10.120
and not given the numb the the amount of storage per disk,
818
00:44:10.740 --> 00:44:13.875
which I I don't think that the Kafka community understands
819
00:44:14.335 --> 00:44:15.315
how it affects,
820
00:44:16.255 --> 00:44:18.035
some Kafka cluster owners
821
00:44:18.335 --> 00:44:20.195
to decide whether they go raid
822
00:44:20.780 --> 00:44:22.240
or, or j bot.
823
00:44:22.780 --> 00:44:28.160
It's it's like something that if it will be fixed, the the amount of cost that will be saved
824
00:44:29.565 --> 00:44:33.345
for disk and maybe for cluster will be pretty big,
825
00:44:34.365 --> 00:44:34.865
because
826
00:44:35.244 --> 00:44:35.905
there are
827
00:44:36.640 --> 00:44:44.260
there are ops teams that say, okay. I don't want to go JBOD because because of this. I don't want to handle the spread of the data
828
00:44:44.625 --> 00:44:45.445
among the disk.
829
00:44:46.065 --> 00:44:51.365
But other than this feature, feature slash bug, of spreading the data evenly
830
00:44:51.990 --> 00:44:54.410
based according to storage and not
831
00:44:55.270 --> 00:44:56.410
not number of partitions.
832
00:44:58.870 --> 00:45:04.875
I do I I must I I must admit that I'm not in the I'm not in the applicative side
833
00:45:05.335 --> 00:45:07.035
anymore for a long time because
834
00:45:07.495 --> 00:45:10.875
to to be honest, it it's it interests me much less
835
00:45:11.260 --> 00:45:12.480
than the upside.
836
00:45:13.260 --> 00:45:13.760
So
837
00:45:14.060 --> 00:45:16.079
my focus is on the upside, and
838
00:45:18.195 --> 00:45:20.375
and then it comes down to, like,
839
00:45:21.235 --> 00:45:23.815
this this question comes down to, like
840
00:45:24.435 --> 00:45:26.455
it could be asked regarding every
841
00:45:27.150 --> 00:45:29.490
big data cluster, but in Kafka,
842
00:45:30.990 --> 00:45:32.130
it's really hard,
843
00:45:33.070 --> 00:45:36.610
to detect to understand the root cause of production issues.
844
00:45:37.214 --> 00:45:39.075
To be honest, I don't know why,
845
00:45:40.095 --> 00:45:44.515
fix a problem in Druid or Spark or Spark Streaming or Treyno
846
00:45:44.900 --> 00:45:47.080
takes much less time than understanding
847
00:45:48.180 --> 00:45:50.580
why why, why data was,
848
00:45:51.220 --> 00:45:52.520
it was lost on Kafka
849
00:45:54.025 --> 00:45:56.525
or why the disk utilization is so high.
850
00:45:57.145 --> 00:45:59.005
I don't fully understand it,
851
00:45:59.545 --> 00:46:02.845
but, like, if I take the amount of time that I invested
852
00:46:03.290 --> 00:46:08.350
in every Kafka production issue compared to other class, it will be I'm not getting, like, 10 times
853
00:46:08.810 --> 00:46:11.850
between 5 times and 10 times more. So I think that
854
00:46:13.115 --> 00:46:16.255
and this brings me back to the motivation of writing the book
855
00:46:16.795 --> 00:46:21.055
that Kafka uses the operating system in a way that
856
00:46:21.619 --> 00:46:22.119
no
857
00:46:22.580 --> 00:46:25.320
other open source that I know of uses it,
858
00:46:25.700 --> 00:46:26.200
especially
859
00:46:27.220 --> 00:46:35.670
not the CPU, by the way. It's really like, on the CPU, it's, like, very simple. User time, and that's it if if you do it correct.
860
00:46:36.150 --> 00:46:40.890
But the usage of the page cache, the trashing of the page cache in in cases that you have legs,
861
00:46:41.750 --> 00:46:43.210
and the the
862
00:46:43.510 --> 00:46:44.569
the amount of stress
863
00:46:45.895 --> 00:46:47.595
that comes down on the disks
864
00:46:48.135 --> 00:46:48.875
is something
865
00:46:49.255 --> 00:46:51.595
that I I don't think the the the
866
00:46:51.935 --> 00:46:52.435
the
867
00:46:52.935 --> 00:46:54.395
those who created Kafka
868
00:46:55.250 --> 00:46:56.230
or those who
869
00:46:56.930 --> 00:46:59.510
who develop it or contribute to it,
870
00:47:00.290 --> 00:47:01.510
I think there's some
871
00:47:02.244 --> 00:47:05.464
split between those who develop and those who who maintain,
872
00:47:07.765 --> 00:47:09.305
and and there's not enough
873
00:47:10.270 --> 00:47:12.450
connection between the 2 2 sides.
874
00:47:13.710 --> 00:47:16.610
And maybe this this is another when I think of it,
875
00:47:16.990 --> 00:47:22.955
I'm thinking out loud now. This is another reason why the book is important both for developers and for
876
00:47:23.575 --> 00:47:28.055
ops team in order to not not only understand Kafka, but also understand
877
00:47:28.440 --> 00:47:30.780
for developers to understand the ops team,
878
00:47:32.440 --> 00:47:36.265
because there are so much problems that are being caused by
879
00:47:36.664 --> 00:47:38.845
1 disk that goes rogue.
880
00:47:39.704 --> 00:47:40.525
1 disk.
881
00:47:41.625 --> 00:47:46.045
You can you can have a cluster with tens of disk and 1 disk go bad,
882
00:47:46.650 --> 00:47:47.150
and
883
00:47:47.690 --> 00:47:50.830
the this can cause the whole cluster to halt.
884
00:47:52.010 --> 00:47:52.510
And
885
00:47:53.694 --> 00:47:57.875
a healthy cluster should not suffer from such an issue. Remember,
886
00:47:58.335 --> 00:48:00.915
getting a call about 5 years ago
887
00:48:01.339 --> 00:48:04.000
from a customer side that had 3 brokers,
888
00:48:04.619 --> 00:48:06.960
6 disks, HDD disks,
889
00:48:07.819 --> 00:48:09.440
configured in RAID 10,
890
00:48:10.625 --> 00:48:13.925
and 1 disk went bad. So you see
891
00:48:14.385 --> 00:48:18.645
but but in RAID 10 in RAID, you see only 1 logical disk.
892
00:48:18.960 --> 00:48:22.660
So and the utilization in Iosat shows the utilization
893
00:48:22.960 --> 00:48:23.700
of the,
894
00:48:24.480 --> 00:48:28.420
the highest utilization among all the disks in the RAID, and then you see 100
895
00:48:29.155 --> 00:48:30.135
percent. So
896
00:48:30.755 --> 00:48:36.295
I had to I guess that that, it was 1 machine the 1 disk that was,
897
00:48:36.835 --> 00:48:37.815
that got screwed.
898
00:48:38.309 --> 00:48:40.970
And I told the support engineer, go to the room
899
00:48:41.990 --> 00:48:43.930
and check the if the light,
900
00:48:44.470 --> 00:48:44.970
waves
901
00:48:45.485 --> 00:48:49.505
on 1 of the disc, and he told me, yes. There's the light was on 1 of them.
902
00:48:50.125 --> 00:48:50.625
But
903
00:48:51.005 --> 00:48:52.285
imagine that you need to do
904
00:48:53.005 --> 00:48:54.225
I needed to get
905
00:48:54.780 --> 00:48:55.440
to get
906
00:48:55.900 --> 00:48:59.119
this because because of the lack of monitoring. So
907
00:49:00.060 --> 00:49:01.359
when I think of it,
908
00:49:01.820 --> 00:49:04.825
then the work on something needs to change
909
00:49:05.285 --> 00:49:05.785
on
910
00:49:06.325 --> 00:49:06.825
like,
911
00:49:07.285 --> 00:49:10.825
not on the work on the disk because this is what Kafka does. It writes
912
00:49:11.240 --> 00:49:13.900
massive amounts of data to the disk, and it reads
913
00:49:14.440 --> 00:49:20.335
it tries not to read from the disk, but at many customer sites or cloud
914
00:49:20.875 --> 00:49:22.815
or cluster, it reads from disk.
915
00:49:23.115 --> 00:49:26.335
And first of all, ops team dot node, it reads from disk.
916
00:49:26.715 --> 00:49:27.215
Secondly,
917
00:49:27.915 --> 00:49:28.415
like,
918
00:49:29.740 --> 00:49:32.480
many cluster have deployment of RAID. So
919
00:49:32.940 --> 00:49:34.960
how can you assist them in understanding
920
00:49:35.500 --> 00:49:37.119
the that 1 disk
921
00:49:37.445 --> 00:49:38.105
got screwed?
922
00:49:39.125 --> 00:49:45.145
They don't have a and in the cloud also, by the way. Why when 1 disk gets bad,
923
00:49:47.090 --> 00:49:47.730
there is
924
00:49:48.210 --> 00:49:55.185
the cloud provider won't tell you that because they they can't really tell you, okay. I'm on the disk is the disk is
925
00:49:55.565 --> 00:49:56.785
on 80% utilization
926
00:49:57.645 --> 00:49:58.785
for 30 minutes.
927
00:49:59.245 --> 00:50:02.625
We don't know if it's good or not, so we will not tell you ever anything.
928
00:50:03.240 --> 00:50:06.620
And then you get into high, high weight, and you replace the broker.
929
00:50:07.320 --> 00:50:08.540
So the mitigation
930
00:50:09.080 --> 00:50:10.860
for these issues in Kafka
931
00:50:11.725 --> 00:50:12.205
is,
932
00:50:12.605 --> 00:50:13.585
is is
933
00:50:14.205 --> 00:50:15.505
is brute force.
934
00:50:15.965 --> 00:50:16.465
And
935
00:50:17.245 --> 00:50:19.905
and figuring out that you have a disk issue,
936
00:50:20.570 --> 00:50:23.950
you need to be a magician in order to to to know this.
937
00:50:26.170 --> 00:50:28.830
And and developers of Kafka just assume that,
938
00:50:29.940 --> 00:50:33.295
okay. Someone will handle it. It's not us. We just develop.
939
00:50:33.915 --> 00:50:34.815
And I think
940
00:50:35.435 --> 00:50:41.540
for me coming, I did a shift, a career shift, which is not common, like going from developer to
941
00:50:41.920 --> 00:50:42.660
ops. So
942
00:50:43.120 --> 00:50:46.100
I I understand the frustration from both sides,
943
00:50:47.295 --> 00:50:49.875
and I think that there should be more people
944
00:50:50.255 --> 00:50:51.075
that know,
945
00:50:51.935 --> 00:50:52.835
both paradigms.
946
00:50:53.775 --> 00:50:54.275
And
947
00:50:54.710 --> 00:50:56.730
because a lot of production issues
948
00:50:57.829 --> 00:50:58.329
originate
949
00:50:58.950 --> 00:51:00.170
from the lack of understanding
950
00:51:00.549 --> 00:51:01.049
or
951
00:51:01.430 --> 00:51:02.250
the lack of
952
00:51:02.875 --> 00:51:10.815
cooperation or knowledge sharing between the between the 2. If if the Kafka community had better communication between the developers and the ops team,
953
00:51:11.280 --> 00:51:11.780
then
954
00:51:13.120 --> 00:51:15.360
I think it would be much easier to detect,
955
00:51:15.680 --> 00:51:16.500
disk issues,
956
00:51:17.040 --> 00:51:17.860
which cause,
957
00:51:19.120 --> 00:51:20.900
I bet it causes, like,
958
00:51:21.995 --> 00:51:23.695
at least third of the problems
959
00:51:24.555 --> 00:51:25.215
in Kafka.
960
00:51:26.075 --> 00:51:29.135
Yeah. It's, definitely always a challenge balancing
961
00:51:29.470 --> 00:51:39.685
the developer of, I just wanna get something shipped and do something cool with some fancy new feature in the operations team of, I just want you to stop crushing my machines so that I can sleep at night.
962
00:51:40.305 --> 00:51:40.805
Yeah.
963
00:51:42.545 --> 00:51:44.805
Yes. But but I I it's
964
00:51:45.105 --> 00:51:47.150
like in the Kafka community,
965
00:51:48.490 --> 00:51:52.735
I think there is lack of ops team that will tell that will check
966
00:51:53.855 --> 00:51:54.915
these these developments.
967
00:51:56.015 --> 00:51:58.355
And but it's it's not even new development.
968
00:51:58.895 --> 00:52:03.859
Just for I am telling I'm asking the Kafka community, like, go to deployments
969
00:52:04.560 --> 00:52:06.020
and of Kafka
970
00:52:06.480 --> 00:52:08.180
and check ask customers,
971
00:52:09.200 --> 00:52:10.260
what's the percentage
972
00:52:11.055 --> 00:52:13.875
of production issues which is caused by disks.
973
00:52:14.815 --> 00:52:15.954
And then
974
00:52:16.895 --> 00:52:22.730
try to I don't know. Maybe maybe every Kafka tool needs a better monitoring of of of disks.
975
00:52:23.270 --> 00:52:23.770
Like
976
00:52:24.630 --> 00:52:26.410
and, also, if there is monitoring,
977
00:52:26.870 --> 00:52:27.590
how do you read
978
00:52:28.535 --> 00:52:32.234
like, Brandon Gregg has an excellent explanation of how to read
979
00:52:32.775 --> 00:52:36.155
a IOSTAT, the output of IOSTAT. So you have the utilization.
980
00:52:36.680 --> 00:52:41.740
Like, the service time is obsolete. No 1 needs to look at it. So every utilization,
981
00:52:42.200 --> 00:52:50.365
you have throughput, which is the read megabyte per sec and write megabyte per sec, And you have IOPS, which is the read per second, write per sec.
982
00:52:51.065 --> 00:52:51.565
And
983
00:52:54.025 --> 00:52:56.260
and this is a fact that not many
984
00:52:57.120 --> 00:52:58.580
people know. The saturation
985
00:52:58.880 --> 00:53:00.420
of disks is 60%,
986
00:53:01.255 --> 00:53:03.994
Only 60%, which means that every increase
987
00:53:04.375 --> 00:53:05.515
in the disc utilization
988
00:53:06.615 --> 00:53:09.275
going is bringing the situation,
989
00:53:09.760 --> 00:53:13.540
the the level of IO 8 will become worse and worse and worse.
990
00:53:14.400 --> 00:53:17.140
And it's not like in CPU. In CPU, it could be,
991
00:53:18.335 --> 00:53:20.915
like, the recommendation on cloud is, like, 75%
992
00:53:21.775 --> 00:53:22.275
CPU
993
00:53:22.575 --> 00:53:23.234
CPU saturation.
994
00:53:23.615 --> 00:53:26.195
Above that, your load average will increases
995
00:53:26.750 --> 00:53:28.210
in a nonlinear way.
996
00:53:29.069 --> 00:53:29.569
So
997
00:53:30.109 --> 00:53:32.690
how do you read the how Kafka,
998
00:53:33.549 --> 00:53:39.385
use ops team should read the output of iostat? Well, it's simple. Look at the disk utilization.
999
00:53:40.005 --> 00:53:47.099
The disk utilization of 100 I saw developer ops team telling me, okay. We reached 100%. This is not good. No. This is okay
1000
00:53:47.400 --> 00:53:54.625
because Kafka works in bursts. So it writes a lot of data for a small amount of time, so we'll have high disk utilization
1001
00:53:55.005 --> 00:53:55.825
caused by
1002
00:53:56.260 --> 00:54:02.680
read meg write megabyte per second. Write write write per second. This this is good. And then you should see 0.
1003
00:54:03.060 --> 00:54:05.480
And then again, you have a burst of writes.
1004
00:54:06.045 --> 00:54:06.545
And
1005
00:54:07.085 --> 00:54:08.445
but if you have burst of,
1006
00:54:09.005 --> 00:54:10.865
if you have 100% utilization
1007
00:54:11.165 --> 00:54:12.225
because of reads,
1008
00:54:12.685 --> 00:54:19.030
then it means that you read from the disk, which means that you have a problem somehow. Maybe you have a a consumer lag.
1009
00:54:19.330 --> 00:54:21.010
Maybe you replicate some of
1010
00:54:21.570 --> 00:54:23.590
a a broker that replicates the data
1011
00:54:24.125 --> 00:54:25.905
is is is, lagging
1012
00:54:26.205 --> 00:54:26.705
behind,
1013
00:54:27.725 --> 00:54:30.045
which means that this partition is not in the it's
1014
00:54:31.090 --> 00:54:33.990
this broker is not in your ISR list of this partition.
1015
00:54:34.850 --> 00:54:38.390
So just look at the output of the iostat, minus 6.
1016
00:54:38.850 --> 00:54:44.335
It prints every 1 second. And if you have several seconds of 100% utilization
1017
00:54:44.715 --> 00:54:45.215
from
1018
00:54:45.995 --> 00:54:49.055
writes, then this is okay. But if you have the same
1019
00:54:50.320 --> 00:54:51.460
100% utilization
1020
00:54:52.160 --> 00:54:54.980
from grids, then this is not okay.
1021
00:54:55.440 --> 00:54:58.445
Now if the community will take this and make it
1022
00:54:59.245 --> 00:55:01.185
an alert or some monitoring
1023
00:55:01.485 --> 00:55:02.705
tool, then
1024
00:55:03.085 --> 00:55:07.020
it will reduce, and ops team will understand know how to read this,
1025
00:55:07.500 --> 00:55:09.360
then it will reduce the
1026
00:55:10.220 --> 00:55:13.440
it will reduce the frequency of production issues.
1027
00:55:13.980 --> 00:55:17.355
It it is more than 10%, for sure. Just this
1028
00:55:18.055 --> 00:55:20.075
feature, detect leg detect
1029
00:55:20.695 --> 00:55:21.195
legs,
1030
00:55:22.935 --> 00:55:24.795
by looking at the diastat.
1031
00:55:25.160 --> 00:55:25.660
And
1032
00:55:26.839 --> 00:55:27.740
in your experience
1033
00:55:28.920 --> 00:55:30.940
of working with Kafka clusters
1034
00:55:31.319 --> 00:55:32.859
and helping customers
1035
00:55:33.160 --> 00:55:35.020
and end users manage
1036
00:55:35.465 --> 00:55:40.445
and ensure their uptime? What are some of the most interesting or unexpected or challenging
1037
00:55:40.745 --> 00:55:44.525
production problems that you've had the, opportunity to diagnose?
1038
00:55:45.940 --> 00:55:47.320
Okay. I will tell
1039
00:55:48.020 --> 00:55:48.680
the most
1040
00:55:50.020 --> 00:55:50.840
this is
1041
00:55:52.535 --> 00:56:01.275
the okay. The most bizarre. I will tell the most bizarre production issues, the most the most interesting ones. So the most bizarre, I I I already
1042
00:56:03.000 --> 00:56:04.140
mentioned it before,
1043
00:56:04.760 --> 00:56:09.820
was a cluster of, 3 brokers, sick 18 disks, 6 per
1044
00:56:10.275 --> 00:56:10.775
broker
1045
00:56:11.155 --> 00:56:12.615
configured in RAID 10,
1046
00:56:13.475 --> 00:56:13.975
that,
1047
00:56:14.915 --> 00:56:16.970
1 1 disk, 1 40 disks
1048
00:56:17.690 --> 00:56:20.349
1 40 disk crippled down the cluster because,
1049
00:56:21.690 --> 00:56:22.190
because,
1050
00:56:22.730 --> 00:56:27.775
it was on ray 10. So, in fact, only 2 disks were up, and the disk utilization
1051
00:56:28.555 --> 00:56:34.015
were already was already pretty high, so it just added to the party this problem.
1052
00:56:34.960 --> 00:56:38.180
And then I had to guess that 1 disk was faulty,
1053
00:56:38.880 --> 00:56:47.105
and this was the support engineer really saw that 1 disk was faulty after he went to the server room. And, of course, the the other it's it's,
1054
00:56:47.645 --> 00:56:50.305
the it's twin disk because of straight m
1055
00:56:50.820 --> 00:56:51.320
also,
1056
00:56:51.700 --> 00:56:52.680
didn't function.
1057
00:56:53.300 --> 00:56:53.800
And
1058
00:56:54.260 --> 00:57:01.145
so on that and the disc utilization was already very high. So it brought the disk utilization to 100%
1059
00:57:01.445 --> 00:57:03.385
on the brokers. So 3rd of the
1060
00:57:03.685 --> 00:57:07.225
of the leaders, partition leaders were on that broker, and then
1061
00:57:07.700 --> 00:57:08.200
producers
1062
00:57:08.820 --> 00:57:12.200
stopped, couldn't cry to them. Consumers didn't read from them.
1063
00:57:12.500 --> 00:57:13.720
And once consumers
1064
00:57:14.339 --> 00:57:19.625
don't read even from 1 partition, then the day then they just cannot function.
1065
00:57:20.005 --> 00:57:23.465
It depends, of course, on the nature of your consumers, but this was
1066
00:57:24.609 --> 00:57:27.430
a streaming application that had to read from all consumers.
1067
00:57:28.130 --> 00:57:35.895
So it got stuck. Even if it wasn't the application that needs to read all consumers, then you have partial data. So
1068
00:57:36.275 --> 00:57:38.615
that was but that was a very bizarre
1069
00:57:39.155 --> 00:57:39.655
problem.
1070
00:57:40.275 --> 00:57:45.309
That combine on prem and high disk utilization and guessing that 1 disk is 40.
1071
00:57:45.770 --> 00:57:46.910
The most interesting
1072
00:57:47.609 --> 00:57:48.109
problem
1073
00:57:48.970 --> 00:57:50.190
that I ran into
1074
00:57:50.875 --> 00:57:51.375
was,
1075
00:57:52.875 --> 00:57:53.615
it took,
1076
00:57:54.235 --> 00:57:56.895
it took several weeks, I think. And,
1077
00:57:57.835 --> 00:57:58.575
it involved
1078
00:58:00.260 --> 00:58:00.760
brokers
1079
00:58:02.340 --> 00:58:02.820
that,
1080
00:58:03.300 --> 00:58:09.080
from a certain point in time, every broker that was added to the cluster due to some
1081
00:58:09.915 --> 00:58:10.955
failure of other broker,
1082
00:58:12.475 --> 00:58:13.855
every broker was,
1083
00:58:15.675 --> 00:58:18.790
got its, at some point, to 100%
1084
00:58:19.250 --> 00:58:19.990
disk utilization,
1085
00:58:21.089 --> 00:58:21.589
and
1086
00:58:23.010 --> 00:58:29.525
and nobody managed to write and read for me. Of course, no manage nobody managed to write, so there was nothing to consume.
1087
00:58:30.305 --> 00:58:30.805
And
1088
00:58:31.105 --> 00:58:32.644
every time we replaced
1089
00:58:33.160 --> 00:58:36.780
this broker with another broker, and again, the same phenomena happened.
1090
00:58:37.640 --> 00:58:38.140
And,
1091
00:58:39.160 --> 00:58:39.660
like,
1092
00:58:40.815 --> 00:58:45.394
and then again looking from the IOSTAT after a long, long, long time
1093
00:58:45.855 --> 00:58:49.875
of trying to understand what is going on here, it was like voodoo,
1094
00:58:51.440 --> 00:58:52.900
we noticed that
1095
00:58:54.480 --> 00:58:56.100
that at some brokers,
1096
00:58:56.800 --> 00:58:58.575
the disks reach 100%
1097
00:58:58.875 --> 00:58:59.375
utilization,
1098
00:59:00.875 --> 00:59:01.375
with
1099
00:59:02.475 --> 00:59:02.975
because
1100
00:59:03.355 --> 00:59:06.495
for for all the time, they were in 100% utilization
1101
00:59:07.549 --> 00:59:08.529
because they
1102
00:59:10.270 --> 00:59:11.970
but but when we looked at the throughput,
1103
00:59:12.270 --> 00:59:13.809
we saw that they are
1104
00:59:14.109 --> 00:59:14.609
half
1105
00:59:14.990 --> 00:59:15.650
the throughput
1106
00:59:16.135 --> 00:59:21.434
of the throughput that causes other discs on other brokers to be an 1 or 2 percent utilization.
1107
00:59:22.135 --> 00:59:28.640
So not only did the healthy brokers, reach 100% utilization for small amounts or few seconds and then
1108
00:59:29.100 --> 00:59:30.720
the utilization went down,
1109
00:59:31.474 --> 00:59:36.775
These 40 brokers these brokers with 40 disks reached 100% disk utilization,
1110
00:59:37.555 --> 00:59:38.615
and it kept
1111
00:59:38.970 --> 00:59:40.110
100% utilization
1112
00:59:40.730 --> 00:59:42.750
and while the throughput was half.
1113
00:59:43.370 --> 00:59:44.430
So just dividing
1114
00:59:45.610 --> 00:59:48.904
just correlating the disk utilization with the throughput
1115
00:59:49.365 --> 00:59:54.184
and the amount of time the disk utilization was another was 100%
1116
00:59:55.510 --> 00:59:59.370
led us to the understanding that these are just faulty disks.
1117
00:59:59.910 --> 01:00:03.290
But this took a lot of excels and the and just
1118
01:00:03.815 --> 01:00:05.994
checking trying to correlate every
1119
01:00:06.695 --> 01:00:08.234
OS mon OS metric
1120
01:00:09.015 --> 01:00:13.840
until we we found it out, and that was by far the most interesting
1121
01:00:14.940 --> 01:00:16.160
production issue
1122
01:00:16.620 --> 01:00:17.920
that I stumbled into
1123
01:00:18.300 --> 01:00:19.040
in Kafka.
1124
01:00:20.035 --> 01:00:23.734
And in your work of writing the book and
1125
01:00:24.035 --> 01:00:24.535
consolidating
1126
01:00:24.994 --> 01:00:30.600
all of the information and experience that you've had working with Kafka. I'm wondering if there are
1127
01:00:30.900 --> 01:00:36.440
any insights that that helped you gain or any new knowledge that you were able to obtain in the process.
1128
01:00:38.595 --> 01:00:40.135
Of course. Yeah. Because,
1129
01:00:40.915 --> 01:00:41.415
mainly,
1130
01:00:43.155 --> 01:00:46.535
like, from there there were 2 issues, like,
1131
01:00:47.500 --> 01:00:48.000
like,
1132
01:00:48.700 --> 01:00:50.240
it helped me formulate
1133
01:00:50.700 --> 01:00:51.760
the the
1134
01:00:52.300 --> 01:00:55.115
the 3 legs that Kafka stands on. Kafka
1135
01:00:55.655 --> 01:01:00.715
needs to understand, let's say, which is the data part, the OS part, and the Kafka part.
1136
01:01:01.335 --> 01:01:04.250
I was surprised to see that the Kafka part is
1137
01:01:04.870 --> 01:01:06.410
is only third of the book,
1138
01:01:07.190 --> 01:01:10.570
which shows how much the data part is is important, how
1139
01:01:11.125 --> 01:01:11.625
how
1140
01:01:12.005 --> 01:01:13.065
the the
1141
01:01:13.685 --> 01:01:14.905
how the way that
1142
01:01:15.205 --> 01:01:16.825
the data is spread
1143
01:01:17.285 --> 01:01:18.425
among the partitions
1144
01:01:19.070 --> 01:01:22.770
is so important for the, for the health of the cluster.
1145
01:01:23.550 --> 01:01:24.050
And,
1146
01:01:24.510 --> 01:01:25.010
also,
1147
01:01:26.190 --> 01:01:30.015
I was surprised of how many issues production issues
1148
01:01:30.635 --> 01:01:31.775
originate from
1149
01:01:32.475 --> 01:01:34.335
a problem with with storage.
1150
01:01:36.019 --> 01:01:36.500
And,
1151
01:01:37.779 --> 01:01:38.279
and,
1152
01:01:39.220 --> 01:01:39.720
also,
1153
01:01:40.260 --> 01:01:43.640
I found out several producer and consumer metrics
1154
01:01:44.255 --> 01:01:45.954
that that were new to me,
1155
01:01:46.815 --> 01:01:49.234
because I thought that
1156
01:01:49.855 --> 01:01:50.850
I that
1157
01:01:51.570 --> 01:01:54.870
many issues can be fixed with, with tuning the linger
1158
01:01:55.410 --> 01:01:58.045
and the batch size on the producer and that found out
1159
01:01:58.684 --> 01:02:01.905
several very important metrics in the consumer and producer.
1160
01:02:02.765 --> 01:02:04.305
So this was also
1161
01:02:05.300 --> 01:02:08.840
new to me. I ran them I ran across them during the time,
1162
01:02:09.780 --> 01:02:12.760
like, from production issues that I dealt with
1163
01:02:13.164 --> 01:02:16.065
during the time that I I wrote the book.
1164
01:02:16.605 --> 01:02:17.105
And
1165
01:02:18.365 --> 01:02:22.990
I must say that I must say that a lot of things written in the book were not things
1166
01:02:23.770 --> 01:02:25.310
that only I discovered.
1167
01:02:25.610 --> 01:02:27.630
So I worked with several
1168
01:02:28.410 --> 01:02:29.630
with several people
1169
01:02:30.170 --> 01:02:30.570
that,
1170
01:02:31.050 --> 01:02:35.714
that that I work we worked together and found the issues together. So
1171
01:02:36.095 --> 01:02:37.795
in part, it was just documenting
1172
01:02:38.655 --> 01:02:40.835
what team of ops people,
1173
01:02:41.790 --> 01:02:42.530
found out,
1174
01:02:43.630 --> 01:02:44.930
including me, but also
1175
01:02:46.350 --> 01:02:49.570
other people. I I I I have, like,
1176
01:02:50.855 --> 01:02:57.915
in order to to make this more more specific, I have an for example, the the issue of the storage.
1177
01:02:58.295 --> 01:02:59.195
Just to emphasize,
1178
01:03:01.000 --> 01:03:03.339
when I wrote the the part of the storage usage,
1179
01:03:03.720 --> 01:03:04.460
turns out
1180
01:03:04.920 --> 01:03:06.040
that it's much,
1181
01:03:07.160 --> 01:03:08.540
that it's more vast
1182
01:03:09.155 --> 01:03:12.695
than what I thought of. So if just we if we just take,
1183
01:03:13.315 --> 01:03:15.494
this issue, the storage usage issue.
1184
01:03:15.795 --> 01:03:16.295
So,
1185
01:03:17.530 --> 01:03:18.090
for example,
1186
01:03:19.530 --> 01:03:22.270
running out running out of disk space
1187
01:03:22.570 --> 01:03:24.110
due to retention configuration.
1188
01:03:24.570 --> 01:03:26.270
What happens when you configure
1189
01:03:26.795 --> 01:03:28.255
both time based
1190
01:03:28.555 --> 01:03:29.615
and size based?
1191
01:03:30.875 --> 01:03:38.560
The way the the the option there is an option to lose data, and I was surprised to to see how 2 simple configurations
1192
01:03:38.860 --> 01:03:40.320
like this can cause
1193
01:03:41.100 --> 01:03:41.920
data loss.
1194
01:03:42.460 --> 01:03:42.960
And
1195
01:03:43.500 --> 01:03:46.525
so retention policy and its effect on the data loss,
1196
01:03:47.244 --> 01:03:47.744
the
1197
01:03:48.285 --> 01:03:54.065
explaining, like, how to add storage to a cluster, how it differs between on prem and on the cloud.
1198
01:03:54.530 --> 01:03:55.750
The the fact that
1199
01:03:56.050 --> 01:03:59.190
that when you're on prem, this is not only technical decision.
1200
01:03:59.730 --> 01:04:00.710
It's also managerial
1201
01:04:01.090 --> 01:04:03.430
or financial decision because you can't tell
1202
01:04:03.775 --> 01:04:14.490
the owner of the data center, okay. I I I I had a mistake. I don't need the 2 terabyte disk. I need 4 terabyte disks. So I need to throw away all the 2 terabyte and buy 4 terabyte disks
1203
01:04:14.950 --> 01:04:16.970
so it will not go smoothly.
1204
01:04:18.069 --> 01:04:19.049
So understanding
1205
01:04:19.349 --> 01:04:20.329
these aspects,
1206
01:04:20.924 --> 01:04:24.704
like, this chapter became partially technical and partially
1207
01:04:25.404 --> 01:04:27.345
how is a is a provider
1208
01:04:27.860 --> 01:04:29.240
for an on prem customer,
1209
01:04:30.500 --> 01:04:32.680
how you manage this this issue?
1210
01:04:33.193 --> 01:04:33.693
Of
1211
01:04:34.260 --> 01:04:40.115
again, in DIMMs, it's the same issue. Okay. I I I it was a mistake. I bought 16 megabyte
1212
01:04:40.415 --> 01:04:42.835
this 16 gigabyte and it 32 or 64.
1213
01:04:43.349 --> 01:04:46.650
How do you pass this decision? How do you mix DIMMs
1214
01:04:47.109 --> 01:04:48.170
in on prem?
1215
01:04:49.109 --> 01:04:49.849
And, also,
1216
01:04:50.765 --> 01:04:57.905
the effect of of the retention on data replay. Something you need to replay the data because you did some wrong transformations.
1217
01:04:58.650 --> 01:05:00.430
So what's the effect of, like,
1218
01:05:01.530 --> 01:05:02.030
customer
1219
01:05:02.570 --> 01:05:03.870
ops team need to understand
1220
01:05:04.650 --> 01:05:06.510
that they need storage also for
1221
01:05:07.065 --> 01:05:08.525
a for a for replay.
1222
01:05:09.385 --> 01:05:14.205
The data skew, how data skew can affect data loss even you if you have a lot of storage.
1223
01:05:14.505 --> 01:05:16.685
If you don't partition the data correctly,
1224
01:05:17.369 --> 01:05:20.430
then you will get data loss at some point, even 1 partition.
1225
01:05:20.730 --> 01:05:26.349
And it's for certain consumers, this is like data loss in all partition. You need to replay the data
1226
01:05:26.985 --> 01:05:30.765
again. So the data aspects were also something that I learned
1227
01:05:31.225 --> 01:05:31.965
along the way.
1228
01:05:34.250 --> 01:05:36.830
So this is only an example of
1229
01:05:37.210 --> 01:05:41.025
1 chapter. The issue is discussed in the chapter of storage usage.
1230
01:05:41.505 --> 01:05:49.924
But I not only that I learned during the writing of the book, I think that if I wouldn't write the book, I would forget almost
1231
01:05:50.224 --> 01:05:50.724
everything.
1232
01:05:52.770 --> 01:05:54.150
So for me,
1233
01:05:54.610 --> 01:05:59.045
the for for the my the personal benefit for me is that
1234
01:05:59.345 --> 01:06:00.965
I know I remember
1235
01:06:01.345 --> 01:06:01.845
stuff,
1236
01:06:02.305 --> 01:06:04.805
really, like, Kafka related stuff that I knew,
1237
01:06:05.850 --> 01:06:10.110
and and I didn't forget them, but also that I learned along the way.
1238
01:06:11.610 --> 01:06:13.790
So I think the investing
1239
01:06:15.765 --> 01:06:16.585
10 months,
1240
01:06:17.685 --> 01:06:18.665
during the weekends,
1241
01:06:18.965 --> 01:06:23.065
every weekend for the for this amount of period to write the book was
1242
01:06:23.445 --> 01:06:23.945
beneficial
1243
01:06:25.609 --> 01:06:28.029
for my technical knowledge. Absolutely.
1244
01:06:29.130 --> 01:06:34.994
Are there any other aspects of the work that you've done with Kafka, your work on the book, the overall
1245
01:06:35.295 --> 01:06:35.795
Kafka
1246
01:06:36.575 --> 01:06:41.155
operations ecosystem that we didn't discuss yet that you'd like to cover before we close out the show?
1247
01:06:42.690 --> 01:06:44.470
I think we covered a lot of
1248
01:06:44.849 --> 01:06:45.349
technical
1249
01:06:46.049 --> 01:06:48.630
technical stuff. We we can discuss the
1250
01:06:49.170 --> 01:06:50.069
the cost reduction,
1251
01:06:52.085 --> 01:06:55.365
but but in very short, like, I would like to mention that,
1252
01:06:57.045 --> 01:07:00.345
there is a chapter on cost reduction in in Kafka,
1253
01:07:00.880 --> 01:07:03.700
but it relates it relates to like, I
1254
01:07:04.000 --> 01:07:06.980
I brought 6 examples, 6 real world examples
1255
01:07:07.599 --> 01:07:09.619
of cluster that I stumbled upon,
1256
01:07:10.000 --> 01:07:10.500
which
1257
01:07:11.795 --> 01:07:14.375
per each example, I I I write
1258
01:07:14.995 --> 01:07:15.635
how much,
1259
01:07:16.115 --> 01:07:17.975
I specify how much CPU,
1260
01:07:18.435 --> 01:07:20.535
RAM, and disk each cluster has,
1261
01:07:21.670 --> 01:07:25.130
and also the usage of each of these resources.
1262
01:07:25.670 --> 01:07:28.890
And then I ask whether the cluster can be scaled down
1263
01:07:29.589 --> 01:07:30.615
or scaled in.
1264
01:07:31.494 --> 01:07:32.715
And then I discuss
1265
01:07:34.615 --> 01:07:35.275
other metrics,
1266
01:07:35.895 --> 01:07:36.715
monitoring metrics.
1267
01:07:37.095 --> 01:07:41.130
And by correlating this monitoring, Kafka monitoring metrics, and
1268
01:07:41.750 --> 01:07:42.809
the operating system
1269
01:07:43.109 --> 01:07:44.089
metrics usage,
1270
01:07:44.710 --> 01:07:45.930
I I give recommendation
1271
01:07:46.470 --> 01:07:47.369
regarding whether
1272
01:07:47.670 --> 01:07:51.155
you can scale in or scale down the cluster.
1273
01:07:51.695 --> 01:07:52.755
And I think,
1274
01:07:53.135 --> 01:07:53.635
like,
1275
01:07:54.575 --> 01:07:55.635
this is, like,
1276
01:07:57.190 --> 01:07:59.450
the the cost of the Kafka cluster,
1277
01:07:59.830 --> 01:08:04.090
it's not big, I think, compared to other clusters in an organization.
1278
01:08:05.295 --> 01:08:05.795
But,
1279
01:08:06.415 --> 01:08:09.075
because for for cloud based,
1280
01:08:09.935 --> 01:08:13.635
I I assume that most of the deployments are on demand,
1281
01:08:16.080 --> 01:08:20.580
So even if you have reservation, again, it's on demand, it's not spot, so it's important,
1282
01:08:21.454 --> 01:08:21.954
especially
1283
01:08:22.335 --> 01:08:23.635
in today's market,
1284
01:08:24.094 --> 01:08:27.394
to squeeze in every penny that, that you can save.
1285
01:08:29.670 --> 01:08:32.170
So the the cost reduction part is
1286
01:08:32.870 --> 01:08:33.850
something that
1287
01:08:34.230 --> 01:08:35.210
that can help
1288
01:08:36.045 --> 01:08:37.165
to reduce cost,
1289
01:08:37.725 --> 01:08:38.545
on Kafka.
1290
01:08:40.605 --> 01:08:47.570
But there is a part that I didn't talk about, which is might be a bigger part even than the machines themselves,
1291
01:08:48.590 --> 01:08:49.969
which is the data transfer
1292
01:08:50.429 --> 01:08:51.409
between consumers
1293
01:08:51.710 --> 01:08:52.850
and brokers.
1294
01:08:53.635 --> 01:08:55.715
Because if you don't configure
1295
01:08:56.035 --> 01:08:58.375
if there is no rack awareness in the cluster,
1296
01:08:58.755 --> 01:09:00.615
then consumers will read data,
1297
01:09:01.635 --> 01:09:02.775
only from leaders,
1298
01:09:03.280 --> 01:09:05.300
and these leaders can be
1299
01:09:06.400 --> 01:09:09.940
can be will will most of the leaders statistically
1300
01:09:10.320 --> 01:09:12.100
won't be in the same AZ.
1301
01:09:12.945 --> 01:09:14.245
And for some,
1302
01:09:15.105 --> 01:09:17.045
for some companies, this can save
1303
01:09:18.225 --> 01:09:21.685
100 of 1, 000 of dollar per year configuring reco reco awareness.
1304
01:09:22.590 --> 01:09:25.409
But since I don't have an experience with reco awareness,
1305
01:09:25.869 --> 01:09:27.170
I I didn't,
1306
01:09:27.789 --> 01:09:29.409
discuss it thoroughly.
1307
01:09:30.030 --> 01:09:30.530
But
1308
01:09:31.195 --> 01:09:32.735
for for those who listen,
1309
01:09:33.835 --> 01:09:37.295
checking for echo checking if you can configure echo awareness,
1310
01:09:38.395 --> 01:09:40.175
in your consumers and producers,
1311
01:09:41.260 --> 01:09:43.360
and consumers and brokers, sorry,
1312
01:09:43.980 --> 01:09:47.840
this can be beneficial. So you need to check your data transfer cost,
1313
01:09:48.585 --> 01:09:51.945
and maybe it's worthwhile for you to invest in a
1314
01:09:52.825 --> 01:09:56.045
in in deploy in test in deploying and testing,
1315
01:09:56.950 --> 01:09:58.570
and validating the track awareness,
1316
01:09:59.750 --> 01:10:04.010
works. And then you will read not from the leaders, but from the closest replica
1317
01:10:04.715 --> 01:10:05.355
and save,
1318
01:10:06.155 --> 01:10:07.454
data transfer costs.
1319
01:10:07.994 --> 01:10:12.804
Yeah. That can definitely be a substantial cost when running in the cloud, and that's always 1 of those,
1320
01:10:13.840 --> 01:10:17.860
surprise gotchas when you're first getting up and running in a cloud environment.
1321
01:10:19.920 --> 01:10:23.545
Not not only when you're getting started, but also when you're,
1322
01:10:24.565 --> 01:10:25.385
after years,
1323
01:10:25.844 --> 01:10:37.600
then you you see the the data transfer. Yeah. This brings us back to what we started with regarding cloud versus on prem. Thus, it is the the amount of reasons why
1324
01:10:38.380 --> 01:10:39.760
it becomes, like,
1325
01:10:40.165 --> 01:10:42.264
I don't know, like, for some clusters,
1326
01:10:42.965 --> 01:10:46.184
it might be a good idea to to check the,
1327
01:10:47.125 --> 01:10:47.625
like,
1328
01:10:48.350 --> 01:10:48.850
to
1329
01:10:49.310 --> 01:10:51.650
I'm saying this because I came from on prem,
1330
01:10:51.950 --> 01:10:53.970
so it's not Chinese for me.
1331
01:10:54.430 --> 01:10:54.930
And,
1332
01:10:55.845 --> 01:10:58.185
okay, so you don't have managed services,
1333
01:10:59.285 --> 01:11:01.705
but it might be from
1334
01:11:02.570 --> 01:11:03.390
from the
1335
01:11:04.650 --> 01:11:05.710
financial perspective,
1336
01:11:06.730 --> 01:11:10.670
the valid choice for some clusters to to be deployed
1337
01:11:11.844 --> 01:11:12.505
on prem.
1338
01:11:13.844 --> 01:11:30.005
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd just like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today?
1339
01:11:31.505 --> 01:11:32.165
Oh, also,
1340
01:11:32.784 --> 01:11:33.445
I I usually
1341
01:11:34.225 --> 01:11:34.965
work with
1342
01:11:36.705 --> 01:11:37.205
analytics
1343
01:11:37.970 --> 01:11:38.470
clusters.
1344
01:11:39.490 --> 01:11:39.990
And
1345
01:11:40.690 --> 01:11:41.830
I think that
1346
01:11:43.570 --> 01:11:45.030
if there was a tool
1347
01:11:45.455 --> 01:11:46.095
that wouldn't
1348
01:11:46.655 --> 01:11:47.715
would would show
1349
01:11:48.095 --> 01:11:51.555
at any given point of time a correlation between the traffic,
1350
01:11:52.735 --> 01:11:54.460
whether it's the incoming traffic
1351
01:11:55.420 --> 01:11:58.080
or a query or query load.
1352
01:11:58.940 --> 01:12:00.320
So correlating between
1353
01:12:00.700 --> 01:12:02.640
the the load on the cluster
1354
01:12:03.875 --> 01:12:04.775
and the
1355
01:12:05.395 --> 01:12:06.935
and the usage of the cluster
1356
01:12:07.715 --> 01:12:08.695
in terms of,
1357
01:12:09.395 --> 01:12:10.455
like, CPU,
1358
01:12:10.915 --> 01:12:11.975
RAM, disk,
1359
01:12:12.410 --> 01:12:17.450
or even internal usage. For example, let's say, a droid cluster that uses
1360
01:12:17.930 --> 01:12:20.270
sometimes the bottleneck is the number of workers,
1361
01:12:21.685 --> 01:12:28.265
or for Trello clusters, the number of split queries. So if there was something that some tool that would show correlation
1362
01:12:29.190 --> 01:12:31.270
between the load and the cluster and the real,
1363
01:12:31.670 --> 01:12:32.170
usage
1364
01:12:32.789 --> 01:12:33.530
and cost,
1365
01:12:35.030 --> 01:12:36.010
it would allow
1366
01:12:36.789 --> 01:12:39.225
a ops team to teams to
1367
01:12:39.605 --> 01:12:41.625
to better understand whether they can
1368
01:12:42.085 --> 01:12:45.305
save cost on the cluster, whether they can scale it down,
1369
01:12:45.910 --> 01:12:58.095
or, maybe replace on demands with spots, or maybe replace, on demand reservation with on demands without reservation, and then auto scaling them. So something that we show some tool that will show correlation
1370
01:12:59.115 --> 01:12:59.915
between usage
1371
01:13:00.635 --> 01:13:01.775
applicative usage
1372
01:13:02.370 --> 01:13:03.510
and resource usage,
1373
01:13:03.889 --> 01:13:05.429
and will enable to
1374
01:13:05.889 --> 01:13:08.370
to save costs because especially today
1375
01:13:09.090 --> 01:13:09.750
in today's
1376
01:13:10.610 --> 01:13:11.110
economic
1377
01:13:12.265 --> 01:13:13.405
economics, it's
1378
01:13:13.785 --> 01:13:17.085
it's becomes pretty important to save costs.
1379
01:13:17.545 --> 01:13:23.030
Alright. Well, thank you very much for taking the time today to join me and share your experiences
1380
01:13:23.410 --> 01:13:31.995
of running and operating Kafka clusters and the work that you've done on the book to make that easier for everybody else to do as well. It's definitely a very
1381
01:13:32.375 --> 01:13:33.675
challenging and necessary
1382
01:13:34.055 --> 01:13:40.235
task. And as you said, Kafka is very widely deployed, so I appreciate the time and energy that you put into
1383
01:13:40.550 --> 01:13:44.890
sharing your hard 1 knowledge with everyone else, and I hope you enjoy the rest of your day.
1384
01:13:45.670 --> 01:13:48.810
Cool. Thank you very much. Again, thank you for hosting me.
1385
01:13:49.145 --> 01:13:49.645
And
1386
01:13:50.985 --> 01:13:54.425
I hope that the audience will, will gain something from,
1387
01:13:54.985 --> 01:13:55.965
from this podcast.
1388
01:14:02.560 --> 01:14:06.405
Thank you for listening. Don't forget to check out our other shows, podcast.init,
1389
01:14:06.945 --> 01:14:12.910
which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
1390
01:14:13.290 --> 01:14:17.790
which helps you go from idea to production with machine learning. Visit the site at dataengineeringpodcast.com
1391
01:14:19.210 --> 01:14:28.280
to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it. Email hosts atdataengineeringpodcast.com
1392
01:14:29.940 --> 01:14:36.120
with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.