Speed Up And Simplify Your Streaming Data Workloads With Red Panda

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 11:54:07 AM
Duration: 3580.564
Channels: 1
1
00:00:12.914 --> 00:00:16.935
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:17.320 --> 00:00:24.860
Are you bogged down by having to manually manage data access controls, repeatedly move and copy data, and create audit reports to prove compliance?
3
00:00:25.505 --> 00:00:29.365
How much time could you save if those tasks were automated across your cloud platforms?
4
00:00:30.305 --> 00:00:35.810
Immuta is an automated data governance solution that enables safe and easy data analytics in the cloud.
5
00:00:36.190 --> 00:00:41.915
Their comprehensive data level security, auditing, and de identification features eliminate the need for time
6
00:00:42.315 --> 00:00:53.040
manual processes, and their focus on data and compliance team collaboration empowers you to deliver quick and valuable data analytics on the most sensitive data to unlock the full potential of your cloud data platforms.
7
00:00:53.980 --> 00:01:00.545
Learn how they streamline and accelerate manual processes to help you derive real results from your data at data engineering pod cast.com/immuta,
8
00:01:01.805 --> 00:01:03.105
that's immu
9
00:01:03.885 --> 00:01:06.715
t a, and get a 14 day free trial. And
10
00:01:07.140 --> 00:01:22.455
when you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends over at Linode. With their managed Kubernetes platform, it's now even easier to deploy and scale your workflow, so try out the latest Helm charts from tools like Pulsar, Packaderm, and Dagster.
11
00:01:22.870 --> 00:01:30.010
With simple pricing, fast networking, object storage, and worldwide data centers, you've got everything you need to run a bulletproof data platform.
12
00:01:30.445 --> 00:01:31.985
Go to data engineering podcast.com/linode,
13
00:01:33.485 --> 00:01:35.104
that's l I n
14
00:01:35.405 --> 00:01:56.735
o d e, today and get a $60 credit to try out a Kubernetes cluster of your own. Don't forget to thank them for their continued support of this show. Your host is Tobias Macy. And today, I'm interviewing Alexander Gallego about his work at vectorized building Red Panda as a performance optimized drop in replacement for Kafka. So, Alexander, can you start by introducing yourself? Hi, Tobias. Thanks for having me here.
15
00:01:57.130 --> 00:01:59.710
I am the founder and CEO of Vectorize,
16
00:02:00.170 --> 00:02:05.070
working on Red Panda, which is a drop in replacement for Kafka for mission critical systems.
17
00:02:05.505 --> 00:02:11.845
Been working on streaming for the last 10 years. Kind of started actually on the computational side of things.
18
00:02:12.145 --> 00:02:15.430
I was the CTO and founder of a company called Concorde,
19
00:02:15.890 --> 00:02:20.150
which is similar to Apache Flink, but we wrote it in c plus plus
20
00:02:20.530 --> 00:02:22.470
And we sold that to Akamai in 2016.
21
00:02:22.915 --> 00:02:31.575
During my tenure at Concore, we basically couldn't find a storage system that could keep up with with that system, and I've now moved to a storage side of things with Red Panda.
22
00:02:32.079 --> 00:02:34.900
And do you remember how you first got involved in the area of data management?
23
00:02:35.760 --> 00:02:36.260
After
24
00:02:36.719 --> 00:02:51.810
college, I I actually went to school for crypto, and I decided to drop out of grad school. Anyways, after college, I went to work for this ad tech company in New York City. And from a technical perspective, it was super cool. It was the idea of, like, let's compete with Google on AdTech
25
00:02:52.430 --> 00:02:54.610
for the premium publishers and advertisers.
26
00:02:55.230 --> 00:03:00.755
I was the 1st engineer there, so it was a really exciting opportunity for me as a as a recent college grad.
27
00:03:01.295 --> 00:03:06.835
And what we did there was we built this real time machine learning for ad bidding pipeline,
28
00:03:07.220 --> 00:03:10.920
which it doesn't sound super cool today. But honestly, 10 years
29
00:03:11.300 --> 00:03:17.875
ago, it was really innovative, and we were basically printing money 10 years ago by doing that. You know, that's how I got started.
30
00:03:18.495 --> 00:03:33.505
You mentioned that Red Panda is this drop in replacement for Kafka with a focus on mission critical applications. So I'm wondering if you can give a bit more context as to what type of applications those might be and some of the ways that Red Panda facilitates that use case.
31
00:03:34.305 --> 00:03:37.765
Effectively, what we care about through a lot of the market research
32
00:03:38.305 --> 00:03:55.865
is we focus on 3 tenets for for the product. And then I'm gonna tell you a little bit more about, like, the particular use case that I think we're a really good fit for. So we basically went, and before I started out this company as a as a full company, I interviewed something like 40 companies, and they spent everything from IoT to
33
00:03:56.245 --> 00:04:03.469
the big tech, even some things. And what we found is that the majority of the companies were actually unhappy running Kafka. And
34
00:04:03.769 --> 00:04:08.670
what was really good about Kafka was actually the entire ecosystem. Like, the Kafka API
35
00:04:09.095 --> 00:04:10.075
sort of won,
36
00:04:10.375 --> 00:04:11.355
but the guarantees
37
00:04:12.055 --> 00:04:34.285
of Kafka, whether it was data safety or running Kafka at a scale, a lot of those companies were having difficulties with that. I think the complaint from the majority of this survey, so 50% of these companies were really having difficulties running Kafka. And for those that did manage to operate Kafka at a scale, so the usual, like, you know, Netflix, Dropbox, etcetera, those people that have deep distributed system expertise,
38
00:04:34.720 --> 00:04:40.260
they weren't getting enough performance. And so what you end up with is, like, massively overprovisioning clusters.
39
00:04:40.560 --> 00:04:43.005
And for some small percentage, they actually cared
40
00:04:43.885 --> 00:04:50.705
a lot about data loss. And what's interesting is actually through the last probably a 100 enterprises that we've talked to,
41
00:04:51.009 --> 00:04:57.669
the majority of enterprises actually run Kafka in a in an unsafe mode. And so when we say mission critical systems
42
00:04:58.025 --> 00:05:14.360
are really people that want to manage 1 system. They wanna have 1 fault domain. They don't wanna manage zookeeper. They don't wanna understand how to manage zookeeper snapshot or run TLS on the leader election port of zookeeper or restate the snapshot or understand how to tune the JVM.
43
00:05:14.775 --> 00:05:24.475
They really want sort of this streamlined system that kind of gets out of the way. That's really the first 1. Right? So operational simplicity was really a big 1. The second 1 was really safety. And so
44
00:05:24.840 --> 00:05:34.535
if you look at kind of what happened in hardware over the last 10 years, a decade ago, there's a company called Blackblaze, and they published this statistics about hardware. And
45
00:05:34.915 --> 00:05:39.175
so 10 years ago, to run a terabyte of SSD was like $2, 500.
46
00:05:40.435 --> 00:05:45.740
But now the cost of running a terabyte SSD is like $200. Right? So it's sort of the first bottleneck.
47
00:05:46.120 --> 00:05:55.555
And because hardware got better on many levels, both on on networking, on disk, and on CPU, the rise of many core systems, we are now able to give stronger guarantees
48
00:05:56.255 --> 00:06:07.490
to our users about data safety. And so the main kind of critical piece about our architecture that has had a profound impact for a lot of our customers, whether they're in oil or ad tech or whatever,
49
00:06:07.925 --> 00:06:11.305
is really that we're safe. Right? We give them the linearizable
50
00:06:11.765 --> 00:06:12.505
Raft implementation
51
00:06:12.885 --> 00:06:18.979
to save their data with a sound fault domain and and a sound failure model. Like, as a developer,
52
00:06:19.440 --> 00:06:29.355
you actually understand exactly what it means to have 2 out of 3 replicas up and running. You understand that there'll be no gaps in the logs. So you have this really strong long completeness guarantee.
53
00:06:29.815 --> 00:06:35.310
You know, if if you contrast that with the default settings of Kafka, you can run Kafka by the way safely,
54
00:06:35.610 --> 00:06:45.134
but no 1 that we've talked to in the last a 100 enterprises does it because the performance impact of flushing every message to disk is too slow. So if you contrast kind of this approach
55
00:06:45.435 --> 00:06:51.455
versus Kafka's, we give basically safety by default for users. And we allow them to both run
56
00:06:52.080 --> 00:06:58.340
with low latency, high throughput, and safety. And so that's really what we mean when we say mission critical systems.
57
00:06:59.175 --> 00:07:09.250
And in terms of the safety aspect, you mentioned that with Kafka, you can flush everything to disk on every right. But because of the overhead of the JVM, it ends up being slow.
58
00:07:09.630 --> 00:07:21.245
I'm curious if in your implementation, you're using something akin to write a head log such as what Postgres does for being able to handle that safety guarantee while also being able to handle some of the
59
00:07:21.610 --> 00:07:24.190
overall performance throughput on the underlying hardware?
60
00:07:24.810 --> 00:07:32.915
This is like a deep technical topic, so let me give you the big highlights, here. So it is actually not just the JVM. So what the JVM
61
00:07:33.375 --> 00:07:35.315
adds to a system is really latency.
62
00:07:35.695 --> 00:07:45.419
And so there's 2 aspects to this question here. It's it's understanding the difference between latency and throughput here. Adding the garbage collection, what it does and we've seen this in production.
63
00:07:45.735 --> 00:07:53.835
We send system, you know, regularly with 2 second JVM latency pauses and in some systems with 6 seconds latency pauses kind of on the outliers.
64
00:07:54.135 --> 00:07:55.435
So that's with the JVM.
65
00:07:55.960 --> 00:08:07.245
But what I'm talking about is really throughput. And the reason most people don't run Kafka with safety settings is because it affects not only their latency, but particularly their throughput. And so what happens
66
00:08:07.865 --> 00:08:11.245
when you issue an f sync on a particular file handle
67
00:08:11.545 --> 00:08:12.605
is that that injects
68
00:08:13.040 --> 00:08:15.700
a global barrier on the kernel. And so
69
00:08:16.080 --> 00:08:21.140
that file, right, so if if you write into a file, it actually injects a barrier on the file system itself
70
00:08:21.514 --> 00:08:32.640
because you can't do anything else. You have to wait for the media to be flushed to to disk. We have to flush the file too to guarantee data safety, but but we do kind of a lot of tricks around batch coalescing
71
00:08:33.260 --> 00:08:36.720
and deep bouncing the rights on our raft implementation.
72
00:08:37.100 --> 00:08:42.995
The second thing that we did there is from a technical level, we don't use the page cache. Right? And so we eliminate
73
00:08:43.375 --> 00:08:51.910
a a very large set of locks that have been in the kernel. On the first step. We'll talk actually about the networking in just a second. But when a message comes in, we're not consuming
74
00:08:52.370 --> 00:09:03.975
any resources from the page cache, which means the global logs and resources that happen on a particular file handle basis, they're just not taken. They're completely eliminated. The second thing is that we have this adaptive file allocation technique.
75
00:09:04.390 --> 00:09:13.745
And so what it means, a lot of the barriers or a lot of the performance limitations come around with synchronizing the metadata on the Linux kernel. And so by preallocating
76
00:09:14.205 --> 00:09:15.505
the file system space
77
00:09:15.885 --> 00:09:20.305
and coalescing the flushes and doing kind of all of these low level
78
00:09:20.970 --> 00:09:22.910
tricks, we're really able to drive
79
00:09:23.290 --> 00:09:29.870
probably 4 x the throughput of of a cluster while being safe and deliver about 10 x lower latency.
80
00:09:30.255 --> 00:09:34.675
So hopefully, that disambiguates some of the performance things that we've seen with our Raft implementation.
81
00:09:35.615 --> 00:09:44.699
In addition to the challenges of running Kafka in a safe mode and the performance impacts that that brings, what are some of the other areas where Kafka
82
00:09:45.160 --> 00:09:46.060
has limitations
83
00:09:46.519 --> 00:09:52.574
with Red Panda in terms of simplifying the operability of the system beyond just the failure domains?
84
00:09:52.875 --> 00:09:56.334
1 of the things that we're really proud of is our auto tuner.
85
00:09:56.899 --> 00:10:04.279
Our team has a lot of deep technical expertise from big companies like Microsoft, Yandex, Red Hat, Akamai.
86
00:10:04.954 --> 00:10:11.454
And 1 of the things that you have to do at a scale is, like, you kinda have to basically automate yourself out of a job almost.
87
00:10:11.995 --> 00:10:12.495
So
88
00:10:12.800 --> 00:10:15.779
1 of the most challenging things actually in running any storage system
89
00:10:16.160 --> 00:10:16.660
is
90
00:10:17.040 --> 00:10:21.779
for the person that is running the system, not the expert that wrote the system,
91
00:10:22.345 --> 00:10:24.764
is for this operator to understand,
92
00:10:25.305 --> 00:10:27.725
honestly, what are the best settings for my computer?
93
00:10:28.185 --> 00:10:47.565
You know, like, how much memory do I allocate in particular for the JVM? Like, what's the what's the garbage collection? Like, how do I tune these these things? So, of course, eliminating the JVM kind of eliminates an entire category of problems. Right? So that kinda makes it simple. The second thing that we do is we try to really deeply understand what are the kernel parameters
94
00:10:48.090 --> 00:10:52.350
that affect the runtime performance both for throughput and latency of RedPand.
95
00:10:53.050 --> 00:11:06.759
And instead of exposing this as a list of checklists that say, you know, run this file system, disable no merges on the Linux IO scheduler, enable your multi queue network, and kind of all of these very deep technical things, we ship an auto tuner.
96
00:11:07.060 --> 00:11:09.480
And so even before Red Panda starts,
97
00:11:09.860 --> 00:11:15.514
we guarantee that by the time Red Panda starts, the binary starts, which by the is is just a single file,
98
00:11:16.135 --> 00:11:23.070
the kernel has the optimal settings. I'll tell you 3 or 4 things that we do to optimize sort of the experience
99
00:11:23.690 --> 00:11:30.430
for the person that's managing the storage system. So the first thing that we do is we probe the hardware for
100
00:11:30.795 --> 00:11:40.095
whether the mult whether the NIC can support multi queue networking. And if it does, we can enable multi queue networking. The second thing that we do is we detect if you're running an SSD.
101
00:11:40.510 --> 00:11:47.090
And if you are, we probe the hardware and then we measure with some target latency of 500 microseconds.
102
00:11:47.665 --> 00:11:50.725
How much data can we push to this NVMe device
103
00:11:51.185 --> 00:11:52.485
given this target latency?
104
00:11:52.945 --> 00:11:57.045
The third thing we do is we tune all other couple of Linux kernel settings.
105
00:11:57.450 --> 00:12:14.720
1 big 1 that has an impact is because we use DMA, because we don't use the page cache, because we talk effectively to the device controller, we tell the Linux controller not to merge our IO blocks. And in fact, I was like, don't do anything because we, the application, understand much better how to align our memory buffers
106
00:12:15.339 --> 00:12:17.839
so that, you know, they're basically block aligned
107
00:12:18.220 --> 00:12:21.600
like the file system expects them. And so all of this is automated.
108
00:12:22.085 --> 00:12:44.305
This kind of a step forward in managing the system is really a system that kind of gets out of the way. You run 1 binary. You do a systemctl star red panda. If you run-in system d, it auto tunes the machine and then it runs the thing. Of course, this happens super fast, which is very anticlimactic when I'm trying to show this to my wife. I was like, look at all the things that we do, and it happens in, like, a 100 milliseconds. But from a management perspective,
109
00:12:44.765 --> 00:12:46.785
that's sort of that operational simplicity
110
00:12:47.245 --> 00:12:59.725
is really what the CEOs are buying. The technical person, the engineer in the room, they wanna hear all the details about how we do, you know, debounce the rights to raft. How do we batch thing? How do we improve latency?
111
00:13:00.105 --> 00:13:13.060
How do we do kind of all these low level things, whether it's from the data management the data structure perspective at the core. But what the CIO cares about is, like, okay. How hard is it to run this system? That's the first thing that they care about. And secondly,
112
00:13:13.565 --> 00:13:22.385
what are the things that you're improving on on the status quo? Given this 1 binary is really, I think, the only way the the reason why we're getting so much traction with some of our customers,
113
00:13:22.759 --> 00:13:36.705
Because not only do we improve the safety guarantees of Kafka, the throughput, we give lower latency, we give higher throughput, we give them safety, But we also, from a management perspective, actually reduce the operational overhead of what it means to run a Kafka cluster on a scale.
114
00:13:37.165 --> 00:13:39.905
You mentioned that there are some areas for improvement
115
00:13:40.269 --> 00:13:41.170
on the specifics
116
00:13:41.630 --> 00:13:47.490
of how Kafka is implemented. But at the same time, the overall ecosystem around Kafka is obviously
117
00:13:48.135 --> 00:13:57.595
large and vibrant enough for you to target it as being a drop in replacement. So I'm wondering if you can talk through the overall strengths of the surrounding ecosystem
118
00:13:58.120 --> 00:14:00.380
and why it is that you are
119
00:14:00.760 --> 00:14:06.555
choosing to be an easy replacement for Kafka rather than building your own protocols and trying
120
00:14:07.035 --> 00:14:09.375
to grow your own ecosystem in place of it?
121
00:14:09.755 --> 00:14:13.435
Yeah. We have to really give credit to Kafka here. What Kafka is
122
00:14:14.010 --> 00:14:19.710
what they figure is Kafka has won as the de facto API. Right? We we talk to
123
00:14:20.170 --> 00:14:24.404
actually, literally, hun by now, 100 of enterprises, probably around a 100 enterprises,
124
00:14:24.785 --> 00:14:27.605
high level executives about running Kafka, and fundamentally,
125
00:14:28.065 --> 00:14:39.415
the Kafka API won. And if you think about the Kafka API has actually been the de facto standard, they have a huge ecosystem. There are millions of lines of code. You can take TensorFlow
126
00:14:39.955 --> 00:14:47.095
and push it into a topic and then have that consumed by a Spark ML and push it back onto a topic. And then eventually that makes it to Elasticsearch.
127
00:14:47.630 --> 00:14:54.450
And all of this sudden, with 3 or 4 little processes, you have a very sophisticated machine learning pipeline. Right? And so the API
128
00:14:54.990 --> 00:14:58.835
is really, in my opinion, the thing that won. Now if you compare
129
00:14:59.295 --> 00:15:00.515
the Kafka API
130
00:15:01.135 --> 00:15:04.275
to what SQL was for all of these databases,
131
00:15:04.680 --> 00:15:09.819
is kinda really how we're treating the Kafka API. We said, okay, this protocol has 1. Developers
132
00:15:10.279 --> 00:15:20.135
love it, and fundamentally, there's millions of lines of code that are written into this, and we wanna plug into that ecosystem. We wanna give them a better system for it. If you compare
133
00:15:20.675 --> 00:15:21.415
Red Panda
134
00:15:21.730 --> 00:15:23.030
and the Kafka API
135
00:15:23.490 --> 00:15:29.430
with SQL and kind of the whole evolution on, you know, you went from relational databases to NoSQL databases
136
00:15:29.810 --> 00:15:34.485
to, like, Cassandra. Right? So it was Postgres, and then everyone moved to Cassandra type systems.
137
00:15:34.865 --> 00:15:39.605
And then the last 1 is the NewSQL or the CockroachDB system, which gives them, you know, distribution,
138
00:15:40.250 --> 00:15:43.230
geo distribution, and so on. We really think that
139
00:15:43.530 --> 00:16:03.650
the Kafka API really needs a new system that gives people the guarantees that they're used. Right? So there's this new stack of people that don't wanna run JVM system, that want a system that integrates well with Kubernetes and all of these modern systems, but it's easy to run. It gives them better safety, or it gives them kind of the things that they expect the queue to give them. But in terms of the Kafka API,
140
00:16:04.065 --> 00:16:05.365
it was hugely important.
141
00:16:05.905 --> 00:16:12.965
And, honestly, from a business perspective, it was important that we leverage their system. There was no way actually, the first couple of months
142
00:16:13.380 --> 00:16:28.575
of the company, I tried to boil the ocean. And and clearly, it didn't work. I was like, look. I'm gonna invent this new API. We're gonna solve a bunch of things, like head of line blocking and all of this, you know, very low level of details. And then I didn't take into account this this super rich ecosystem.
143
00:16:29.089 --> 00:16:32.470
But after talking with a bunch of enterprises, it kind of became clear
144
00:16:32.930 --> 00:16:45.574
that what people wanted was not to touch their production application. They just wanted to point their application to a better system, and off you go at 10 x the speed. That's really what they wanted. And they want a reduction in hardware.
145
00:16:45.900 --> 00:16:53.520
They wanted to get back, for example, engineering capacity, because now you can run the same system with a part time person rather than 2 full time people.
146
00:16:53.875 --> 00:16:57.415
So we really thought that the Kafka API was, like, this super rich ecosystem
147
00:16:57.795 --> 00:16:59.495
that where we can give better guarantees
148
00:16:59.875 --> 00:17:01.735
without breaking any existing application.
149
00:17:02.510 --> 00:17:10.690
And so can you dig a bit more into how Red Panda itself is actually implemented and some of the ways that you are able to be a drop in replacement for Kafka?
150
00:17:11.385 --> 00:17:13.325
Kafka has actually
151
00:17:13.865 --> 00:17:15.565
50 current different APIs.
152
00:17:16.825 --> 00:17:24.880
And each API, by the way, has 12 different versions. So let's look at fetch, for example. Just basically the consume API. Let's see. There are 12 versions,
153
00:17:25.340 --> 00:17:28.480
and then you have an encoder and a decoder
154
00:17:28.825 --> 00:17:29.565
per version,
155
00:17:30.105 --> 00:17:47.395
per request and response. Right? So I send the message, and that has an encoder and decoder, so 2 things. And then I received a message that has an encoder and a decoder, 2 more things. And for those 4 things, you have 12 particular versions. So there's no way we were going to manually generate this. And so
156
00:17:47.695 --> 00:17:48.435
we spent
157
00:17:48.895 --> 00:17:54.355
the bulk of late last year honestly writing a parser combinator library where we could just cogenerate
158
00:17:55.240 --> 00:18:00.860
the entire Kafka API ecosystem. And so Kafka has these JSON files, and and it describes
159
00:18:01.720 --> 00:18:04.059
it's like a variant of JSON, but it describes
160
00:18:04.445 --> 00:18:07.665
the type kind of what the JVM type system expects.
161
00:18:07.965 --> 00:18:22.434
So we can consume those files directly, and then we added an enhancement of that with some of the c plus plus 20 features to actually give stronger guarantees and prevent some abuse from the native API. That's sort of the first level. And so if you look at Feds, for example,
162
00:18:22.735 --> 00:18:32.509
it would be 48 different scenarios that you have to take into account. So there's no way we would be able to keep up with the Kafka. Like, we wouldn't have a product if we had to manually generate this a 160 APIs.
163
00:18:32.809 --> 00:18:37.230
A couple more deep level kind of technical things, you know, how we guarantee compatibility.
164
00:18:37.905 --> 00:18:38.965
The next 1 is
165
00:18:39.425 --> 00:18:57.684
because the ecosystem is so rich, right, and we really are just targeting at the API level. We say, talk to this TCP endpoint, and then we look like Kafka, we talk like Kafka, there's exactly no difference, and all of your applications work, which means we can download all of these open source projects that are already interacting with Kafka,
166
00:18:57.985 --> 00:18:59.765
that rely on bug compatibility
167
00:19:00.465 --> 00:19:17.305
in, you know, in the case of like the Saramaj driver, they have some particular bugs that we had to like imitate, which by the way upstream does the same. And so we imported a large set of of unit tests from the Liberty Kafka test, and then we just pointed at at our IP. And because we speak Kafka, it sort of simplifies this giant
168
00:19:17.925 --> 00:19:29.825
set of testing matrix and that all we have to do is run these open source systems and put them on Amazon or Google. It it really doesn't matter. And they say, you know, test this and let us know where we failed. Those are the 2 kind of guarantees.
169
00:19:30.205 --> 00:19:30.945
And then
170
00:19:31.325 --> 00:19:35.664
on the safety level, you know, we hired Dennis out of the Cosmos DB team
171
00:19:36.030 --> 00:19:37.570
to kinda come in and implement
172
00:19:38.030 --> 00:19:40.290
Jepsen. In fact, we actually extended Jepsen
173
00:19:40.750 --> 00:19:48.975
to be able to analyze longer histories and without getting too much of the rabbit safety, which we can next. So we give them safety and API compatibility,
174
00:19:49.435 --> 00:19:50.735
and all of the product
175
00:19:51.035 --> 00:19:57.610
features actually have an integration test. 1 of the most fun things that that we spent probably last year,
176
00:19:57.910 --> 00:19:59.210
a couple of months writing,
177
00:19:59.510 --> 00:20:05.695
this really interesting fault injection test. And we've been running this 247 since last year. And every a 150
178
00:20:06.075 --> 00:20:06.575
seconds,
179
00:20:06.875 --> 00:20:10.495
we inject a fatal failure, whether we delete
180
00:20:10.850 --> 00:20:13.830
like, a rough group or we crash the file system
181
00:20:14.450 --> 00:20:18.870
or we inject network partitions, we delete topics, we create topics,
182
00:20:19.215 --> 00:20:26.355
and we have to give the system, you know, time to recover because at the same time, by the way, we're pushing a gigabyte of data per second. And so we can simulate
183
00:20:26.920 --> 00:20:42.705
a much larger fault scenario than most of our customers can. And we know because we are the expert, kinda what are the difficult things. And in addition to that, we run them to all of these, you know, fault exploration frameworks, these these safety frameworks, these compatibility frameworks.
184
00:20:43.005 --> 00:20:44.945
And that's how we are trying to guarantee
185
00:20:45.279 --> 00:20:46.580
safety and and compatibility.
186
00:20:47.600 --> 00:20:50.659
Another interesting element of this is that
187
00:20:51.039 --> 00:20:54.740
while Kafka has become the de facto API for
188
00:20:55.265 --> 00:20:56.085
streaming storage systems,
189
00:20:56.625 --> 00:20:57.605
there are some
190
00:20:58.145 --> 00:20:59.125
other contenders,
191
00:20:59.425 --> 00:21:11.370
most notably being Pulsar, which has also admittedly added a protocol compatibility layer for Kafka. But then there's also the work being done for the open streaming standard to try and
192
00:21:11.705 --> 00:21:18.284
coalesce around a particular interface for streaming applications. And I'm wondering what your overall thoughts are on the
193
00:21:18.799 --> 00:21:20.900
viability of some of these other protocols
194
00:21:21.360 --> 00:21:22.740
and the potential
195
00:21:23.200 --> 00:21:34.005
areas for improvement in the Kafka APIs and ways that the overall ecosystem might be able to progress to a shared understanding of how best to interact with these types of systems?
196
00:21:34.730 --> 00:21:48.675
Let's tackle each of those in particular. And I think there's also a set of things we haven't yet talked about, which is what we see as the future of streaming. In a way, as a company, we're trying to position ourselves as the future of streaming for the new stack.
197
00:21:48.975 --> 00:21:52.690
So talk about Pulsar. From a market distribution perspective,
198
00:21:53.150 --> 00:21:58.530
I think something like 90% of all the companies that we talk to use Kafka. There's some companies that that use Pulsar.
199
00:21:58.925 --> 00:22:15.350
Maybe there's bias in the people that want to talk to us. But that's just, you know, the stats that we're seeing. There's very few people that are running NATS and all the other streamings, you know, RabbitMQ, there's very, very few. And just really in terms of the market. I was actually speaking with an executive from IBM the other day, and literally Kafka
200
00:22:15.955 --> 00:22:23.255
crashed multibillion dollar products for IBM. Like, that's how popular it is. And I think the reason is not necessarily
201
00:22:23.794 --> 00:22:27.559
the protocol, but the amount of connectors. Right? The fact that you can take TensorFlow
202
00:22:28.179 --> 00:22:30.279
and plug it straight into Elasticsearch
203
00:22:30.659 --> 00:22:33.320
or plug it into a Cassandra compatible database
204
00:22:33.825 --> 00:22:35.445
or, you know, push it to DynamoDB.
205
00:22:35.745 --> 00:22:36.804
It's kind of this
206
00:22:37.105 --> 00:22:50.490
Lego component for a lot of the architectures, whether you're trying to do it, to use it to detect real time fraud, or you're trying to use Kafka to deal with back pressure from databases that have a hard time with the scaling, like Elasticsearch,
207
00:22:50.965 --> 00:23:03.440
is the thing that is 1. So to answer your question about streaming and the protocol, I think it's great. I think the great thing about open source and the market is that people will build the applications that they need, and they don't care
208
00:23:03.900 --> 00:23:05.360
whether it is a protocol
209
00:23:05.660 --> 00:23:11.845
or it is a standard or something. They're just trying to get work done. And the majority of people are getting work done with
210
00:23:12.225 --> 00:23:25.794
Kafka. In a way, it's really easy for us to provide another parser mechanism or, like, you know, change the format from the Kafka format to JSON. It doesn't matter. But I think what matters is really this plug and play
211
00:23:26.095 --> 00:23:27.794
architecture where you get to leverage
212
00:23:28.255 --> 00:23:38.580
all of the Kafka connectors. And in fact, all of the new database, whether it's like OmniSight or CillaDB or CockroachDB, they even all actually, main SQL too has a transactional Kafka connector.
213
00:23:38.880 --> 00:23:41.139
So the Kafka API is so popular
214
00:23:41.475 --> 00:23:46.615
that all of these modern database systems actually have built in customized drivers to consume
215
00:23:47.075 --> 00:23:52.170
data from Kafka in a way that gives them higher and better guarantees or different guarantees
216
00:23:52.630 --> 00:23:55.130
than the standard connectors. So I'm not sure
217
00:23:55.430 --> 00:24:01.005
whether the open messaging standard is ultimately going to win. We're just trying to give users
218
00:24:01.465 --> 00:24:02.524
a better system
219
00:24:02.825 --> 00:24:04.205
for their existing code
220
00:24:04.585 --> 00:24:17.535
currently. It's really hard to tell. I think it's great that there are, you know, innovations in this space, and it would make, honestly, a lot of our jobs simpler if it wasn't the Kafka API, if it was, like, a standard that had, like, you know, well understood semantics.
221
00:24:17.835 --> 00:24:19.775
Some of the things that we have to guarantee
222
00:24:20.315 --> 00:24:21.215
are really hard
223
00:24:21.595 --> 00:24:26.370
because it's not like the bugs are not documented, of course, And you really kinda have to run an application,
224
00:24:26.750 --> 00:24:41.225
download a bunch of open source project to really detect these edge conditions, which is why we leverage the entire open source ecosystem to test our code. But ultimately, I think the Kafka APIs, I think that's 1 today. It's very cool that the other messaging frameworks also take off.
225
00:24:41.820 --> 00:24:49.120
Going a little bit further down this path, the other aspect to the streaming ecosystem and 1 of the things that Pulsar
226
00:24:49.525 --> 00:24:59.150
touts as an advantage is that it also supports the more traditional messaging paradigm similar to what RabbitMQ does, where it doesn't rely on this persisted
227
00:24:59.530 --> 00:25:09.015
append only log stream for being able to send messages between systems. And I'm wondering what your thoughts are on the utility of that being collocated
228
00:25:09.315 --> 00:25:13.175
within a system such as Kafka or Red Panda or Pulsar
229
00:25:13.750 --> 00:25:20.490
versus running it as a dedicated system where you're relying on some of the time tested applications such as RabbitMQ?
230
00:25:21.395 --> 00:25:25.975
The request response for for messaging, I think is it's super useful.
231
00:25:26.675 --> 00:25:30.215
I think the the answer to that in the Kafka ecosystem
232
00:25:30.559 --> 00:25:31.700
is a thing called
233
00:25:32.000 --> 00:25:37.140
Kafka REST or like a proxy. We actually have our own. It's written in c plus plus API compatible,
234
00:25:37.600 --> 00:25:39.940
where you kinda send and and receive messages.
235
00:25:40.345 --> 00:25:50.909
Underneath the scenes, it kind of proxies the protocol from REST for this request response to the Kafka protocol. For the majority of use cases that we talk to, which are
236
00:25:51.210 --> 00:25:52.669
either large web 2.0
237
00:25:53.130 --> 00:25:58.110
companies, like the largest database in the world, analytical database in the world, the largest CDN,
238
00:25:58.475 --> 00:26:12.520
people doing fraud detection for the largest banks, or on the hedge fund side where they're trying to do settlements. They really care quite a bit about data loss as you would imagine. Those are probably our focus. So I haven't really seen that much demand from a market perspective
239
00:26:12.900 --> 00:26:19.184
to run a lot of Kafka proxy stuff. In fact, I the real driver for the Kafka proxy is
240
00:26:19.565 --> 00:26:20.765
the Kafka driver
241
00:26:21.245 --> 00:26:23.265
when I mean the driver, I mean the client code
242
00:26:23.720 --> 00:26:27.420
has a lot of knowledge. It is a very sophisticated piece of technology.
243
00:26:28.040 --> 00:26:36.725
And the reason to use the proxy is really to bridge gaps. So when you connect to Kafka via whether it's Go or or c plus plus or Java,
244
00:26:37.105 --> 00:26:40.405
you know, really any driver, but those are the most popular ones.
245
00:26:40.960 --> 00:27:00.280
The driver has to understand the internal IPs because it does literal election on the consumer groups. It, you know, has, like, a lot of protocols embedded into the client, and so it needs to understand the internal cluster state. It actually fetches all the metadata from the controller and so on. And so the HTTP proxies, I think, Les used a way to replace,
246
00:27:00.820 --> 00:27:01.320
RabbitMQ
247
00:27:01.700 --> 00:27:03.000
in the request response
248
00:27:03.540 --> 00:27:04.200
type workload
249
00:27:04.740 --> 00:27:08.695
and more actually to bridge external systems. So if you're trying
250
00:27:09.475 --> 00:27:13.414
to proxy your streaming rights to your infrastructure via an HTTP proxy,
251
00:27:13.794 --> 00:27:21.880
then you would put our, you know, c plus plus proxy in front of Kafka. And then anyone that talks HTTP can talk straight to this proxy
252
00:27:22.340 --> 00:27:25.000
without having to understand the mechanics or the internal
253
00:27:25.380 --> 00:27:26.279
client protocols.
254
00:27:26.865 --> 00:27:31.205
And for the majority of whether it's your machine learning applications or your fraud detection
255
00:27:31.585 --> 00:27:37.900
or your real time ad bidding, most people just really use that. And I think that Kafka ultimately replaced RabbitMQ,
256
00:27:38.680 --> 00:27:40.620
at least in the market and what we've seen.
257
00:27:41.160 --> 00:27:44.195
And going back to the specifics of your implementation,
258
00:27:44.575 --> 00:27:47.395
what are some of the ways that the overall design
259
00:27:47.775 --> 00:27:55.690
or the product direction has changed or evolved since you first began working on it, and any initial assumptions that you had going into
260
00:27:56.150 --> 00:28:00.705
the project that have been updated or invalidated in the process?
261
00:28:01.485 --> 00:28:03.345
I would say the main
262
00:28:03.725 --> 00:28:05.185
thing that changed for us
263
00:28:05.779 --> 00:28:08.200
was our ability to add coprocessors.
264
00:28:08.820 --> 00:28:15.635
So let me give you kind of a little bit of background here. For most APIs currently, right, let's use gRPC
265
00:28:16.335 --> 00:28:21.554
to be executive with our time. You define inside a protocol file. You define
266
00:28:21.855 --> 00:28:25.620
the data exchange format, so, like, the data structure you're gonna exchange,
267
00:28:26.160 --> 00:28:31.220
and then you run it through an ideal generator, like protoc or g r p gRPC compiler,
268
00:28:31.565 --> 00:28:34.945
and it takes the schema and it generates HTTP stubs
269
00:28:35.405 --> 00:28:37.665
for a multitude of languages, Python,
270
00:28:38.045 --> 00:28:41.050
Java, you know, it doesn't matter. And that's roughly how, like, a lot of
271
00:28:48.890 --> 00:28:53.785
serialization format, a known h point, and a known way to interact with these endpoints.
272
00:28:54.325 --> 00:29:04.410
What doesn't exist to this day in a way that is scalable, and I'll mention the the specific details in a second, is really a way to provide users with a data API
273
00:29:04.790 --> 00:29:06.570
in a way that goes
274
00:29:07.030 --> 00:29:08.250
beyond serialization,
275
00:29:08.915 --> 00:29:12.215
in a way that gives deep data guarantees.
276
00:29:12.835 --> 00:29:16.455
And so something that has evolved from the beginning of the product
277
00:29:16.770 --> 00:29:18.630
was our idea of coprocessors.
278
00:29:19.090 --> 00:29:20.870
So what's a coprocessor is
279
00:29:21.170 --> 00:29:23.429
Red Panda allows for inline WASM
280
00:29:23.955 --> 00:29:24.455
transforms.
281
00:29:24.914 --> 00:29:39.640
So as you're sending the data, we can run it through, yeah, through, like, a WASM transform so you can upload either a JavaScript or WASM. We run it inside the V 8 engine, and then we can materialize that transformation to a different topic. This may sound trivial, but I'm gonna walk you through example.
282
00:29:40.015 --> 00:29:47.075
1 of our customers where we took 5 clusters of 20 nodes each, and we reduced it to 1 20 node cluster.
283
00:29:47.419 --> 00:29:49.840
And this goes to the idea of the data API.
284
00:29:50.140 --> 00:29:52.880
So what they were doing is that they were sending information.
285
00:29:53.340 --> 00:29:59.125
And for the sake of this discussion, let's say that it had 3 fields, first name, last name, and some PII information.
286
00:29:59.585 --> 00:30:02.245
Right? And so the other cluster were literally
287
00:30:02.865 --> 00:30:06.805
materialized views of this original cluster. And the way it's done today
288
00:30:07.170 --> 00:30:32.519
is you either use, Kafka streams or Flink or Spark or some other job to consume data from 1 topic and push it to another topic. We think that that is not scalable. The reason it's not scalable is because you're consuming network resources from the clusters that you're consuming to and producing to. And so our answer to that and we're still in early stages of dates, and we think this is the right approach. Ultimately, the market will decide.
289
00:30:33.045 --> 00:30:37.865
But given people the ability to upload a Wasm transform scripts, basically,
290
00:30:38.485 --> 00:30:38.985
decouples
291
00:30:39.445 --> 00:30:43.660
the production of data from the consumption of data with a data API.
292
00:30:44.040 --> 00:30:49.660
So 1 example here is you can push some person object with some PII information.
293
00:30:50.304 --> 00:30:55.044
And so, of course, the simple thing to do is you say, okay. I'm gonna remove all of the PII fields
294
00:30:55.985 --> 00:31:01.010
and push them to a separate topic. So let's take change data capture. You have Debezium pushing
295
00:31:01.470 --> 00:31:02.290
your transactions
296
00:31:02.670 --> 00:31:07.090
to a database, and now you can run an inline transform, and you can guarantee
297
00:31:07.710 --> 00:31:08.610
that the consumers
298
00:31:09.230 --> 00:31:10.755
of of a particular topic
299
00:31:11.135 --> 00:31:13.075
will not see PII information.
300
00:31:13.535 --> 00:31:23.560
But then you can even go beyond that. Right? What you can say is, like, I am going to transform the same 3 fields, first name, last name, and some PI information. Let's take Social Security number,
301
00:31:23.895 --> 00:31:30.554
and it's gonna have the same 3 fields. So it's not just about removing or adding fields or data enrichment, which by the way is useful,
302
00:31:30.919 --> 00:31:33.980
but it's about changing the meaning of the data.
303
00:31:34.440 --> 00:31:37.820
And so, you know, being able to upload inline transformations
304
00:31:38.360 --> 00:31:40.620
is I think what is fundamentally missing
305
00:31:41.065 --> 00:31:51.870
from being able to give people API like guarantees like you do for your microservices, but now this is for your data. You can upload and you can say, oh, I am going to divorce
306
00:31:52.409 --> 00:31:57.549
the producers of this data and the version in this game of my sources of data and guaranteed
307
00:31:58.155 --> 00:32:05.535
whether it's for your machine learning, workloads, whether it's a uniform contract, a uniform serialization format with a uniform set of fields,
308
00:32:05.840 --> 00:32:11.460
or I'm gonna enrich some data with a little bit of things. This 1 shot transformation, this is, I think, what is missing
309
00:32:11.840 --> 00:32:20.005
from being able to truly expose the idea of the data APIs, which is, again beyond serialization. It's about deep introspective guarantees
310
00:32:20.465 --> 00:32:21.285
of the data.
311
00:32:21.585 --> 00:32:24.245
And what's useful about our particular implementation
312
00:32:24.910 --> 00:32:27.810
is that you get parent stream correctness guarantees,
313
00:32:28.270 --> 00:32:28.930
which means
314
00:32:29.310 --> 00:32:40.384
because they execute locally, you're not consuming network transfer. You're not consuming CPU. You're simply pushing it through a materialization engine, like a V 8 engine, and say, hey. Run this function.
315
00:32:40.840 --> 00:32:47.740
And whatever I get back, I'm gonna write it to 2 topics. Right? So it's very cheap for us to do it in a very scalable way, because the only thing that is increasing
316
00:32:48.120 --> 00:32:50.565
is disk. But from a business perspective,
317
00:32:50.865 --> 00:32:57.365
it now gives you the ability to guarantee GDPR compliance. You can say all of the tuples that come through this stream
318
00:32:57.670 --> 00:32:58.570
have guaranteed
319
00:32:59.270 --> 00:33:02.650
removed the Social Security number for everyone in here, or have removed
320
00:33:02.950 --> 00:33:07.145
their driver's license information, or have masked their address, and so on.
321
00:33:07.545 --> 00:33:16.445
Yeah. It's definitely very interesting and useful approach to that problem. And that brings me to my next question too, which is trying to explore
322
00:33:16.990 --> 00:33:34.785
some of the ways that you are able to innovate on the overall space beyond just performance when you are tying yourself to the specifics of the API and some of the room for growth or other areas of the streaming ecosystem that are not being thoroughly explored at the current point in time?
323
00:33:35.220 --> 00:33:38.039
That's always so fun. I think that the most fun
324
00:33:38.419 --> 00:33:40.200
as a technology is building.
325
00:33:40.740 --> 00:34:06.085
So every time someone asked me this question, it's it's a lot of fun for me to talk about. We're solving real problems right now, which is to some extent, we had to start with something. You know? You know, I had Emacs and GCC on my computer, and I was like, okay. I'm gonna build the system. 18 months later, we have the system that is compatible with Kafka. And that's where we had to start. Right? We had to start with better guarantees, better performance, kind of all of these things that we've been talking about through the podcast. I think the future though
326
00:34:06.465 --> 00:34:08.404
is much more interesting, in particular,
327
00:34:08.944 --> 00:34:10.325
on the disaster recovery
328
00:34:10.865 --> 00:34:15.049
and the compliance space and the idea that we call the shadow indexing.
329
00:34:15.510 --> 00:34:17.130
And so shadow indexing
330
00:34:17.750 --> 00:34:24.305
is actually you push your data to the Red Panda cluster. Right? And instead of deleting files locally,
331
00:34:24.765 --> 00:34:29.425
before we delete the file locally from this, we upload them to s 3 or Google Cloud
332
00:34:29.840 --> 00:34:32.820
storage or Azure Blob. It doesn't matter to some durable,
333
00:34:33.280 --> 00:34:44.785
very cheap storage. Right? So that's the first 1. So that that ability is tier to storage in you know, Kafka's working on a KIP and Pulsar has something like that, but that's not enough. What we also upload
334
00:34:46.009 --> 00:35:09.420
is the shadow index, and it's basically a self discovery metadata about the state. What that allows us to do is add the ability to have a true global Kafka cluster. How does that work? So let's say that you're running to US east and you wanna read from US west. So US east post their data and, you know, it's working as a regular Red Panda cluster. It uploads their data to s 3.
335
00:35:09.720 --> 00:35:15.484
When US west wants to read data that was written to US East, instead of using MirrorMaker,
336
00:35:15.944 --> 00:35:21.360
which is how most of this is done. Right? You you have this thing. It kinda like mirrors the topic,
337
00:35:21.740 --> 00:35:23.280
and it consumes resources
338
00:35:23.820 --> 00:35:29.985
both on the raft level, on the disk level, on the compression, on the network, on both sides of the cluster.
339
00:35:30.445 --> 00:35:37.990
We tell s 3 the API, and I was like, hey, s 3. Copy object, which is is an API that effectively allows you to move
340
00:35:38.290 --> 00:35:56.880
the object or basically point the source to the other data center, let's say in US east. Okay. But fundamentally, what is shadow index? Shadow index then transfer a little bit of metadata. We're talking about kilobytes for, like, a petabyte range of of actually true data. So what it gives is we actually don't consume any resources, and we have this shadow index, this metadata
341
00:35:57.339 --> 00:36:00.400
that tells us, oh, if you wanna read the entire historical
342
00:36:00.779 --> 00:36:03.185
data of this topic, you can.
343
00:36:03.645 --> 00:36:10.385
It is on US east. Let me copy the data to US west s 3, and it's very cheap. It's, like, 2¢ per gigabyte,
344
00:36:10.850 --> 00:36:16.870
and only materialize what the client gives me. Why is this a game changer, I think, for the future of streaming?
345
00:36:17.170 --> 00:36:17.670
Fundamentally,
346
00:36:18.425 --> 00:36:20.925
what you're giving people is the unification
347
00:36:21.385 --> 00:36:23.645
of the historical file system access
348
00:36:24.185 --> 00:36:28.300
and the real time access. You're giving them the exact same Kafka API.
349
00:36:28.600 --> 00:36:44.280
And so, of course, on US West, if you request historical data that was written in US east, you know, it'll has to fetch it from s 3 and it'll materialize it on some scratch space of the file system, but it'll give you the data. Let's say as a machine learning person in an organization,
350
00:36:44.820 --> 00:36:48.680
which by the way, it keeps the access control and stuff. You don't have to understand
351
00:36:49.505 --> 00:36:56.805
where the data comes from. Right? Like Red Panda will manage the hierarchy of data. It's not just uploading. It's I think it's the ability to query
352
00:36:57.299 --> 00:37:00.920
historical and real time with the exact same Kafka API.
353
00:37:01.220 --> 00:37:05.400
So that's just sort of all offloaded onto here. And I think really that idea
354
00:37:05.700 --> 00:37:07.115
of, like, leveraging
355
00:37:07.575 --> 00:37:08.555
the cloud providers
356
00:37:09.175 --> 00:37:09.675
native,
357
00:37:09.975 --> 00:37:26.055
you know, truly becoming cloud native. I think cloud native is really thrown around. My view of it is you leverage the cost saving infrastructure of Amazon with s 3 or or Azure, all of these dynamic compute resources to build a better system. That is what, to me, what is cloud native. And so s 3
358
00:37:26.595 --> 00:37:29.415
allows us to give them a global view
359
00:37:29.880 --> 00:37:39.915
of Kafka with unlimited replicas in any cluster that is super cheap. Right? I mean, it's it's literally 2¢ per gigabyte transfer, and that's what you're paying on your on your s 3 bill
360
00:37:40.295 --> 00:37:55.190
without the need to materialize it locally and for that cluster to have overload. So from a system performance perspective, you could literally proxy the reads of petabytes and petabytes of data with actually a very small memory footprint on the box. So that's a big 1. That
361
00:37:55.515 --> 00:38:00.174
coupled with our this inline WASM transform, right, this 1 shot transformations,
362
00:38:00.635 --> 00:38:08.610
which, you know, it doesn't solve all the use cases if you're merging a a ton of streams and, you know, there are other systems like Flink is is is excellent, in my opinion.
363
00:38:08.910 --> 00:38:13.315
They they do a better job of, like, merging a stream and doing more computation. But this 1 shot transformation,
364
00:38:13.695 --> 00:38:15.234
given data API guarantees
365
00:38:15.855 --> 00:38:28.260
combined with our idea of shadow indexing, is I really think is sort of the the future of streaming. It's like, you know, read it anywhere. It's scalable. It's cheap. You can run it. It gives you safety guarantees. It give you correctness guarantees.
366
00:38:28.615 --> 00:38:29.595
We give you a linearizable
367
00:38:29.895 --> 00:38:30.795
raft implementation.
368
00:38:31.255 --> 00:38:34.155
We give you, you know, some low latency, some high throughput
369
00:38:34.455 --> 00:38:35.435
for every cluster.
370
00:38:38.010 --> 00:38:44.670
If you're looking for a way to optimize your data engineering pipeline with instant query performance, look no further than cubes.
371
00:38:45.505 --> 00:38:53.365
Cubes is next generation OLAP technology built for the scale of big data from UST Global, a renowned digital services provider.
372
00:38:54.040 --> 00:39:02.674
Cubes lets users and enterprises analyze data on the cloud and on premise with blazing speed while eliminating the complex engineering required to operationalize
373
00:39:02.974 --> 00:39:04.194
analytics at scale.
374
00:39:04.895 --> 00:39:06.994
With an emphasis on visual data engineering,
375
00:39:07.375 --> 00:39:10.914
connectors for all major business intelligence tools and data sources,
376
00:39:11.280 --> 00:39:16.900
cubes allows users to query OLAP cubes with sub second response times on 100 of billions of rows.
377
00:39:17.440 --> 00:39:21.515
To learn more and sign up for a free demo, visit data engineering pod cast.com/cubes
378
00:39:22.855 --> 00:39:25.195
today. That's data engineering podcast.com/qubz.
379
00:39:30.420 --> 00:39:46.715
For people who are interested in exploring Red Panda and deploying it into production or a test environment, what is actually involved in getting a cluster set up and joining the nodes together and managing the auto tuning that you mentioned? And I'm also curious
380
00:39:47.070 --> 00:39:56.945
to understand some of the challenges or complexities that you face in being able to automatically discover those optimal settings and how cloud environments complicate that problem.
381
00:39:57.645 --> 00:40:04.600
By the time this podcast is out on September 22nd, we're gonna have a free download. People can just go to our website, but put it on the link of the
382
00:40:04.900 --> 00:40:05.400
podcast.
383
00:40:05.700 --> 00:40:19.569
They could try it themselves. They get their own kind of private repo. The auto tuning and discovery, it happens as long as you're running system d. This auto tuning, because it's hardware, it literally runs at hardware speed, which is, like I mentioned, is really anticlimactic
384
00:40:19.869 --> 00:40:21.089
when you're trying to show
385
00:40:21.390 --> 00:40:22.369
someone new camera
386
00:40:22.750 --> 00:40:33.865
that doesn't understand the system. You're like, oh, look. I did all these cool things, and it's just like, you know, a bunch of terminal lines. But anyway, so that happens automatically, and then Red Panda starts. Now the exciting part about this question is configuration.
387
00:40:34.549 --> 00:40:46.704
The reason we chose Raft you know, actually, the original protocol I wrote was chain replication. But when you add chain like, node removal and node addition and a bunch of stuff, you end up with a thing that looks exactly like Raft, except
388
00:40:47.085 --> 00:40:47.825
that Raft
389
00:40:48.125 --> 00:40:50.305
gives you a proof on configuration
390
00:40:50.605 --> 00:40:56.619
changes. There are 2 types of changes in Raft. I'm only gonna talk about the thing that is gonna be available on September 22nd,
391
00:40:56.920 --> 00:40:58.540
which is called joint consensus.
392
00:40:59.079 --> 00:40:59.819
And so
393
00:41:00.175 --> 00:41:02.675
the reason for us choosing Raft, in addition
394
00:41:02.975 --> 00:41:03.715
to safety
395
00:41:04.015 --> 00:41:05.155
and to having
396
00:41:05.615 --> 00:41:19.525
that this log completeness guarantee, which says there's no gaps in the log or you can't write data. It was configuration changes, sound configuration changes. And so when you add a node through the cluster, you have to give it basically 3 or 4 IPs
397
00:41:19.904 --> 00:41:40.765
just to for failure recovery, but it'll ask the current nodes. It's like, hey. Who is the leader in the cluster for the controller? It's a very low low throughput cluster. It really it's only used when you're creating a topic or you add a node. It's basically otherwise idle. So So there's 1 node in the cluster that does this, and it uses, like, the RAF leader election to move around in case there is failure. Anyways, you ask the controller,
398
00:41:41.120 --> 00:41:42.100
and then the controller
399
00:41:42.400 --> 00:41:44.180
initiates a joint consensus,
400
00:41:44.960 --> 00:41:45.460
protocol,
401
00:41:45.920 --> 00:41:51.515
which is exactly how you add or remove nodes. Right? So you can just expand or remove notes
402
00:41:51.815 --> 00:42:06.490
like you would. You know, you start up a thing, and then you'll go through this sound configuration changes, which means you add them, you know, you make sure that they have the controller metadata as updated and so on, and then they can join the cluster and start the voting. So that's how you add a node. It's really,
403
00:42:07.005 --> 00:42:19.890
ultimately, some of the systems have some of the open source systems, we're probably going to open source a large part of this later this year, have offloaded a lot of this complexity onto the users. And so instead, we unloaded the complexity
404
00:42:20.190 --> 00:42:21.730
of, like, node discovery,
405
00:42:22.445 --> 00:42:25.185
configuration changes onto the system itself,
406
00:42:25.485 --> 00:42:30.225
and then we added the complexity of having to implement the joint consensus algorithm
407
00:42:30.685 --> 00:42:39.010
to add and remove nodes. So it's really, I think, you know, quite trivial. It's like you would have on on any node. You say, I've got install Red Panda.
408
00:42:39.315 --> 00:42:44.775
It gives you the thing. You give it the seed IPs, and then then it's up and running from from that point.
409
00:42:45.395 --> 00:42:46.215
Digging into
410
00:42:46.515 --> 00:42:48.135
the business side of things,
411
00:42:48.500 --> 00:42:53.240
what is your overall approach to being able to grow this platform
412
00:42:53.780 --> 00:42:56.095
and expand its capabilities
413
00:42:56.395 --> 00:43:00.494
and ensure that it's sustainable from a technological and business perspective?
414
00:43:01.355 --> 00:43:04.895
The first thing that we had to do was really be API compatible.
415
00:43:05.210 --> 00:43:08.190
Basically, that opened up every enterprise in the world.
416
00:43:08.810 --> 00:43:13.790
It's amazing how many Kafka installations are in the world. I think I expect something like a 120, 000
417
00:43:14.835 --> 00:43:16.135
total Kafka installations
418
00:43:16.515 --> 00:43:17.095
to date.
419
00:43:17.395 --> 00:43:19.415
So the market for us is really huge.
420
00:43:19.795 --> 00:43:31.500
We have seen demands in the high volume space. So like pushing a couple of petabytes per week, you know, let's say 4, 5, 6 gigabytes per second, like huge, huge volumes per cluster,
421
00:43:32.065 --> 00:43:37.845
and in the finance sector. Right? And the reason for that is that they care about safety, and the only 1 cares about throughput and scalability.
422
00:43:38.305 --> 00:43:44.390
What we haven't really addressed is the middle of the market, the majority. And so, you know, what's good about where we started
423
00:43:44.770 --> 00:43:49.270
is an oil company is always gonna have money to process, like, IoT devices,
424
00:43:49.655 --> 00:43:55.035
same thing with the largest analytical space, same thing with the largest CDN and telco providers. That's like the big volume.
425
00:43:55.335 --> 00:43:58.234
On the finance side, you know, hedge funds really have no price sensitivity.
426
00:43:58.670 --> 00:44:03.570
So that's also a good market for us. But ultimately, this won't be like a massively successful product,
427
00:44:03.870 --> 00:44:30.444
I think, unless we give this product to the masses. And so on September 22nd, we're gonna have a free download trial where anyone can just try it. It's the the entire product. They can test it. They can see if what we're saying is true. Hopefully, they find bugs. It's great when your customers find faults and you fix them and you make the system better. And then ultimately, towards the later part of the year, we want to make this available for a lot of people that don't have the money to pay. And so what that means is
428
00:44:30.744 --> 00:44:33.164
probably a community license of sorts similar.
429
00:44:33.550 --> 00:44:37.810
My current thinking on this is similar to the CockroachDB license, where
430
00:44:38.190 --> 00:44:47.075
if you're a cloud provider, you have to pay. If you are vendor in data services where Kafka is kinda like, you know, they think that you really are selling plus or minus some proxies,
431
00:44:47.455 --> 00:44:57.430
you have to pay. But if you wanna use it, basically, all of the other users are free for people to use. And I think that's how we move forward. So that's 1. And then the second 1 is I think there's this largely
432
00:44:57.810 --> 00:44:58.310
unaddressed
433
00:44:58.770 --> 00:44:59.270
huge
434
00:44:59.570 --> 00:45:12.070
developer ecosystem, which is like, the JavaScript, the Python developers, and so on. Because our system is so easy to run, like, no 1 complains honestly about running Nginx or running Node. Js. It's like it's a single binary that you run.
435
00:45:12.370 --> 00:45:16.310
You know, I think it's to some extent, the reason why Cockroach has seen a lot of growth
436
00:45:16.610 --> 00:45:21.145
is the system is really easy to run. It was 1 binary, so we put a lot of effort
437
00:45:21.525 --> 00:45:24.585
into making it super, super easy to run, no tunables,
438
00:45:25.045 --> 00:45:25.545
autodiscoverable,
439
00:45:26.440 --> 00:45:29.100
where people could just get to focus on their applications.
440
00:45:29.560 --> 00:45:30.300
And so
441
00:45:30.680 --> 00:45:33.980
I think that's a large market that, you know, we're seeing some pickup traction
442
00:45:34.515 --> 00:45:35.815
where a lot of node
443
00:45:36.115 --> 00:45:42.214
developers, they wanna do real time fraud detection. They have the same problems that some of the Java enterprise shops
444
00:45:42.770 --> 00:45:47.830
have, but maybe they don't have the internal sophistication to manage JVM system, to understand ZooKeeper,
445
00:45:48.290 --> 00:46:13.855
to understand, like, Snapshot, TLS, literal election, kind of all of these really complicated things. And they just want a system that gets out of the way. And then 2, that they can interact with the streams and simply upload another, you know, JavaScript and Wazhm scripts to give you data API concept I've been talking about. And so I think those are how we address maybe the 3 parts of the market. So we already target the fortune, basically, 2, 000 with what we have today. They care about reducing
446
00:46:14.155 --> 00:46:16.015
the number of nodes from 300
447
00:46:16.619 --> 00:46:20.880
to 3. This is actually a real use case, and I can talk more later. They care about
448
00:46:21.260 --> 00:46:25.359
hardware reduction, cost, performance optimization, getting back engineering capacity
449
00:46:25.965 --> 00:46:31.585
on the finance side. So, like, very high end. They have no price sensitivity. They care about safety, data correctness, guarantees.
450
00:46:31.965 --> 00:46:36.859
The majority of the market kind of like, you know, most businesses running 3, 5, 7 nodes
451
00:46:37.240 --> 00:46:44.225
will benefit from our open source. You know? So that's the strategy. And then hopefully, towards the end of the year, we're gonna announce our cloud.
452
00:46:44.685 --> 00:46:50.145
And our cloud is, you know, effectively what every data vendor gives you. It's like you offload
453
00:46:50.970 --> 00:46:52.270
everything to the vendor.
454
00:46:52.570 --> 00:47:03.484
You know, they offer you a a mutual TLS authenticated port, and he goes pushed to this port and consumed from this port, and the management, they they kinda take advantage of that. That's what we're thinking in terms of business.
455
00:47:04.265 --> 00:47:13.200
In terms of the ways that you have seen some of your initial customers using Red Panda, what are some of the most most interesting or innovative or unexpected use cases that you've seen?
456
00:47:13.900 --> 00:47:15.200
We have 1
457
00:47:15.900 --> 00:47:16.400
incredibly
458
00:47:16.860 --> 00:47:21.535
intriguing use case. It's We've only heard of this 1 customer of ours,
459
00:47:22.155 --> 00:47:26.655
and I've never even thought about it. And I've been doing a streaming for 10 years.
460
00:47:27.000 --> 00:47:33.099
So they are a database company fundamentally. Right? They provide some services, but at its core, they're a database company.
461
00:47:33.400 --> 00:47:41.144
They're like a cloud native database company. And so they're actually thinking because of our tiered storage, the idea that we talk about shadow indexing,
462
00:47:41.765 --> 00:47:43.944
Kafka, in terms of offset managing,
463
00:47:44.440 --> 00:47:51.820
gives you a totally addressable log. Right? Like, from offset 0 all the way up to, like, you in 64 max,
464
00:47:52.325 --> 00:48:00.665
you can address every single record in a Kafka log. If you couple that with our shadow indexing idea, which means any cluster
465
00:48:01.200 --> 00:48:04.180
can read for any topic, can go and read
466
00:48:04.560 --> 00:48:12.285
any batch in the history of this, and we manage the data hierarchy and what we materialize on this versus what's uploaded onto
467
00:48:12.665 --> 00:48:13.165
durable
468
00:48:13.465 --> 00:48:17.485
storage like s 3, then what they're using us for is that they actually upload
469
00:48:18.140 --> 00:48:36.375
a megabyte of data for every record batch. And so every record is a megabyte of blob is, like, their own format. We don't really know what format it is. They just send a a megabyte. When they receive queries from their users, like, let's say somebody connects like a Tableau or something like that, and they're trying to run their data, their query processes,
470
00:48:36.970 --> 00:48:40.350
they actually fetch arbitrary offsets in the lineage
471
00:48:40.650 --> 00:48:47.204
of this totally ordered log to fetch BLOBs in a way that gives them hierarchical storage,
472
00:48:47.505 --> 00:49:09.605
that gives them low latency and high throughput for your streaming rights. So data that's hot is gonna come back hot. And data that's cold is still gonna come back relatively fast because we can fetch it from s 3, decode it really fast, and kinda like return the data to the user. So they get this totally addressable log with automatic data hierarchy management. That's probably the most technically
473
00:49:09.905 --> 00:49:11.365
interesting use case.
474
00:49:11.730 --> 00:49:14.390
And from a business use case is
475
00:49:14.770 --> 00:49:17.109
we are talking to a cancer lab,
476
00:49:17.410 --> 00:49:19.670
and they basically wanna do DNA sequencing
477
00:49:20.085 --> 00:49:24.105
because we can manage such large volumes of data and such a small amount of hardware.
478
00:49:24.485 --> 00:49:26.505
They want to retest cancer patients
479
00:49:26.965 --> 00:49:33.740
before they leave the wet lab. And so, you know, you spit on it or you're like, you know, you urinate in a cup and then they analyze the thing
480
00:49:34.599 --> 00:49:38.395
and they do some sequencing on it. Eventually, if they push that data to Red
481
00:49:38.875 --> 00:49:45.215
Panda and they run like SparkML on it, they can actually crunch through gigabytes of data per second.
482
00:49:45.650 --> 00:50:05.510
And you can just tell your patient, Hey, can you wait in the waiting room for the next 2 hours while I'm going through, you know, let's say like a terabyte of data, and then you have the results. And so then being able to test a patient before they leave is maybe the most interesting, maybe business use case. And so I think those are the maybe the 2 most most wild use case that we've seen with Red Panda.
483
00:50:06.050 --> 00:50:09.910
From talking to you and learning more about the system, Red Panda is definitely a very
484
00:50:10.605 --> 00:50:17.905
interesting and impressive piece of technology and something that I'm personally looking forward to experimenting with. But what are the cases where it's the wrong choice?
485
00:50:18.740 --> 00:50:20.920
Yeah. We don't support transactions
486
00:50:21.220 --> 00:50:21.720
yet.
487
00:50:22.020 --> 00:50:22.520
From
488
00:50:22.900 --> 00:50:30.755
customer's experience, like, kind of the cool thing about being in the data space is that it really doesn't matter what our claims are or what our
489
00:50:31.535 --> 00:50:34.914
competitors' claims are. Like, ultimately, the clients will tell you,
490
00:50:35.220 --> 00:50:38.440
you know, this is what worked, and and this is what didn't work. And
491
00:50:38.820 --> 00:50:45.285
to them, a lot of them have scalability limitations running the current implementation of the Kafka transactions.
492
00:50:45.825 --> 00:50:46.485
And so
493
00:50:47.185 --> 00:50:48.565
I would say it's probably
494
00:50:49.345 --> 00:50:54.325
less than 10% of the people that are running Kafka transactions. So we're not a good fit
495
00:50:54.770 --> 00:50:57.030
if you require Kafka transactions. Currently,
496
00:50:57.570 --> 00:50:58.630
we only solve
497
00:50:59.170 --> 00:51:01.190
the base API, which is deceptively
498
00:51:02.085 --> 00:51:05.465
simple to say. Remember that Kafka has over a 160 APIs
499
00:51:05.845 --> 00:51:07.625
with, like, versioning and so on.
500
00:51:07.925 --> 00:51:10.505
So we covered the majority of them, but we don't cover transactions.
501
00:51:11.210 --> 00:51:14.750
We are basically thinking through how do we improve
502
00:51:15.130 --> 00:51:17.390
the throughput of transactions. Right? So transactions
503
00:51:17.930 --> 00:51:25.855
are really not for low latency. They're for data safety. And what people care about is, like, they wanna push a bunch of transactions per second, but they're not trying to optimize
504
00:51:26.395 --> 00:51:33.299
the last 500 microseconds or the last 200 microseconds. That's not what transaction users are are using transactions for upstream.
505
00:51:33.760 --> 00:51:34.420
And so
506
00:51:34.799 --> 00:51:42.525
we did drive the draft for transactions. We didn't like the performance of it or, like, the design trade offs. So that's still very much in in active research,
507
00:51:43.065 --> 00:51:46.125
and it's probably going to be on, I would say,
508
00:51:46.500 --> 00:52:08.770
q 1 next year. You know, if you are using the Kafka transactions, then that's fundamental to how you're building your use case. Like, you know, we're not a good fit there. I think for the other type of systems, we're really I think the future of streaming with our co processors, with the Wazim engine, and because we also sort of give them the entire ecosystem. Right? We have in the schema registry. We have an HTTP proxy.
509
00:52:09.285 --> 00:52:15.065
We give you API compatibility. We give you these 2 things, which are the future of streaming, leveraging cloud native technologies.
510
00:52:15.525 --> 00:52:20.859
But if you're definitely in the transaction space, I think currently Kafka is probably your only option.
511
00:52:21.720 --> 00:52:25.260
Are there any other aspects of the overall streaming ecosystem
512
00:52:25.640 --> 00:52:38.290
or the Kafka space or the work that you're doing at Vectorized and Red Panda that we didn't discuss that you'd like to cover before we close out the show or any particular ways that the audience can engage with you to help you drive this product forward?
513
00:52:38.910 --> 00:52:45.625
Yeah. For sure. I really encourage people to try it. It's gonna be free to download, you know, free to run. There's gonna be no limits,
514
00:52:46.005 --> 00:53:01.825
you know, right now. So download, try it, you know, let us know what you think. We just love to to see how we can improve. That's really sort of the first thing. Ultimately, all of our features have ring. When you start a company, you kinda have this grand idea of how things are gonna work out.
515
00:53:02.204 --> 00:53:07.839
You know, for me, it was like, I wanna build the super fastest storage engine. I'm gonna solve all the problems.
516
00:53:08.140 --> 00:53:10.400
And people just wanted, you know, a better Kafka.
517
00:53:10.700 --> 00:53:18.575
Now we're building the future with co processors and stuff. I really think if you are, especially in the JavaScript community, in the Python community,
518
00:53:18.875 --> 00:53:20.894
and you have always been wanting to
519
00:53:21.400 --> 00:53:51.565
try and test out and and build real time streaming pipelines in a way that's easy for you to manage, which is, you know, it's a single binary. I think we really are calling for for this community to help us create a system that they wish they had for them. We think that the ability to upload these co processors, given these data API guarantees is gonna be super useful for effectively tensor flow shops, where you push data to a topic. They really are not either latency sensitive or throughput sensitive. They just want convenience.
520
00:53:51.944 --> 00:53:56.525
And so this 1 binary is, I think, addressing the need of running a simple system
521
00:53:56.830 --> 00:53:58.770
that gives them data API guarantees.
522
00:53:59.310 --> 00:54:03.890
So I think that's really sort of the future. And and, you know, later this year, we're gonna have
523
00:54:04.295 --> 00:54:07.595
also part of the product, a big part of the product, probably the majority,
524
00:54:08.055 --> 00:54:16.570
is going to be open source and available for people to hack. It is written in c plus plus 20, with, like, you know, coroutines and future. So
525
00:54:16.870 --> 00:54:23.255
probably not a lot of contributors. But if you are looking to hack around this, stay tuned. The best way to reach out to me,
526
00:54:23.555 --> 00:54:30.375
Twitter at vectorize io or my personal Twitter, which is called emaxrdna, which is the largest number in the Linux kernel.
527
00:54:30.740 --> 00:54:37.400
And, yeah, and I think, you know, feel free to reach out, and and we're happy to to start the conversation. 1 of the things that we're most excited about
528
00:54:37.700 --> 00:54:39.480
is I'm a Latino founder,
529
00:54:40.474 --> 00:54:42.015
and there aren't that many
530
00:54:42.555 --> 00:54:45.615
black, Latinos, and females working on hard distributed systems.
531
00:54:45.915 --> 00:54:47.375
We have created a scholarship
532
00:54:47.780 --> 00:54:50.120
called Hack the Planet, which we're super psyched.
533
00:54:50.500 --> 00:54:51.000
And
534
00:54:51.460 --> 00:54:53.080
I think if if there are any listeners
535
00:54:53.380 --> 00:54:55.395
that are underrepresented in tech,
536
00:54:55.795 --> 00:54:59.255
definitely reach out. The application process is open.
537
00:54:59.635 --> 00:55:02.935
And this came about, honestly, recently through all of the
538
00:55:03.289 --> 00:55:06.109
recent events, which are very sad. We wanted to do something.
539
00:55:06.410 --> 00:55:06.910
And
540
00:55:07.369 --> 00:55:20.095
I think that the best thing in addition to donating to the organizations that we do is really to donate our time. We have experts that have run some of the largest systems in the world at Microsoft, at Red Hat, at Akamai,
541
00:55:20.730 --> 00:55:25.150
and we can have a significant impact in your life if you're listening to this podcast.
542
00:55:25.450 --> 00:55:30.165
You get to keep the entire IP. You get to work on the project that you want. We don't
543
00:55:30.545 --> 00:55:46.470
want anything of the IP. We wanna give you money to hack on something that is hard, that is in distributed systems as long as you're part of this, and you get the mentoring. You get to talk to CEO of me once a month on anything, whether it's starting a business or running distributed systems. And you also get mentorship
544
00:55:47.005 --> 00:55:49.265
from all of our senior architects,
545
00:55:49.565 --> 00:55:51.265
Dennis, Noah, Meha,
546
00:55:51.724 --> 00:56:00.440
Ben. I mean, mostly everyone that I work with has been in this business for, like, you know, 10, 15, 20 years. We really hope that you take advantage of it. Definitely
547
00:56:00.820 --> 00:56:02.200
hit up the vectorize.iosolarship.
548
00:56:03.540 --> 00:56:08.145
We want to help you. Yeah. And we'd love to be the change that we want to see today.
549
00:56:08.685 --> 00:56:22.310
For anybody who does want to get in touch or reach out about any of those things, I'll have your contact information in the show notes. And as a final question, I would just like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
550
00:56:22.805 --> 00:56:28.105
It's a really fun question to always look at the future. You know, the biggest piece missing to me
551
00:56:28.405 --> 00:56:30.425
is tooling that gives
552
00:56:30.890 --> 00:56:31.789
deep guarantees
553
00:56:32.170 --> 00:56:40.795
about the data. We have an idea and we have an opinion, which is our co processors, but, ultimately, people will decide if that's useful. You know? I think that the way
554
00:56:41.095 --> 00:56:53.210
we advance is we ask some of our users, we propose a system, we build a system that we think is the future. There are other alternatives that I can think of, but I think tooling that gives deep introspective guarantees
555
00:56:53.830 --> 00:57:12.420
about the data. So 1 example is whether you're a database or streaming system, it ultimately doesn't matter. That's you what we have to move away is from kind of this low bids like vectorize. You know, that's for us, for very few people to really spend all of their time and expertise focusing on moving data around on the bids. But from a business perspective,
556
00:57:12.720 --> 00:57:14.560
what I think is missing is allowing
557
00:57:15.165 --> 00:57:22.865
you know, building system that give users higher guarantees. 1 example is if you put data on any system, that that system guarantees
558
00:57:23.165 --> 00:57:24.065
data provenance,
559
00:57:24.365 --> 00:57:45.720
which is the right people through the entire transformation of the data only have access to the data. Whether that system goes through Red Panda and moves through HBase and then back to and it doesn't matter what the pipeline is. But, like, that kind of data prominence, you know, just keeping all the permission in around all of the systems, something that manages that, I think, is missing.
560
00:57:46.020 --> 00:57:54.095
The second 1, I would say, is the ability to give users higher level guarantees. Like, as long as you put JSON in here, we'll make
561
00:57:54.555 --> 00:57:56.895
sure that no private information is leaked.
562
00:57:57.355 --> 00:58:03.880
But those things are guaranteed by this storage system. In a way, I really think that GDPR compliance
563
00:58:04.180 --> 00:58:10.885
is a contract that databases have to adhere to, or fundamentally, the storage system. Like, we have to be able to map some of this
564
00:58:11.445 --> 00:58:20.425
more complicated business metrics and goals that they're trying to achieve, which is, like, you know, protect the the users' data and so on, and map it onto the storage system themselves
565
00:58:21.190 --> 00:58:29.115
that own the actual data, you know, and be able to expose it to developers in a way that is easy to consume. I think that there's a lot of tooling that is missing there.
566
00:58:29.515 --> 00:58:35.615
Well, I appreciate you taking the time today to join me and discuss all the work that you've been doing with Red Panda and Vectorized
567
00:58:35.915 --> 00:58:48.505
and exploring and expanding the streaming space. It's definitely a very interesting product and 1 that I plan to take a closer look at myself. So I appreciate all the time and energy you've put into that, and hope you enjoy the rest of your day. Thanks, Tobias. Talk to you soon.
568
00:58:54.255 --> 00:58:56.970
Listening. Don't forget to check out our other show, podcast.init@pythonpodcast.com
569
00:58:59.509 --> 00:59:03.930
to learn about the Python language, its community, and the innovative ways it is being used.
570
00:59:04.305 --> 00:59:06.405
And visit the site of data engineering podcast.com
571
00:59:06.945 --> 00:59:16.420
to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show, then tell us about it. Email hosts at data engineering podcast.com
572
00:59:16.960 --> 00:59:22.260
with your story. And to help other people find the show, please leave a review on Itunes and tell your friends and coworkers.