Interactive Exploratory Data Analysis On Petabyte Scale Data Sets With Arkouda

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 10:55:48 AM
Duration: 2437.144
Channels: 1
1
00:00:14.195 --> 00:00:18.350
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:18.890 --> 00:00:25.789
When you're ready to build your next pipeline or want to test out the projects you hear about on the show, you'll need somewhere to deploy it. So check out our friends at Linode.
3
00:00:26.155 --> 00:00:29.855
With their new managed database service, you can launch a production ready MySQL,
4
00:00:30.315 --> 00:00:37.400
Postgres, or MongoDB cluster in minutes with automated backups, 40 gigabit connections from your application hosts, and high throughput SSDs.
5
00:00:38.340 --> 00:00:39.080
Go to dataengineeringpodcast.com/linode
6
00:00:41.140 --> 00:00:50.745
today and get a $100 credit to launch a database, create a Kubernetes cluster, or take advantage of all of their other services. And don't forget to thank them for their continued support of this show.
7
00:00:51.570 --> 00:00:53.830
Data stacks are becoming more and more complex.
8
00:00:54.210 --> 00:00:59.405
This brings infinite possibilities for data pipelines to break and a host of other issues, severely deteriorating
9
00:00:59.785 --> 00:01:02.605
the quality of the data and causing teams to lose trust.
10
00:01:03.065 --> 00:01:10.880
Siflae solves this problem by acting as an overseeing layer to the data stack, observing data and ensuring it's reliable from ingestion all the way to consumption.
11
00:01:11.500 --> 00:01:21.234
Whether the data is in transit or at rest, Ciflae can detect data quality anomalies, assess business impact, identify the root cause, and alert data teams on their preferred channels,
12
00:01:21.534 --> 00:01:45.160
all thanks to over 50 quality checks, extensive column level lineage, and over 20 connectors across the data stack. In addition, data discovery is made easy through Siflae's information rich data catalog with a powerful search engine and real time health status. Listeners of the podcast will get $2,000 to use as platform credits when signing up to use Siflae. Siflae also offers a 2 week free trial. Find out more at data engineering podcast.com/ciflae
13
00:01:46.500 --> 00:01:55.695
today. That's s I f f l e t. Your host is Tobias Macy. And today, I'm interviewing David Bader about Arcuda, a horizontally
14
00:01:56.050 --> 00:02:02.230
scalable parallel compute library for exploratory data analysis in Python. So, David, can you start by introducing yourself?
15
00:02:02.770 --> 00:02:03.270
Sure.
16
00:02:03.765 --> 00:02:07.945
It's nice to see you. My name is David Bader. I'm a distinguished professor
17
00:02:08.325 --> 00:02:16.260
and director of the Institute For Data Science at the New Jersey Institute of Technology, where I've also founded the Department of Data Science
18
00:02:16.960 --> 00:02:21.425
this past fall that has degree offerings in data science.
19
00:02:21.825 --> 00:02:24.885
And do you remember how you first got started working in the area of data?
20
00:02:25.425 --> 00:02:26.485
My work with data
21
00:02:26.945 --> 00:02:30.885
actually goes back decades upon decades. So I've always been fascinated
22
00:02:31.330 --> 00:02:32.550
by graph analytics
23
00:02:33.330 --> 00:02:33.830
and
24
00:02:34.209 --> 00:02:34.709
understanding
25
00:02:35.170 --> 00:02:36.310
from large datasets
26
00:02:37.330 --> 00:02:42.545
going back literally to the 19 eighties. So this has been a passion of mine.
27
00:02:43.165 --> 00:02:54.440
So in terms of the Arcuda project, I'm wondering if you can share a bit about what it is and some of the story behind how it came to be and why you decided to build this tool and some of the problems that it's aimed at solving.
28
00:02:55.060 --> 00:02:56.920
Sure. Great question. Arcuda,
29
00:02:57.245 --> 00:02:59.504
which is the Greek word for bear, it's spelled
30
00:03:00.284 --> 00:03:00.784
arkouda,
31
00:03:02.765 --> 00:03:04.144
is an open source
32
00:03:04.525 --> 00:03:05.584
framework for
33
00:03:05.885 --> 00:03:06.625
big data,
34
00:03:07.230 --> 00:03:10.530
and it's available from GitHub, so anyone can check it out.
35
00:03:10.989 --> 00:03:13.170
We noticed the tension
36
00:03:13.470 --> 00:03:15.489
in data science where we have productivity
37
00:03:16.175 --> 00:03:17.474
languages like Python,
38
00:03:17.935 --> 00:03:18.915
where on
39
00:03:19.215 --> 00:03:20.515
a desktop or laptop,
40
00:03:21.295 --> 00:03:26.480
many programmers are able to write codes when the data fits on their laptop. Very easy
41
00:03:26.780 --> 00:03:30.960
to learn all of these tools like NumPy and Pandas and
42
00:03:31.265 --> 00:03:31.765
NetworkX
43
00:03:32.144 --> 00:03:36.644
and all of these great tools. But then when you have a large dataset,
44
00:03:37.105 --> 00:03:39.205
and by large, when your datasets
45
00:03:39.940 --> 00:03:44.680
overwhelm what you can fit onto your laptop or desktop, so those datasets
46
00:03:44.980 --> 00:03:49.075
tend to be multi terabyte in size, the number of tools quickly
47
00:03:49.375 --> 00:03:51.875
falls off, and you need to use supercomputers
48
00:03:52.815 --> 00:03:55.875
where I have quite an extensive background.
49
00:03:56.175 --> 00:03:59.380
And what we thought about was how do we make supercomputing
50
00:03:59.840 --> 00:04:02.020
for large data as productive
51
00:04:02.720 --> 00:04:04.260
as using Python?
52
00:04:04.855 --> 00:04:06.555
How do we combine that productivity
53
00:04:07.334 --> 00:04:08.155
with performance?
54
00:04:08.775 --> 00:04:11.754
So Arcuda was born about 2 years ago.
55
00:04:12.130 --> 00:04:16.150
And, again, as an open source project where the end user
56
00:04:16.850 --> 00:04:19.750
can write their analytics in a Jupyter notebook
57
00:04:20.205 --> 00:04:22.465
with constructs that look very similar to NumPy.
58
00:04:22.765 --> 00:04:23.585
But in reality,
59
00:04:24.125 --> 00:04:29.880
all of their secret sauce and development kicks in and they're actually using a back end supercomputer
60
00:04:30.500 --> 00:04:35.800
with an open source compiler called Chapel that is doing a lot of the heavy lifting.
61
00:04:36.525 --> 00:04:37.905
In terms of the
62
00:04:38.205 --> 00:04:46.230
core audience and the focus of what you're trying to solve for them, I'm curious if you can talk to how you thought about that formulation
63
00:04:46.610 --> 00:04:48.070
as you started building Arcuda.
64
00:04:48.690 --> 00:04:53.590
What we realized was that Python is really the lingua de franca of data scientists.
65
00:04:54.215 --> 00:04:56.795
Everybody is able to work in Python.
66
00:04:57.335 --> 00:05:00.955
And we wanted to make big data analytics accessible
67
00:05:01.415 --> 00:05:04.170
to those that know how to use Python
68
00:05:04.790 --> 00:05:08.170
and to put their workflows in Jupyter Notebooks, for instance.
69
00:05:08.870 --> 00:05:12.010
This is really an area that is affecting
70
00:05:12.625 --> 00:05:13.365
all enterprises,
71
00:05:14.145 --> 00:05:15.045
all organizations,
72
00:05:15.825 --> 00:05:18.645
because we know data just keeps increasing in size.
73
00:05:19.105 --> 00:05:19.605
And
74
00:05:19.930 --> 00:05:25.550
when we put datasets together, for instance, we often get datasets that are larger than
75
00:05:25.930 --> 00:05:28.590
the space on our workstations.
76
00:05:29.565 --> 00:05:33.345
They tend to be multi terabytes and more tens of terabytes.
77
00:05:33.725 --> 00:05:35.505
So we're really addressing a need
78
00:05:35.805 --> 00:05:39.759
that the enterprise has for being able to rapidly
79
00:05:40.300 --> 00:05:41.840
and productively do analytics
80
00:05:42.220 --> 00:05:44.960
once those datasets are just so large.
81
00:05:45.384 --> 00:05:47.005
And we wanna have interactivity.
82
00:05:47.465 --> 00:05:49.324
So it isn't like supercomputing
83
00:05:49.625 --> 00:05:59.550
of decades ago where you come to it with a problem and it crunches on it for a while and gives you an answer back the next day. We want to have near time responsiveness,
84
00:05:59.850 --> 00:06:07.645
just like our Jupyter Notebook, where we hit return and we get the answer, even if our data set is, say, 25 terabytes in size.
85
00:06:08.345 --> 00:06:11.245
In terms of the exploratory data analysis
86
00:06:11.860 --> 00:06:13.480
aspect of the data life cycle,
87
00:06:13.860 --> 00:06:25.585
it's generally the domain of the data scientist. And when they do start to hit those points where their data doesn't fit on their laptop and they need to actually start turning to things like Arcuda and parallel compute,
88
00:06:26.365 --> 00:06:32.610
how does that generally manifest in terms of who they're asking for help with being able to solve that problem,
89
00:06:32.990 --> 00:06:44.625
how they start to think about approaching that problem, and just how that propagates into the broader team and the broader organization as far as the impact on their ability to be productive and who they're leaning on to help them solve those problems?
90
00:06:45.130 --> 00:06:51.150
Well, as you mentioned, once you get stuck at that point where your data grows larger than than your resources,
91
00:06:51.450 --> 00:07:00.945
you hit a speed bump. You're looking to ask experts, how do I solve this? What tools are out there? How do I get access to resources? And what we're trying to do is democratize
92
00:07:01.590 --> 00:07:02.090
data
93
00:07:02.630 --> 00:07:07.530
science so that anyone who can program in Python will be able to use Arcuda
94
00:07:08.070 --> 00:07:14.705
and escape that barrier without the need for finding a parallel computing expert, such as myself,
95
00:07:15.085 --> 00:07:15.825
that will
96
00:07:16.285 --> 00:07:18.705
be able to show them how to do some
97
00:07:19.220 --> 00:07:20.520
programming for a supercomputer.
98
00:07:21.060 --> 00:07:33.535
So we're trying to make it turnkey so that anyone can just turn the the knob when their datasets gets larger, be able to modify their code so that they can still do exploratory data analysis,
99
00:07:33.915 --> 00:07:35.295
meaning looking at datasets
100
00:07:35.675 --> 00:07:36.815
in the real time,
101
00:07:37.360 --> 00:07:45.220
exploring what if questions on their data, and being able to use tools that seamlessly let them scale from their desktop to a supercomputer.
102
00:07:46.175 --> 00:07:56.470
And at the point where they stop being able to manage that interactivity, it's another aspect of the speed bump of, you know, they do get to the point where they have to submit their batch job to a supercompute cluster
103
00:07:57.250 --> 00:08:15.870
or in past decade, you know, put a MapReduce job into a Hadoop cluster to be able to figure out what comes out the other side. What are some of the ways that that impacts their ability to be productive and some of the problematic behaviors that that might encourage or lead to if they do have these time delays of being able to ask and answer questions?
104
00:08:16.490 --> 00:08:23.310
Often, we want to be productive, meaning that we want to be able to ask questions of our data and get the answers
105
00:08:23.634 --> 00:08:27.815
in the same time that we're thinking about it, so that we can explore these datasets.
106
00:08:28.595 --> 00:08:32.615
And if that turnaround time or those transactions take
107
00:08:33.120 --> 00:08:47.894
minutes, hours, days, then we lose that train of thought. And so it's very important for a data scientist to be able to operate in near real time. So as they post questions, they see the answer right away that can steer them towards what's the next question to ask. So
108
00:08:48.435 --> 00:08:50.055
I should mention with Arcuda,
109
00:08:50.435 --> 00:08:55.480
we are really focused on the analyst, the end user, and giving them a new capability
110
00:08:56.020 --> 00:09:00.200
that is very similar to how they've used Python and NumPy,
111
00:09:00.745 --> 00:09:02.765
pandas, and other such constructs.
112
00:09:03.145 --> 00:09:05.165
But with just a slight modification,
113
00:09:05.785 --> 00:09:08.365
our kudo will drop in to replace NumPy
114
00:09:09.030 --> 00:09:11.530
and give you a incredible capability
115
00:09:11.990 --> 00:09:16.570
for basic data science constructs. But also, we've been building out a rich
116
00:09:16.895 --> 00:09:19.235
set of libraries on graph analytics.
117
00:09:19.855 --> 00:09:32.560
I'm very proud that 1 of their expertise in my lab is graph analytics at a large scale. And in fact, last week, we had a brand new book come out that I edited on massive graph analytics.
118
00:09:33.065 --> 00:09:36.845
So I think this is 1 specialization of data science that is also
119
00:09:37.305 --> 00:09:39.005
really interesting, and Arcuda
120
00:09:39.305 --> 00:09:41.805
powers our graph analytics as well.
121
00:09:42.110 --> 00:10:00.079
That brings up an interesting point about the types of analysis and the types of data that you're working with and how you think about the capabilities to work into Arcuda because there are definitely a number of other projects out there for being able to scale, compute, and scale data access beyond the bounds of a single computer.
122
00:10:00.380 --> 00:10:06.160
I'm thinking in terms of projects like Ray or Dask. There's also another 1, I think, called Bodo.
123
00:10:06.715 --> 00:10:08.815
And I'm curious, what was
124
00:10:09.595 --> 00:10:15.615
missing in those solutions that made something like Arcuda necessary and some of the ways that you think about the
125
00:10:16.010 --> 00:10:23.390
specific problem sets that Arcuda is well suited to and when you might actually want to lean on some of those other frameworks for different problems?
126
00:10:24.045 --> 00:10:31.024
Great question. So we were faced with trying to solve some real world brand challenges. For instance, in cybersecurity,
127
00:10:31.769 --> 00:10:34.589
where often we're collecting up information about
128
00:10:35.050 --> 00:10:43.885
network traffic, and we're trying to identify cyber threats and give attribution to those threats. And there, we have to operate in near real time
129
00:10:44.425 --> 00:10:46.605
and working with analysts to
130
00:10:46.985 --> 00:10:51.569
formulate their questions and problems who are trained in basic data science
131
00:10:51.949 --> 00:10:55.810
using Python and so forth. And these datasets are humongous.
132
00:10:56.350 --> 00:10:56.850
Often
133
00:10:57.235 --> 00:10:59.575
in a large enterprise, we're looking at datasets
134
00:10:59.955 --> 00:11:04.455
that could be tens of terabytes in size. So we wanted something very seamlessly.
135
00:11:05.060 --> 00:11:13.480
Our analyst could just pick up a new tool, take the existing code, and with just slight modifications, be able to scale beyond,
136
00:11:13.975 --> 00:11:15.435
say, tens of gigabytes
137
00:11:16.774 --> 00:11:20.875
to tens of terabytes and beyond. So that was really the
138
00:11:21.230 --> 00:11:29.089
goal for CUDA was to make this easy and productive so that analysts don't need to learn a new framework, a new language,
139
00:11:29.855 --> 00:11:31.074
and all of the
140
00:11:31.375 --> 00:11:34.834
challenges that come along with trying to figure out a new environment.
141
00:11:35.454 --> 00:11:37.154
And, again, 1 that scales
142
00:11:38.500 --> 00:11:39.160
for this
143
00:11:39.540 --> 00:11:46.519
extraordinary size of datasets that I think we're gonna see more and more as we collect more data and put datasets together.
144
00:11:47.155 --> 00:11:51.815
And so in terms of Arcuda itself, can you talk a bit about some of the implementation
145
00:11:52.195 --> 00:11:57.480
and the ways that you thought about approaching this problem and maybe some of the unique
146
00:11:57.860 --> 00:12:01.480
algorithms and capabilities that you've baked into it to be able to
147
00:12:01.940 --> 00:12:07.084
power these interactive use cases on larger than single compute datasets?
148
00:12:07.865 --> 00:12:11.485
Arcuda was started by a team at the US Department of Defense,
149
00:12:12.185 --> 00:12:14.045
and we have been contributing
150
00:12:14.420 --> 00:12:19.320
to Arcuda since its start. So I should first mention that there's a team behind Arcuda,
151
00:12:19.860 --> 00:12:23.720
and I've been responsible for building out the graph analytics,
152
00:12:24.084 --> 00:12:26.265
along with my very capable researchers
153
00:12:26.644 --> 00:12:28.584
at the New Jersey Institute of Technology,
154
00:12:28.885 --> 00:12:31.545
where we've been focusing on adding the capability
155
00:12:32.010 --> 00:12:34.510
to look at data as a graph with relationships
156
00:12:34.970 --> 00:12:37.230
and to build in capabilities for solving
157
00:12:38.010 --> 00:12:39.870
standard graph analytic questions.
158
00:12:40.255 --> 00:12:43.475
For instance, are there communities within the dataset?
159
00:12:44.255 --> 00:12:47.315
Are there paths between particular vertices?
160
00:12:48.380 --> 00:12:52.000
And other sorts of features in the graph to understand
161
00:12:52.460 --> 00:12:53.279
and explore
162
00:12:53.580 --> 00:12:55.440
the dataset as a graph.
163
00:12:55.845 --> 00:13:00.185
Let me take a step back. And when I think of data as a graph, it's really
164
00:13:00.885 --> 00:13:03.465
looking at data through a lens
165
00:13:03.779 --> 00:13:09.160
where we have objects in our data, which we think of as vertices in the graph. And when these objects
166
00:13:09.779 --> 00:13:15.995
interact with each other, we have an edge induced inside of the graph. So there are many problems
167
00:13:16.695 --> 00:13:20.770
that we can represent as these types of graphs, whether it's in cybersecurity,
168
00:13:21.550 --> 00:13:23.010
social media analysis,
169
00:13:23.550 --> 00:13:24.610
personalized health,
170
00:13:25.310 --> 00:13:27.090
and more, we can
171
00:13:27.595 --> 00:13:30.335
move these problems into this graph abstraction
172
00:13:30.715 --> 00:13:33.535
and then solve it with some very powerful graph analytics.
173
00:13:34.370 --> 00:13:39.269
In that graph analytics space, I know that there are a number of interesting capabilities,
174
00:13:39.730 --> 00:13:50.715
as well as an associated set of interesting challenges that factor into it. 1 of the things that I know is most common that people encounter is the question of supernodes and how to handle those,
175
00:13:51.095 --> 00:13:51.710
and then the question
176
00:13:59.790 --> 00:14:07.785
algorithms some of those problems in Arcuda and maybe some of the algorithmic aspects that you've had to develop to be able to support those use cases.
177
00:14:08.350 --> 00:14:15.010
Arcuda isn't the only open source framework that I've developed for graphs. Over the years, I've developed many frameworks,
178
00:14:15.505 --> 00:14:25.790
or I should say I've developed several frameworks for graphs. And there are some specialized frameworks for streaming graphs, and that's graphs where we get edges
179
00:14:26.170 --> 00:14:28.110
from a fire hose, and we wanna
180
00:14:28.649 --> 00:14:29.149
ask
181
00:14:29.450 --> 00:14:38.935
questions to that graph as it changes over time. So we were 1 of the earliest groups to build ACE driven graph analytic framework that we called Stinger.
182
00:14:39.475 --> 00:14:42.455
And in many of these graph analytics, we're faced with
183
00:14:42.980 --> 00:14:48.280
challenges when we find what you called supernodes. Sometimes we call them high degree vertices.
184
00:14:49.060 --> 00:14:51.320
And these often are
185
00:14:52.035 --> 00:14:58.535
problematic if we're trying to partition a graph or we're looking at information flow, and these nodes may bias
186
00:14:59.110 --> 00:15:05.210
the results that we see or really make it a challenge. So often when we're running a graph analytic,
187
00:15:05.670 --> 00:15:06.970
we can threshold
188
00:15:07.585 --> 00:15:09.205
For instance, we're looking at vertices
189
00:15:09.585 --> 00:15:11.045
above a certain degree
190
00:15:11.345 --> 00:15:13.525
and below a certain degree.
191
00:15:14.065 --> 00:15:16.805
And in this way, we can find other
192
00:15:17.730 --> 00:15:19.029
methods for handling
193
00:15:19.410 --> 00:15:27.705
these, what you call supernode vertices in the graph. Let me just give you an example to make it more concrete. We have a social network,
194
00:15:28.085 --> 00:15:33.145
and the degree of a person in the social network is the number of friends that they have.
195
00:15:33.620 --> 00:15:35.640
And I may be trying to analyze
196
00:15:36.020 --> 00:15:36.680
how many
197
00:15:37.300 --> 00:15:41.480
friends are between me and you on the shortest path between friends.
198
00:15:41.945 --> 00:15:44.205
But if we're both friended with, say,
199
00:15:46.105 --> 00:15:46.765
a superstar,
200
00:15:47.785 --> 00:15:52.600
then it doesn't really make sense to count our connection through that superstar.
201
00:15:53.220 --> 00:16:12.980
We want to find connections just through our ordinary friends to connect to me and you. And so often, we'll threshold out when there's a friend and they have a 1000000000 friends out there. Well, it isn't as interesting. And our algorithms let us filter off those vertices to try to answer these questions.
202
00:16:13.519 --> 00:16:21.395
The inverse of the 6 degrees of Kevin Bacon. That's right. In fact, my students were very interested to see in the IMDB, the movie
203
00:16:21.775 --> 00:16:23.155
database on the Internet,
204
00:16:23.920 --> 00:16:26.019
was Kevin Bacon really center
205
00:16:26.720 --> 00:16:28.339
I should say central to
206
00:16:28.959 --> 00:16:32.579
the actors of all movies? And so we actually did that analysis.
207
00:16:33.005 --> 00:16:35.265
Turned out he was just a common actor,
208
00:16:35.644 --> 00:16:37.985
unfortunately, but a great story nevertheless.
209
00:16:38.925 --> 00:16:52.285
As far as the Arcuda project itself, you mentioned it started off as a project in the Department of Defense. I'm curious if you can talk to some of the ways that it has evolved and grown in terms of the goals and scope of the project
210
00:16:52.745 --> 00:16:59.040
and any changes in the implementation details of how you've approached the problems that you're trying to solve with it. Arcuda
211
00:16:59.340 --> 00:17:02.720
is open source. So on GitHub, if you go to the
212
00:17:03.260 --> 00:17:03.760
repositories
213
00:17:04.140 --> 00:17:08.240
under the group bears r us, so bears hyphen r
214
00:17:08.605 --> 00:17:09.105
hyphen
215
00:17:09.645 --> 00:17:11.184
us, and look under Arcuda,
216
00:17:11.485 --> 00:17:12.625
you'll see talks
217
00:17:12.925 --> 00:17:15.595
and all of the source code in a repository
218
00:17:16.360 --> 00:17:21.660
and a lot of the discussion spaces and papers related to Arcuda. So it's a completely
219
00:17:21.960 --> 00:17:22.460
open
220
00:17:23.080 --> 00:17:24.860
and widely shared project.
221
00:17:25.774 --> 00:17:29.154
The goals were really to make a new framework that
222
00:17:29.534 --> 00:17:32.274
everyone around the world can use to do
223
00:17:32.710 --> 00:17:35.130
productive large scale data analysis.
224
00:17:35.830 --> 00:17:41.289
I'm quite excited by it because it's starting out as an open source project, and we'd love to contribute.
225
00:17:41.945 --> 00:17:43.485
And we have collaborators
226
00:17:44.105 --> 00:17:48.284
at this point around the world who are helping to make Arcuda a reality.
227
00:17:48.664 --> 00:17:50.284
It's getting their maturity.
228
00:17:51.030 --> 00:17:52.890
The project started in 2019,
229
00:17:53.350 --> 00:17:57.610
and it's undergone a number of changes. For instance, there's a
230
00:17:58.184 --> 00:17:58.845
sub repository
231
00:17:59.145 --> 00:18:01.804
for all of our graph analytic contributions
232
00:18:02.425 --> 00:18:07.645
and ways to create new modules to build on top of the core Arcuda framework.
233
00:18:08.330 --> 00:18:10.830
Arcuda operates with a user
234
00:18:11.130 --> 00:18:15.230
on the front end interfacing through their Jupyter notebook or with Python,
235
00:18:15.745 --> 00:18:24.085
and they connect to a 0 m queue, a message queue, going to a supercomputer in the back end running the HPE Cray
236
00:18:24.790 --> 00:18:27.050
Chapel compiler that's also open source.
237
00:18:27.510 --> 00:18:32.170
And so we, as developers, are developing all of the plumbing to go from
238
00:18:32.655 --> 00:18:38.595
Python in the front end, all the way through the supercomputer in the back. So that analysts who wish to use Arcuda,
239
00:18:38.895 --> 00:18:41.635
they don't see all of that complexity. They just see a productive
240
00:18:42.230 --> 00:18:44.730
environment where they can scale up their datasets
241
00:18:45.270 --> 00:18:46.570
to tens of terabytes.
242
00:18:49.495 --> 00:18:55.675
Data teams are increasingly under pressure to deliver. According to a recent survey by Ascend. Io, 95%
243
00:18:56.135 --> 00:18:57.675
reported being at or overcapacity,
244
00:18:58.419 --> 00:19:06.120
With 72% of data experts reporting demands on their team going up faster than they can hire, it's no surprise they are increasingly turning to automation.
245
00:19:06.544 --> 00:19:08.085
In fact, while only 3.5%
246
00:19:08.544 --> 00:19:10.725
report having current investments in automation,
247
00:19:11.025 --> 00:19:22.600
85% of data teams plan on investing in automation in the next 12 months. That's where our friends at Ascend dot io come in. The Ascend Data Automation Cloud provides a unified platform for data ingestion, transformation,
248
00:19:22.900 --> 00:19:24.200
orchestration, and observability.
249
00:19:24.965 --> 00:19:30.825
Ascend users love its declarative pipelines, powerful SDK, elegant UI, and extensible plug in architecture,
250
00:19:31.205 --> 00:19:41.910
as well as its support for Python, SQL, Scala, and Java. Ascend automates workloads on Snowflake, Databricks, BigQuery, and open source Spark and can be deployed in AWS, Azure, or GCP.
251
00:19:42.370 --> 00:19:43.995
Go to data engineering podcast.com/ascend
252
00:19:45.255 --> 00:19:50.155
and sign up for a free trial. If you're a data engineering podcast listener, you get credits worth $5,000
253
00:19:50.615 --> 00:19:51.755
when you become a customer.
254
00:19:53.560 --> 00:19:54.460
Given that
255
00:19:54.840 --> 00:20:03.555
it is relying on this super compute framework, I'm wondering how that maps to the actual hardware requirements that are necessary to be able to run Arcuda.
256
00:20:04.095 --> 00:20:07.155
So Chappell should not be a barrier because
257
00:20:07.535 --> 00:20:12.760
it's open source, and it runs on most available compute platforms from desktops
258
00:20:13.380 --> 00:20:16.120
to clusters to dedicated supercomputers.
259
00:20:17.105 --> 00:20:17.605
And
260
00:20:17.985 --> 00:20:20.405
you can acquire your own resources,
261
00:20:20.785 --> 00:20:22.005
where to run Chappell,
262
00:20:22.465 --> 00:20:27.020
where to load your datasets, and then manipulate them through the Sarcuda front end.
263
00:20:27.560 --> 00:20:32.460
Given the fact that there has been so much evolution in hardware capabilities
264
00:20:32.924 --> 00:20:35.505
and the ways that hardware is consumed,
265
00:20:35.965 --> 00:20:42.799
you know, with things like the cloud, but also with the evolution of different CPU architectures and the rise of ARM.
266
00:20:43.260 --> 00:20:45.279
I'm wondering how that evolution
267
00:20:45.659 --> 00:20:47.279
has impacted the
268
00:20:47.804 --> 00:20:58.139
types of processing algorithms and the appetite for space time trade offs and how you approach those algorithmic aspects of being able to work on these data processing
269
00:20:58.519 --> 00:20:59.980
problems at these scales?
270
00:21:00.679 --> 00:21:01.419
Great question.
271
00:21:01.879 --> 00:21:09.565
The Chappell compiler originally came out of a US DARPA, that's the Defense Advanced Research Projects Agency
272
00:21:10.105 --> 00:21:10.605
project
273
00:21:11.225 --> 00:21:13.565
about 20 years ago called High Productivity
274
00:21:13.865 --> 00:21:14.845
Computing Systems.
275
00:21:15.350 --> 00:21:18.490
And the compiler with 20 years of work in it now
276
00:21:18.870 --> 00:21:25.005
was built to be able to take advantage of different processor generations and to do a lot of the transformations.
277
00:21:25.865 --> 00:21:29.245
So the performance engineering is built into the compiler.
278
00:21:29.705 --> 00:21:31.565
Originally, Cray, the supercomputing
279
00:21:31.865 --> 00:21:36.550
company built this compiler, and then recently, HPE acquired
280
00:21:36.850 --> 00:21:40.870
the supercomputing company Cray. And HPE has a great,
281
00:21:41.495 --> 00:21:42.075
fantastic team,
282
00:21:42.935 --> 00:21:44.635
project manager and developers,
283
00:21:44.935 --> 00:21:48.395
who've been working with Chapel for quite a number of years
284
00:21:49.139 --> 00:21:51.159
and put a lot of sophistication
285
00:21:51.860 --> 00:21:56.759
into that compiler to be able to leverage the new processor architectures.
286
00:21:57.654 --> 00:22:04.315
So much of that comes with Chappell, and that's 1 of the reasons why we decided to use Chappell as the back end compiler
287
00:22:04.695 --> 00:22:06.075
for the Arcuda framework.
288
00:22:06.720 --> 00:22:08.340
As far as the
289
00:22:09.040 --> 00:22:10.740
adoption of Arcuda
290
00:22:11.040 --> 00:22:15.764
and how it factors into the development and analysis capabilities
291
00:22:16.065 --> 00:22:24.160
of a team or an organization. I'm wondering if you can talk to some of the process of getting it set up and the end user workflow.
292
00:22:24.940 --> 00:22:27.280
Arcuda, again, is an early project.
293
00:22:27.740 --> 00:22:30.975
It's open source, so every organization can look at the source
294
00:22:31.934 --> 00:22:34.434
code and be able to import it quite easily.
295
00:22:34.975 --> 00:22:38.515
We've been working in my lab on a tutorial for
296
00:22:38.830 --> 00:22:40.929
getting users started with Arcuda.
297
00:22:41.230 --> 00:22:44.049
I teach on it with students who are able to use it,
298
00:22:44.429 --> 00:22:45.490
and we
299
00:22:45.845 --> 00:22:47.225
have tutorials available
300
00:22:47.765 --> 00:22:52.424
for Arcuda. So anyone who's interested, I suggest they head over to the GitHub
301
00:22:52.920 --> 00:22:54.060
website for Arcuda
302
00:22:54.600 --> 00:22:58.780
and explore the resources that we have there. It's pretty simple to install,
303
00:22:59.240 --> 00:23:03.545
and it works well. So I encourage everyone here to try it out.
304
00:23:03.925 --> 00:23:04.825
As far as
305
00:23:05.605 --> 00:23:10.425
the collaboration process around being able to build these analyses with Arcuda,
306
00:23:11.250 --> 00:23:12.070
I'm wondering
307
00:23:12.450 --> 00:23:14.789
if there are any programming patterns or
308
00:23:15.090 --> 00:23:21.355
approaches to how to structure the analyses that helps to make it so that multiple people can be able to
309
00:23:21.655 --> 00:23:29.919
take advantage of maybe intermediate result sets that are generated by each other and just some of the aspects of being able to use this in a team environment?
310
00:23:30.539 --> 00:23:32.559
That's a great question. And, in fact,
311
00:23:32.940 --> 00:23:36.480
the large datasets are stored in the back end where Chappell
312
00:23:37.134 --> 00:23:38.514
has access to them.
313
00:23:38.894 --> 00:23:40.674
And there are constructs
314
00:23:41.134 --> 00:23:45.875
to be able to keep intermediate results in the back end for others to
315
00:23:46.580 --> 00:24:01.434
collaborate on and make use of rather than to have to bring results to a front end to share. Because for large data, we want to keep it in place. We don't want to move it that often, because that takes quite a lot of time and a lot of resources
316
00:24:02.215 --> 00:24:06.780
and a lot of energy. So So we're trying to be very energy efficient as well.
317
00:24:07.240 --> 00:24:08.540
So Arcuda does
318
00:24:08.840 --> 00:24:18.995
include those capabilities that a team that may be looking together at a dataset or derive products can do so quite easily by sharing resource locators
319
00:24:19.535 --> 00:24:20.595
for those datasets.
320
00:24:21.530 --> 00:24:30.105
In terms of the work that you've been doing at Arcuda, as you mentioned, it's a team effort. It's open source, so anybody can contribute to it. But given your expertise
321
00:24:30.725 --> 00:24:32.745
in super compute capabilities
322
00:24:33.125 --> 00:24:36.345
and graph analytics, I'm curious what are some of the
323
00:24:36.690 --> 00:24:50.775
specific contributions that you've been focused on and some of the specific challenges that you've been addressing in maybe some of the sharp edges that people experience when building on top of Arcuda or some of the new capabilities that you're hoping to unlock in this framework?
324
00:24:51.290 --> 00:24:57.710
We've been building out graph analytics that, as I mentioned, are quite important for solving real world grand challenges.
325
00:24:58.345 --> 00:24:59.805
And often graph analytics
326
00:25:00.185 --> 00:25:10.270
look easy on paper, but when you go to implement them, you run into performance issues. For instance, the high degree vertices that were mentioned before could really slow down
327
00:25:10.650 --> 00:25:13.710
a graph analytic. So what we're doing is
328
00:25:14.034 --> 00:25:15.975
analyzing the performance of
329
00:25:16.275 --> 00:25:20.695
algorithms on a wide variety of inputs to try to make sure that
330
00:25:21.049 --> 00:25:24.429
we have the capability to solve many instances
331
00:25:24.809 --> 00:25:31.674
quite fast using the graph analytics that we built into our CUDA. And for example, we're implementing new algorithms.
332
00:25:32.375 --> 00:25:35.915
For instance, 1 algorithm is called triangle centrality
333
00:25:36.455 --> 00:25:38.235
that a colleague of mine
334
00:25:38.760 --> 00:25:39.260
invented.
335
00:25:39.799 --> 00:25:44.299
And it's a centrality that's based on looking at the importance of vertices
336
00:25:44.679 --> 00:25:46.460
based on how many triangles
337
00:25:47.015 --> 00:25:51.915
and the distribution of triangles around them. This analytic, I think, is quite interesting
338
00:25:52.535 --> 00:25:54.155
and is a peer
339
00:25:54.520 --> 00:25:58.300
analytic to other centrality measures like betweenness centrality,
340
00:25:58.840 --> 00:25:59.820
closeness centrality,
341
00:26:00.200 --> 00:26:01.100
degree centrality,
342
00:26:01.400 --> 00:26:07.174
and others. So we're always looking for new, highly capable analytics that will provide new functionality.
343
00:26:07.634 --> 00:26:10.580
And then how do we implement those with high efficiency
344
00:26:11.200 --> 00:26:14.820
and also productively for the end user in the Arcuda framework.
345
00:26:15.360 --> 00:26:18.820
In terms of the applications of this framework,
346
00:26:19.685 --> 00:26:21.625
as you mentioned, it's focused primarily
347
00:26:22.005 --> 00:26:29.000
at data scientists and data analysts who are trying to address some of these real world problems. And I'm wondering what are some of the ways that you've maybe seen
348
00:26:29.720 --> 00:26:37.340
Arcuda used to address some of the problems of data scale and data maintenance as well, where maybe a data engineer would use it to understand
349
00:26:37.945 --> 00:26:39.725
what is the distribution
350
00:26:40.025 --> 00:26:48.350
of data assets or asset categorization that I have in my lake where I'm, you know, working at terabyte or petabyte or exabyte scale.
351
00:26:48.650 --> 00:26:50.669
Arcura really provides a capability
352
00:26:50.970 --> 00:26:54.110
that doesn't exist today. So if you're using a data lake
353
00:26:54.515 --> 00:27:00.135
and you're doing analytics, you would have to pull information out of that lake and find out processing system
354
00:27:00.595 --> 00:27:03.335
where you can ask your analytics or queries.
355
00:27:03.820 --> 00:27:09.840
So a data engineer would have to maintain that lake, and that may be federated across 1 or or more systems.
356
00:27:10.300 --> 00:27:16.424
What we're really doing is providing a new capability for the productive use of those large data sets.
357
00:27:16.885 --> 00:27:18.745
So by productive, I want
358
00:27:19.125 --> 00:27:22.024
a data scientist to be able to ask queries
359
00:27:22.470 --> 00:27:23.290
on terabytes
360
00:27:23.670 --> 00:27:30.170
or even petabytes of data without the need for a data engineer in the middle to be able to gate
361
00:27:30.675 --> 00:27:31.175
and
362
00:27:31.635 --> 00:27:32.135
really
363
00:27:32.595 --> 00:27:33.095
provide
364
00:27:33.475 --> 00:27:39.610
the services that a data scientist would need in order to ask those questions. So I wanna make the data accessible.
365
00:27:39.910 --> 00:27:48.235
I wanna make it productive to be able to access those large data sets, even combine large data sets together. So, again, we're trying to cut out barriers
366
00:27:48.695 --> 00:27:51.275
to solve these really large data science problems
367
00:27:51.815 --> 00:27:56.050
and to be able to do it with removing all of the
368
00:27:56.670 --> 00:28:03.010
roadblocks and remove all of the friction that we would normally face for solving these very large problems.
369
00:28:05.935 --> 00:28:08.595
It's time to make sense of today's data tooling ecosystem.
370
00:28:09.135 --> 00:28:10.755
Go to data engineering podcast.com/rudder
371
00:28:12.095 --> 00:28:17.420
to get a guide that will help you build a practical data stack for every phase of your company's journey to data maturity.
372
00:28:17.880 --> 00:28:22.140
The guide includes architectures and tactical advice to help you progress through 4 stages,
373
00:28:22.485 --> 00:28:27.225
starter, growth, machine learning, and real time. Go to data engineering podcast.com/rudder
374
00:28:28.565 --> 00:28:32.960
today to drop the modern data stack and use a practical data engineering framework.
375
00:28:34.940 --> 00:28:36.480
In terms of the
376
00:28:36.860 --> 00:28:43.024
data management aspect of this, I'm curious, what are some of the specifics in terms of the types of
377
00:28:43.404 --> 00:28:50.490
data that you're able to work with? So thinking in terms of structured versus semi structured versus unstructured or binary and things like that,
378
00:28:50.790 --> 00:28:52.010
and some of
379
00:28:52.470 --> 00:28:53.210
the organizational
380
00:28:53.510 --> 00:28:55.370
aspects of how best to
381
00:28:56.150 --> 00:29:08.390
work to the strengths of Arcuda in as to how you organize and structure the data so that it's able to be able to parallelize and take advantage of being able to shard and work across the data
382
00:29:08.790 --> 00:29:11.130
in parallel in isolation from each other?
383
00:29:11.510 --> 00:29:13.930
Great question. So natively, Arcuda
384
00:29:14.550 --> 00:29:18.544
manages 1 dimensional arrays, so collections of 1 d arrays.
385
00:29:18.845 --> 00:29:21.505
In our graph analytics, we built new data structures
386
00:29:22.205 --> 00:29:28.190
to have a native graph data structure to do these graph analytics on.
387
00:29:28.570 --> 00:29:29.870
And for many applications,
388
00:29:30.809 --> 00:29:36.755
we have data sets where we can decompose them into sets of these 1 d arrays.
389
00:29:37.215 --> 00:29:40.360
That said, the user doesn't have to worry about sharding
390
00:29:40.820 --> 00:29:42.680
or other partitioning techniques.
391
00:29:43.140 --> 00:29:46.120
The data will sit on a back end, and Chappell
392
00:29:46.420 --> 00:29:47.480
will have the responsibility
393
00:29:47.860 --> 00:29:48.600
of managing
394
00:29:49.225 --> 00:29:53.965
the distribution of that dataset across the available compute resources.
395
00:29:54.345 --> 00:29:57.165
So that sophistication is built into Chappell
396
00:29:57.759 --> 00:29:59.620
and means that we don't
397
00:30:00.080 --> 00:30:04.019
have to be as concerned about it because we have quite a sophisticated
398
00:30:04.695 --> 00:30:08.315
compiler and runtime system that's managing many of those aspects.
399
00:30:09.255 --> 00:30:11.674
And so in terms of the opportunities
400
00:30:11.975 --> 00:30:12.875
that you see
401
00:30:13.190 --> 00:30:14.730
as far as algorithmic
402
00:30:15.030 --> 00:30:20.090
advances for being able to work across these large datasets and being able to
403
00:30:20.655 --> 00:30:27.235
accelerate the time to insight, I'm wondering what are some of the open questions or unrealized
404
00:30:27.615 --> 00:30:28.115
capabilities
405
00:30:28.495 --> 00:30:29.475
of how to
406
00:30:29.790 --> 00:30:33.490
expand the functionality of things like Arcuda and just general
407
00:30:33.870 --> 00:30:35.330
large scale data analysis?
408
00:30:36.030 --> 00:30:41.715
Up to now, we really haven't had platforms where we could experiment with multi terabyte datasets
409
00:30:42.255 --> 00:30:46.195
productively in real time. And so this really opens your imagination
410
00:30:46.655 --> 00:30:50.710
for new types of analytics. For instance, let me focus on the graph space.
411
00:30:51.090 --> 00:31:05.505
There are some very capable tools out there that operate for analytics on graphs, for instance, on our laptops and small clusters. But once the graph becomes larger than a certain size, those frameworks typically don't allow
412
00:31:06.040 --> 00:31:06.540
those
413
00:31:06.840 --> 00:31:10.140
analytics to be run. They will be too time consuming
414
00:31:10.520 --> 00:31:12.620
or just too large for the analytic,
415
00:31:13.115 --> 00:31:20.335
and we can't explore that space. So I believe with Arcuda, we'll actually have the ability for the first time to
416
00:31:20.635 --> 00:31:22.415
ask questions that we thought
417
00:31:22.740 --> 00:31:24.920
were never possible previously
418
00:31:25.300 --> 00:31:26.840
on some of these large datasets.
419
00:31:27.140 --> 00:31:30.920
That may lead to new insights or even new algorithms and analytics
420
00:31:31.465 --> 00:31:32.605
or interrogating
421
00:31:33.304 --> 00:31:36.924
and really getting more insights from these large datasets.
422
00:31:37.705 --> 00:31:39.164
In terms of your experience
423
00:31:39.610 --> 00:31:49.070
of working with Arcuda and supporting teams who are building analyses on top of Arcuda, what are some of the most interesting or innovative or unexpected ways that you've seen it applied?
424
00:31:49.455 --> 00:31:58.115
Right now, Arcuda has a dozen to 20 plus users. It's really in its infancy providing this very highly capable framework.
425
00:31:58.610 --> 00:32:07.510
And so we're also looking forward to seeing people download it, clone it from the Git repository, and use it, and hearing more about those success stories.
426
00:32:07.815 --> 00:32:10.395
It's been quite useful for the enterprise
427
00:32:10.695 --> 00:32:19.490
and the users that we've seen pick it up and use it. Many of them are working in places with large data sets, for instance, from cybersecurity
428
00:32:20.830 --> 00:32:22.690
detecting cyber threats. And
429
00:32:22.990 --> 00:32:27.125
some are working in social network analysis with very large social networks.
430
00:32:27.505 --> 00:32:31.205
And we hope to continue to see success stories from Arcuda
431
00:32:31.664 --> 00:32:33.125
in the coming months
432
00:32:33.505 --> 00:32:38.250
years as more and more users adopt this highly productive and capable framework.
433
00:32:38.950 --> 00:32:49.735
And in your own experience of working on Arcuda and contributing to it and using it for some of your own research? What are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
434
00:32:50.355 --> 00:32:51.095
My background
435
00:32:51.530 --> 00:32:53.870
comes from high performance computing and supercomputing.
436
00:32:54.330 --> 00:32:57.870
So at this point, nothing really surprises me. But my
437
00:32:58.410 --> 00:32:59.390
pleasant surprise
438
00:32:59.915 --> 00:33:02.895
was that the Chappell compiler that we're using in the background
439
00:33:03.195 --> 00:33:04.335
is quite sophisticated.
440
00:33:05.115 --> 00:33:08.895
So many things that we normally would have to do by hand on other systems,
441
00:33:09.470 --> 00:33:11.410
we found that the Chappell compiler
442
00:33:11.870 --> 00:33:19.065
was able to find the performance that we needed. There's a great support team also at HPE with supporting
443
00:33:19.684 --> 00:33:22.745
this Chapel framework, and we found
444
00:33:23.365 --> 00:33:26.985
really learning new ways of implementing our code
445
00:33:27.420 --> 00:33:31.520
that can better leverage the Chapel compiler. So it's been a great experience,
446
00:33:31.820 --> 00:33:32.320
and
447
00:33:32.700 --> 00:33:34.320
we are also
448
00:33:34.654 --> 00:33:37.635
pleased that we're getting the performance that we anticipated,
449
00:33:38.255 --> 00:33:45.410
that there were no major bottlenecks or roadblocks to getting the performance that we expected on these large datasets.
450
00:33:46.190 --> 00:33:58.375
For people who are interested in being able to work at these scales in an interactive fashion, what are the cases where Arcuda is the wrong choice and they're better suited going with some of these other parallel compute frameworks?
451
00:33:59.059 --> 00:34:07.960
That's a great question. So if your dataset isn't massive, you probably don't wanna go through the effort of setting up Arcuda. If you can solve it easily today
452
00:34:08.595 --> 00:34:12.535
on your laptop or on the systems that you have, then
453
00:34:12.914 --> 00:34:17.414
you should stick with what you have. But if you find that you have datasets that you
454
00:34:17.800 --> 00:34:19.660
can't process or that the
455
00:34:20.119 --> 00:34:22.060
speed bump to get your analysts
456
00:34:22.600 --> 00:34:23.820
being able to productively
457
00:34:24.200 --> 00:34:29.375
ask questions of those datasets, then maybe in those cases, you should consider Arcuda.
458
00:34:29.675 --> 00:34:33.855
So, again, if things work well for you today, then maybe you're not
459
00:34:34.290 --> 00:34:41.109
the ideal candidate for moving to Arcuda. It's only when you start facing these issues of having
460
00:34:41.444 --> 00:34:48.184
datasets that are too large or not having the performance that you're seeking, near real time or interactive
461
00:34:48.565 --> 00:34:49.065
performance
462
00:34:49.605 --> 00:34:50.905
on these massive datasets,
463
00:34:51.670 --> 00:34:54.090
then you should consider looking at Arcuda.
464
00:34:54.710 --> 00:35:05.335
As you continue to iterate on and contribute to the Arcuda framework, what are some of the things that you have planned for the near to medium term or any problem spaces that you're excited to dig into?
465
00:35:05.875 --> 00:35:09.335
We're very excited in a few areas. For instance, stringology.
466
00:35:10.000 --> 00:35:12.980
How do we design more data sets that can process
467
00:35:13.280 --> 00:35:20.335
strings of text quite well? So this is important when we're analyzing documents or looking at unstructured text.
468
00:35:20.714 --> 00:35:22.414
We're trying to build those capabilities
469
00:35:22.795 --> 00:35:23.855
into Arcura,
470
00:35:24.155 --> 00:35:25.135
and we have
471
00:35:25.515 --> 00:35:28.095
funding from the US National Science Foundation
472
00:35:28.440 --> 00:35:30.060
to explore these areas
473
00:35:30.440 --> 00:35:31.340
in Arcuda.
474
00:35:31.880 --> 00:35:34.860
Another area that I'm quite excited by is looking at
475
00:35:35.160 --> 00:35:37.420
what we would think of as table joins
476
00:35:37.934 --> 00:35:40.755
that normally are going to be expensive operations
477
00:35:41.055 --> 00:35:42.194
within databases.
478
00:35:42.895 --> 00:35:46.755
Here, we're looking at Arcuda and what does it mean to do a join
479
00:35:47.130 --> 00:35:47.630
of
480
00:35:48.010 --> 00:35:52.029
the datasets that we have, and is there a way to optimize those
481
00:35:52.490 --> 00:35:54.910
to do it in the fastest way possible?
482
00:35:55.355 --> 00:36:01.455
So there's a lot of hard work that we're gonna do at New Jersey Institute of Technology to try to build out new capabilities
483
00:36:02.075 --> 00:36:05.050
for processing different types of large datasets
484
00:36:05.350 --> 00:36:06.730
with the Arcuda framework.
485
00:36:07.270 --> 00:36:12.090
Are there any other aspects of Arcuda or large scale graph analytics
486
00:36:12.435 --> 00:36:23.880
or being able to do interactive analysis on large scale datasets that we didn't discuss yet that you'd like to cover before we close out the show? I think this is an exciting area. We already know datasets are getting larger,
487
00:36:24.420 --> 00:36:27.880
but we also have to understand that often we get data
488
00:36:28.265 --> 00:36:29.724
not as just a big
489
00:36:30.105 --> 00:36:32.924
block of data to process, but as a stream of data.
490
00:36:33.224 --> 00:36:35.885
That stream may be updated every millisecond,
491
00:36:36.265 --> 00:36:38.150
every second, every hour,
492
00:36:38.530 --> 00:36:43.910
every day, and so on. And we wanna have tools that can process those streaming
493
00:36:44.369 --> 00:36:45.190
data streams.
494
00:36:45.865 --> 00:36:49.645
And what I want to be able to do is build out new tools and new capabilities
495
00:36:50.025 --> 00:36:54.685
to look at massive streaming data analytics. I want to get away from
496
00:36:55.040 --> 00:36:56.500
using these resources
497
00:36:57.040 --> 00:36:59.140
just for doing forensic analysis
498
00:36:59.680 --> 00:37:03.140
after something egregious happened, and we're trying to explore,
499
00:37:03.515 --> 00:37:04.335
for instance, in a
500
00:37:04.635 --> 00:37:09.695
cyber hack, how did they get in? What did they destroy? What did they exfiltrate?
501
00:37:09.995 --> 00:37:11.775
How do we protect against it?
502
00:37:12.210 --> 00:37:18.069
That is after the damage has been done. Where I wanna move to is predictive analytics.
503
00:37:18.369 --> 00:37:20.789
Can we take these data streams and detect
504
00:37:21.135 --> 00:37:23.555
that there's a change, a pattern,
505
00:37:24.015 --> 00:37:24.515
or
506
00:37:24.815 --> 00:37:25.795
something emerging
507
00:37:26.095 --> 00:37:28.275
that we can protect against before
508
00:37:28.735 --> 00:37:36.060
some egregious event? So I'm really excited by building out more tools for predictive analytics on these massive datasets.
509
00:37:36.840 --> 00:37:45.085
For anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question,
510
00:37:45.385 --> 00:37:51.480
I'd like to get your perspective on what you see as the biggest gap in the tooling our technology that's available for data management today.
511
00:37:51.940 --> 00:37:57.160
I think the biggest gap that we have is really the seamless integration of multiple tools.
512
00:37:57.705 --> 00:37:59.405
So there's a great
513
00:37:59.785 --> 00:38:04.125
number of tools and suites out there, some commercial, some open source,
514
00:38:04.720 --> 00:38:05.220
and
515
00:38:05.600 --> 00:38:08.100
that they serve many different purposes.
516
00:38:08.640 --> 00:38:10.740
But when we look at the data science environment,
517
00:38:11.280 --> 00:38:12.020
we see
518
00:38:12.400 --> 00:38:12.900
the
519
00:38:13.424 --> 00:38:22.164
ability to, for instance, use Jupyter Notebooks or workflows that we can record, which I think is a great advance. But what I'd like to see more is the
520
00:38:22.490 --> 00:38:22.990
compatibility
521
00:38:23.369 --> 00:38:25.150
among multiple tool sets
522
00:38:25.450 --> 00:38:26.990
so that we can move between
523
00:38:27.290 --> 00:38:32.765
the different tools and environments out there. For instance, I have datasets where sometimes I wanna look at them
524
00:38:33.145 --> 00:38:34.045
as unstructured
525
00:38:35.145 --> 00:38:44.460
datasets. Other times, I wanna view them as a graph. Other times, I wanna view them in a different light as well. And I wanna be able to move through tools that specialize
526
00:38:45.000 --> 00:38:49.135
in that view of the data and to be able to do it seamlessly without having to
527
00:38:49.435 --> 00:38:50.575
modify those datasets
528
00:38:50.955 --> 00:38:53.935
or go through different workflows on different systems.
529
00:38:54.440 --> 00:39:12.205
Yeah. It's definitely a very real problem these days as we get into specialization of these different tools, and I'm definitely excited for some of these investments that are happening in the metadata layer where maybe we can use that as the interchange point without having to do all kinds of custom integration between these different tool sets.
530
00:39:12.540 --> 00:39:14.960
Exactly. I think you hit the nail on the head.
531
00:39:15.420 --> 00:39:22.855
Alright. Well, thank you very much for taking the time today to join me and share the work that you are doing on the Arcuda framework. It's definitely a very interesting
532
00:39:23.875 --> 00:39:29.095
platform, interesting capabilities that it's unlocking. So definitely excited to see that continue to
533
00:39:29.420 --> 00:39:41.745
evolve and grow in terms of capabilities and adoption. So I appreciate all the time that you and the other members of the team are putting into that, and I hope enjoy the rest of your day. Thanks, Tobias. And I really hope anyone out there looks up Arcuda
534
00:39:42.205 --> 00:39:43.825
and works on their large scale
535
00:39:44.285 --> 00:39:49.770
data science problems with productive and capable tool sets. Thanks again for chatting.
536
00:39:55.975 --> 00:39:59.755
Thank you for listening. Don't forget to check out our other shows, podcast.init,
537
00:40:00.295 --> 00:40:06.220
which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning podcast,
538
00:40:06.680 --> 00:40:11.180
which helps you go from idea to production with machine learning. Visit the site at dataengineeringpodcast.com.
539
00:40:12.955 --> 00:40:22.710
Subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a product from the show, then tell us about it. Email hosts at data engineering podcast.com
540
00:40:23.250 --> 00:40:29.355
with your story. And to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.