Building ETL Pipelines With Generative AI

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 12:26:48 PM
Duration: 3096.553
Channels: 1
1
00:00:11.264 --> 00:00:15.445
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.384 --> 00:00:18.289
Introducing RudderStack Profiles.
3
00:00:18.750 --> 00:00:28.535
RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team.
4
00:00:29.154 --> 00:00:35.495
You specify the customer traits, then profiles runs the joins and computations for you to create complete customer profiles.
5
00:00:36.310 --> 00:00:39.450
Get all of the details and try the new product today at dataengineeringpodcast.com/runnerstack.
6
00:00:42.310 --> 00:01:04.700
You shouldn't have to throw away the database to build with fast changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades old batch computation model for an efficient incremental engine to get complex queries that are always up to date. With Materialise, you can. It's the only true SQL streaming database built from the ground up to meet the needs of modern data products.
7
00:01:05.465 --> 00:01:16.910
Whether it's real time dashboarding and analytics, personalization and segmentation, or automation and alerting, Materialise gives you the ability to work with fresh, correct, and scalable results, all in a familiar SQL interface.
8
00:01:17.290 --> 00:01:18.830
Go to data engineering podcast.com/materialize
9
00:01:20.410 --> 00:01:31.305
today to get 2 weeks free. Your host is Tobias Macey. And today, I'm interviewing Jay Mishra about the applications for generative AI in the ETL process. So, Jay, can you start by introducing yourself?
10
00:01:31.920 --> 00:01:32.420
Absolutely.
11
00:01:32.720 --> 00:01:34.180
Thanks for having me, Tobias.
12
00:01:34.480 --> 00:01:40.955
This is Jay Mishra. I am the chief operating officer at Astera. I have been in this field for over, 2 decades,
13
00:01:41.655 --> 00:01:47.330
specifically with the data management over 13 years, and I've been party to a lot of implementations
14
00:01:47.790 --> 00:01:49.250
at fortune 500 companies,
15
00:01:49.950 --> 00:01:51.170
for the ETL implementations,
16
00:01:51.870 --> 00:01:53.490
for data warehousing implementations,
17
00:01:54.475 --> 00:01:58.735
for various other use cases under the umbrella of data management. I have participated,
18
00:01:59.195 --> 00:02:01.375
from beginning all the way to the end, including
19
00:02:02.060 --> 00:02:03.600
implementations using our toolset.
20
00:02:04.060 --> 00:02:07.840
And do you remember how you first got started working in data? Interesting story.
21
00:02:09.100 --> 00:02:10.175
We had
22
00:02:10.715 --> 00:02:12.895
a small module in our product,
23
00:02:13.595 --> 00:02:14.415
that did
24
00:02:15.355 --> 00:02:16.815
user friendly data mapping.
25
00:02:17.400 --> 00:02:22.380
So it was basically a very simple ETL tool that is designed for
26
00:02:22.760 --> 00:02:23.580
non programmers.
27
00:02:24.200 --> 00:02:26.540
Back then, it was a novel idea to give
28
00:02:27.145 --> 00:02:31.085
a GUI based tool to the people who are doing ETL.
29
00:02:31.865 --> 00:02:34.125
It was mostly about schema mapping and transformations.
30
00:02:34.890 --> 00:02:40.909
We presented it to 1 of our largest customers and they liked it. And, that's how our journey started.
31
00:02:41.370 --> 00:02:46.905
And, of course, over the years, we got a lot of feedback from customers, market, kept adding features,
32
00:02:47.685 --> 00:02:50.265
took the feedback of our customers very seriously,
33
00:02:50.645 --> 00:02:50.965
and,
34
00:02:51.525 --> 00:02:53.305
kept building upon it. And,
35
00:02:54.030 --> 00:02:59.409
yeah, 12 years later, here we are. We have a full data stack, that is able to ingest data,
36
00:02:59.790 --> 00:03:02.205
transform, of course, load into any
37
00:03:02.525 --> 00:03:05.345
architectures of your choice, whether it is a data vault or,
38
00:03:06.045 --> 00:03:07.345
star schema or
39
00:03:07.885 --> 00:03:11.665
any of the, choice that you have for architecture for data warehousing,
40
00:03:12.370 --> 00:03:15.810
and, then publish it to your your end users. So using,
41
00:03:16.850 --> 00:03:17.510
no code,
42
00:03:18.050 --> 00:03:19.030
low code APIs.
43
00:03:19.845 --> 00:03:26.825
So we have at this point a full platform that is able to ingest data all the way to publish your data and everything in between.
44
00:03:27.530 --> 00:03:38.095
And in the context of ETL, what are some of the different ways that you're seeing generative AI applied and some of the the types of impact that you would expect
45
00:03:38.395 --> 00:03:40.175
for practitioners who
46
00:03:40.635 --> 00:03:49.340
want to be able to just say, give me this data, bring it over there. I don't wanna have to care about the details. Excellent question. So this is something that we are seeing,
47
00:03:49.820 --> 00:03:53.740
being asked of us, actually as a as an ETL vendor,
48
00:03:54.775 --> 00:03:59.435
very frequently. It started about couple of years ago. The impact of AI
49
00:03:59.815 --> 00:04:02.590
on the entire data space, I would say. Not only ETL,
50
00:04:03.210 --> 00:04:08.830
overall data management. That started to happen about a couple of years ago. We also jumped in around the same time.
51
00:04:09.155 --> 00:04:09.655
And,
52
00:04:10.275 --> 00:04:13.094
we see various areas getting impacted,
53
00:04:13.875 --> 00:04:19.199
by AI, specifically generative AI. So starting with, the data ingestion itself,
54
00:04:19.500 --> 00:04:21.520
the e in ETL is extraction.
55
00:04:22.060 --> 00:04:25.075
And, we see that extraction piece
56
00:04:25.475 --> 00:04:27.975
is heavily impacted with the AI.
57
00:04:28.595 --> 00:04:32.055
Wherever you have unstructured data, data that is text based,
58
00:04:33.199 --> 00:04:34.740
data that has images,
59
00:04:35.199 --> 00:04:36.740
all those areas are getting
60
00:04:37.599 --> 00:04:39.620
helped quite a bit with the AI.
61
00:04:40.255 --> 00:04:40.655
And,
62
00:04:41.375 --> 00:04:42.255
extraction piece,
63
00:04:42.735 --> 00:04:47.955
if it is structured data, not so much because the structured data AI still has inaccuracies.
64
00:04:48.900 --> 00:04:53.320
But unstructured data is where this helping in a big way. Now coming to transformations,
65
00:04:53.940 --> 00:04:57.915
there also we see that some teams, of data engineers
66
00:04:59.014 --> 00:05:02.475
are using actually AI to generate code for transformations,
67
00:05:03.270 --> 00:05:04.490
data quality as well.
68
00:05:04.790 --> 00:05:08.410
And, schema mapping is another 1 where, AI is impacting.
69
00:05:08.950 --> 00:05:10.490
And then, overall
70
00:05:11.350 --> 00:05:11.850
automation.
71
00:05:12.565 --> 00:05:20.025
In my opinion, this is the area where AI has made the the biggest impact, where you can look at repetitive tasks
72
00:05:20.330 --> 00:05:23.150
in the entire solution design and development
73
00:05:23.530 --> 00:05:28.270
and, use AI to automate it. So the usability has gone up. User experience
74
00:05:28.650 --> 00:05:29.630
has changed significantly.
75
00:05:30.194 --> 00:05:32.514
So the user interfaces to the,
76
00:05:33.395 --> 00:05:38.935
what used to be GUI, now it is going 1 level up. We are seeing actually chatbot style
77
00:05:40.210 --> 00:05:40.710
interfaces,
78
00:05:41.490 --> 00:05:46.710
to various applications including some areas in ETL. For example, Dataprep. In fact, our own Dataprep,
79
00:05:47.330 --> 00:05:47.830
functionality
80
00:05:48.435 --> 00:05:56.375
now has an interface where, you know, you can speak, or you can just chat in plain English and give instructions about what you need to do with your data.
81
00:05:56.750 --> 00:05:59.389
And, AI is able to generate the right,
82
00:05:59.870 --> 00:06:01.729
script or right, metadata
83
00:06:02.190 --> 00:06:05.915
for it and do the work for you. So this is how we are seeing,
84
00:06:06.715 --> 00:06:11.375
AI impacting various areas. But to me, the 2 standouts would be usability
85
00:06:11.960 --> 00:06:17.800
and helping repetitive task automation. So these 2 will stand out and, of course, other areas,
86
00:06:18.520 --> 00:06:20.860
are also getting impacted by decision making
87
00:06:21.255 --> 00:06:23.755
that, that is done by typical users.
88
00:06:24.055 --> 00:06:29.850
AI is doing helping, actually, in parts to make the decision for you. And for
89
00:06:30.470 --> 00:06:32.890
the integration of these
90
00:06:33.270 --> 00:06:34.090
AI capabilities
91
00:06:34.390 --> 00:06:35.850
in the ETL process,
92
00:06:36.555 --> 00:06:38.414
how does that shift the
93
00:06:38.715 --> 00:06:39.215
intended
94
00:06:39.595 --> 00:06:43.775
user of that technology where in a straight ETL environment,
95
00:06:44.620 --> 00:06:52.640
typically, you would see that be the responsibility of a data engineer. But as you were mentioning, there are also tools or scenarios where there are nontechnologists
96
00:06:53.264 --> 00:06:57.925
who are domain experts or business experts who want to be able to do that work.
97
00:06:58.305 --> 00:07:06.060
How does the application of AI shift that equation of who is responsible for actually doing this data integration work? Yeah.
98
00:07:06.520 --> 00:07:08.220
That is also changing
99
00:07:09.224 --> 00:07:10.844
rapidly, actually. That whole,
100
00:07:11.224 --> 00:07:11.724
responsibilities
101
00:07:12.185 --> 00:07:12.685
distribution,
102
00:07:13.705 --> 00:07:18.125
is changing. We see that more and more business facing people. People
103
00:07:18.530 --> 00:07:21.030
who have, no background in coding,
104
00:07:22.130 --> 00:07:23.270
no technical background,
105
00:07:23.810 --> 00:07:26.229
they are able to take, some responsibilities
106
00:07:26.690 --> 00:07:27.650
off the shoulders of,
107
00:07:28.235 --> 00:07:40.210
for lack lackability on data people. So people who are responsible for data, they are delegating some of the responsibilities. So the cross functional teams, the nature of that team is changing as well. And, in terms of,
108
00:07:40.930 --> 00:07:41.910
the data engineers
109
00:07:42.530 --> 00:07:43.030
and
110
00:07:43.535 --> 00:07:52.995
the ETL developers, that's the original word, original term for the people who are actually developing the solution. The rule is shifting a little bit. And actually, it is shifting in the right direction.
111
00:07:53.300 --> 00:07:57.080
In my opinion, they should not be tasked with doing,
112
00:07:57.860 --> 00:08:02.280
the same task again over again for several I don't know. We have seen in implementations
113
00:08:02.824 --> 00:08:10.525
for several weeks, several months, you are doing very similar tasks. So for example, if you're doing a data integration task and you have
114
00:08:10.940 --> 00:08:18.080
dozens of tables on the source side and you're building a pipeline, you are building the similar pipelines for each of these entities
115
00:08:18.414 --> 00:08:31.560
and it takes days, sometimes weeks. So this is the kind of task where you don't want to spend your data engineers' time. Rather, they should be focusing on the task that is really, interesting and it is adding more value. So the repetitive,
116
00:08:32.120 --> 00:08:34.380
any any task that is being, that is repetitive,
117
00:08:35.485 --> 00:08:44.545
that is, being automated. And, data engineers are able to focus on more interesting and more valuable tasks. So that's how we we see the the rule shifting.
118
00:08:44.870 --> 00:08:48.170
And, the subject matter experts, they are also coming
119
00:08:48.470 --> 00:08:53.750
into the picture. So they're working closely with the data engineers. So that's how we see the dynamics changing,
120
00:08:54.584 --> 00:08:57.324
in the teams that are implementing, the data solutions.
121
00:08:57.865 --> 00:09:00.204
You mentioned a little bit as to the
122
00:09:00.745 --> 00:09:30.170
specific types of projects or specific types of data where generative AI is going to provide the most impact. But I'm wondering if we can dig a bit deeper into that where you were saying that for highly structured use cases, it's, you know, maybe an incremental win, but with unstructured data is where you're going to see the largest gains. I'm wondering if you can talk to some of the reasons that that is and some of the ways that teams should be thinking about their initial forays into applying AI to their ETL use cases.
123
00:09:30.774 --> 00:09:31.274
Right.
124
00:09:31.735 --> 00:09:33.274
So structured versus unstructured,
125
00:09:33.975 --> 00:09:39.660
that debate has been going on for some time. And we see that, unstructured data, when you're extracting
126
00:09:40.040 --> 00:09:45.240
data from that, it's mostly insight. You're looking at taking a portion of it and,
127
00:09:45.880 --> 00:09:57.840
a little bit of approximation is okay. So So for example, if you have a document and you want to get a summary of that document, that summary doesn't have to be exact. Whereas, if you're looking at a table that where you have structured,
128
00:09:58.540 --> 00:10:02.000
rows and columns, in aggregation, even a small
129
00:10:02.540 --> 00:10:04.080
difference is not okay.
130
00:10:04.595 --> 00:10:05.095
So
131
00:10:05.475 --> 00:10:08.055
that's how I I look at it, and we see that,
132
00:10:08.915 --> 00:10:11.975
AI by nature is going to be nondeterministic.
133
00:10:12.675 --> 00:10:13.334
It has
134
00:10:14.390 --> 00:10:15.770
sometimes seemingly errors.
135
00:10:16.310 --> 00:10:28.774
But if it is 95% accurate, is that acceptable if you are dealing with a structured data? Whereas with unstructured data, 95% accurate is pretty good. So that's the key difference between, the structured versus, unstructured data. And,
136
00:10:29.654 --> 00:10:31.115
unstructured data scenarios,
137
00:10:31.654 --> 00:10:35.980
in fact, we saw that in recent past. All the rule based solutions,
138
00:10:37.000 --> 00:10:48.295
they are being replaced completely. So rule based solutions used to be that, hey, I'm looking for in NL using maybe, NLP then in proximity of this keyword, look at these other keywords.
139
00:10:48.755 --> 00:10:52.535
And if they're matching with the context, give me this in this information.
140
00:10:52.920 --> 00:11:04.285
That's how it used to be. But now with the with the AI, specifically generative AI, you do the similarity search and you you create and you put in your vector d v and then you create, like, top 5
141
00:11:04.825 --> 00:11:06.605
matching ones and send it to AI.
142
00:11:07.065 --> 00:11:08.925
Let's say, for example, OpenAI,
143
00:11:09.779 --> 00:11:19.240
GPT, and, the results are pretty good. So we did experiments and, the insights coming out of those calls are really good for unstructured data.
144
00:11:19.975 --> 00:11:31.540
Now we did the same experiment with us, with structured data, sending a table and asking for certain calculations and all that, and it will have hallucinations. And those are kind of, indicators of,
145
00:11:32.180 --> 00:11:35.720
with the structured data that approximation is not going to work.
146
00:11:36.135 --> 00:11:39.675
So with the structured data, where we see AI helping
147
00:11:40.055 --> 00:11:40.375
is,
148
00:11:41.655 --> 00:11:45.595
where AI can be used to configure the existing ETL solutions
149
00:11:46.089 --> 00:11:53.230
and, automate it. So instead of x number of hours, you're spending only half x numbers of hours,
150
00:11:53.775 --> 00:11:58.735
or 1 1 quarter maybe to configure the solution and your savings are there. So AI on,
151
00:11:59.215 --> 00:12:05.930
unstructured data is involved completely. The structured data, it is helping in configuring the solution and making the usability
152
00:12:06.310 --> 00:12:10.214
or user experience go up. To your point of needing to
153
00:12:10.834 --> 00:12:32.850
bring your data into the context of the vector DB for doing that similarity search, Not quite yet, but a little further along, I wanna talk through some of the architectural aspects of being able to integrate AI into the ETL process. But another aspect of that similarity search brings up the question of entity extraction and entity resolution. I'm curious what you're seeing as the impact of these
154
00:12:33.630 --> 00:12:38.829
AI models for being able to simplify or accelerate the process of doing that,
155
00:12:39.555 --> 00:12:44.535
entity resolution or master data management for the ETL end result?
156
00:12:44.995 --> 00:12:48.990
Yes. The metadata. You bring up a very, important point.
157
00:12:49.630 --> 00:12:50.670
That is something that,
158
00:12:51.310 --> 00:12:58.450
we see where AI is helping in a big way. Not the data, but the metadata itself. Because metadata, again, is a decision making
159
00:12:59.365 --> 00:13:01.125
process. So when someone is,
160
00:13:01.605 --> 00:13:06.345
let's say, a data architect is looking at the data, doing the the initial stages of
161
00:13:06.960 --> 00:13:29.399
data discovery, looking at what I'm dealing with. So a lot of, concepts of metadata are coming from from that stage where you're looking at your data and trying to figure out what you're dealing with. And a lot of decision making is involved there. So not only in MDM, we are seeing it in all different areas where you're looking at the data and, letting AI do the at least initial cut for you and then review it.
162
00:13:29.755 --> 00:13:45.410
And if you like, you move forward with that. That is a model that we see being applied at design time, specifically applied to metadata. 1 example that comes to my mind is, data modeling. So related to MDM, of course, if you're looking at your your source data and you want to
163
00:13:45.774 --> 00:13:46.995
create your entity,
164
00:13:47.535 --> 00:13:54.190
the ER framework that is, you're looking at, building a model where you want to decide that what makes sense and,
165
00:13:54.810 --> 00:13:58.910
what kind of, entities you want to create. And also moving forward,
166
00:13:59.530 --> 00:14:01.710
let's say that you want to design a data warehouse.
167
00:14:02.075 --> 00:14:04.335
In data warehouse, what could be a candidate
168
00:14:05.035 --> 00:14:05.535
for
169
00:14:05.995 --> 00:14:07.455
your facts, for your dimensions?
170
00:14:07.995 --> 00:14:19.199
If you're dealing with a data vault, what could be your hubs and satellites and links? All those things basically take time. So practitioners, they spend quite a bit of time in making those determinations
171
00:14:20.015 --> 00:14:24.675
and then, curating it. And that's how the design process works. And we are seeing that
172
00:14:25.295 --> 00:14:42.365
AI applied can do the first cut for you in a matter of few seconds. And then the first cut, if it is 90% there, your work is reduced by 90%. So that's the gain that we are seeing with the AI in all the metadata based decisions. So wherever you're trying to handle,
173
00:14:43.245 --> 00:14:45.105
metadata related decision making,
174
00:14:45.440 --> 00:14:51.700
AI is is helping in a big way. And now as far as the architectural aspects and the workflow,
175
00:14:52.240 --> 00:14:54.820
you mentioned needing to have some of these,
176
00:14:55.555 --> 00:15:06.390
contextual cues to the language models. I'm wondering if you can just talk to some of the architectural and workflow aspects of being able to bring AI into the process of ETL
177
00:15:06.850 --> 00:15:11.865
rather than ETL being used to feed the feed the AI. So the ETL workflows,
178
00:15:12.485 --> 00:15:14.985
they don't change much. AI comes into
179
00:15:15.365 --> 00:15:24.899
the ETL flow in different stages, at least for now. It's changing rapidly, and we'll we'll we'll come to that question a little later. But at least at this point, what we are seeing is that the ETL workflow
180
00:15:25.200 --> 00:15:35.075
stays the same. And for each of the steps, we see AI being applied. So let's, have a look at, a typical ETL workflow. The first step is data extraction.
181
00:15:35.440 --> 00:15:36.500
And in data extraction,
182
00:15:37.120 --> 00:15:37.839
we do have,
183
00:15:38.720 --> 00:16:09.945
traditional connectors that will go into your source, try to figure out the metadata. And if you have the metadata based on that, it it will do the parsing of your data, bring in the right data into your pipeline, and then then the mapping starts. So in the first step where you are trying to figure out the layout of your source data, when you're trying to read the data. So reading, of course, AI is not helping as much, but figuring out what is the structure of your data. That part is, metadata building, and we see AI being used. In fact, we have released a feature
184
00:16:10.680 --> 00:16:11.160
on,
185
00:16:11.640 --> 00:16:30.639
addressing exactly that point where it is automatically able to figure out what source you're dealing with, what is the structure of it, what are the columns and their data types and so on and so forth. It can handle all of that. So this is AI being applied to the reading part. Now if you have unstructured data, sometimes you are dealing with, for example, let's say,
186
00:16:31.100 --> 00:16:46.840
we have tons of PDF documents contains unstructured data that has paragraphs and tables hidden inside it. And, you have a specific prompt based on which you want to get certain data from each of these documents. Now that becomes your source. So this is your ingestion. You can apply
187
00:16:47.220 --> 00:16:47.720
maybe,
188
00:16:48.580 --> 00:16:49.240
a pretrained,
189
00:16:49.700 --> 00:16:52.440
model or let's say that, a fine tuned LLM.
190
00:16:52.754 --> 00:16:59.574
And we have done experiments with those as well, and it results so beautiful. So you can apply those at the ingestion stage itself
191
00:17:00.160 --> 00:17:05.140
to get you more quality data, more meaningful data. So that is the ingestion state.
192
00:17:05.520 --> 00:17:08.020
Now we go to the next step that is,
193
00:17:08.845 --> 00:17:20.740
mapping, data mapping. In data mapping, again, it's, it is a task that is meant for a combination of subject matter experts and people, of course, who are developing the solution. It takes a lot of time, a lot of,
194
00:17:21.300 --> 00:17:28.514
trial and errors about, hey. This field, does it go here versus here? How do we combine it and so on and so forth? And there also, we are seeing,
195
00:17:30.674 --> 00:17:33.174
AI helping in a big way. You can give
196
00:17:33.634 --> 00:17:34.134
the
197
00:17:34.514 --> 00:17:39.830
context to AI about this is the subject matter and, here is a list of,
198
00:17:40.549 --> 00:17:42.890
my source fields. Here is a list of my destination
199
00:17:43.350 --> 00:17:43.850
candidates
200
00:17:44.335 --> 00:17:51.135
and let it figure out. And we have seen that results again are pretty good. It can do the work in a matter of a second a few seconds instead of,
201
00:17:51.740 --> 00:17:56.960
going through a few iterations and coming up with this, this map, and then you're going to be verifying it,
202
00:17:57.340 --> 00:18:16.065
with your teams and all that. The first cut again is done very quickly. Of course, the verification step is going to be there. But the mapping can be, done, let's say, 1 tenth of the time that was required earlier. Then also in transformations, we are seeing that sometimes there are tools, of course, like ours where the transformations are drag and drop.
203
00:18:16.544 --> 00:18:29.250
And you can map and, you can get going. But there are some tools that use coding. And, in those tools, of course, AI can help you with generating the transformation code automatically. So that is the transformation side and loading as well.
204
00:18:29.710 --> 00:18:32.370
If you're loading into, let's say, a data warehouse,
205
00:18:32.750 --> 00:18:33.889
loading is not easy.
206
00:18:34.495 --> 00:18:40.415
You have to write very complicated SQL code, with the inserts and updates and and so on and so forth. And,
207
00:18:40.975 --> 00:18:51.360
some tools, they do have rule based solution that that can automatically generate the code for you. But if you don't, you can use AI to generate the code for you. So AI is being used as kind of a
208
00:18:51.735 --> 00:19:03.080
assistant or or a helping hand in all the different stages at this point, keeping the same workflow for the ETL. Documentation is where, the last step I would say where, again, AI is helping,
209
00:19:03.540 --> 00:19:04.740
where you can,
210
00:19:05.220 --> 00:19:06.280
generate the documentation,
211
00:19:07.460 --> 00:19:14.995
by giving it the context and giving your models and other pipelines. And it can describe your pipelines in a in a pretty,
212
00:19:15.554 --> 00:19:17.575
I would say, reasonable way,
213
00:19:18.290 --> 00:19:24.630
where you can generate doc document that is going to be describing your ETL pipeline. So that's how we see, AI,
214
00:19:25.090 --> 00:19:26.550
getting plugged into different,
215
00:19:27.205 --> 00:19:30.025
stages of ETL workflow at this point. Later,
216
00:19:30.485 --> 00:19:32.985
we are seeing this area advance pretty quickly.
217
00:19:33.445 --> 00:19:33.945
So,
218
00:19:34.325 --> 00:19:42.630
we are expecting it to not only help in all different areas, but also the decision making that is, at this point, kinda limited to
219
00:19:43.175 --> 00:19:46.475
localized area, it may kind of, grow from there
220
00:19:46.775 --> 00:19:47.995
and make bigger decisions
221
00:19:48.535 --> 00:19:51.515
such as, hey. Now I am looking at these 5 sources.
222
00:19:51.820 --> 00:19:55.440
Can I automatically join these 5 and give you meaningful data in my destination?
223
00:19:56.460 --> 00:20:07.495
So dynamic ETL, real time ETL, and all that, that's where the future is going. But then again, that is, we are not there yet, but we see that the AI can help in, those areas as well in future.
224
00:20:10.200 --> 00:20:18.940
This episode is brought to you by DataFold, a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production.
225
00:20:19.565 --> 00:20:29.264
DataFold leverages data diffing to compare production and development environments and column level lineage to show you the exact impact of every code change on data, metrics, and BI tools,
226
00:20:29.630 --> 00:20:32.130
keeping your team productive and stakeholders happy.
227
00:20:32.830 --> 00:20:39.875
DataFold integrates with DBT, the modern data stack, and seamlessly plugs in your DataCI for team wide and automated testing.
228
00:20:40.415 --> 00:20:46.835
If you are migrating to a modern data stack, DataFold can also help you automate data and code validation to speed up the migration.
229
00:20:47.360 --> 00:20:50.260
Learn more about DataFold by visiting data engineering podcast.com/datafold
230
00:20:51.840 --> 00:20:52.340
today.
231
00:20:54.240 --> 00:20:55.935
And as far as
232
00:20:56.555 --> 00:20:57.535
the model selection,
233
00:20:58.155 --> 00:21:03.535
that is an area that is constantly moving with the OpenAI GPT models
234
00:21:03.870 --> 00:21:09.090
grabbing a lot of the headlines and attention, but the open source models are also very quickly
235
00:21:09.550 --> 00:21:15.635
leapfrogging each other and catching up with the OpenAI models as well as some of the ones available from Anthropic, etcetera.
236
00:21:16.095 --> 00:21:31.015
Has there been any feedback that you've seen as far as which models perform best for which use cases or for particular technology stacks, and I'm wondering what you see as some of the useful benchmarks or metrics for teams who are starting that evaluation process
237
00:21:31.415 --> 00:21:38.395
of, I want to bring in AI, but I need to be aware of some of the platform risk of depending on this for all of my day to day operations.
238
00:21:39.070 --> 00:21:41.650
So this area is changing, rapidly.
239
00:21:42.030 --> 00:21:46.210
When OpenAI started out and we got access to open, OpenAI's APIs,
240
00:21:46.750 --> 00:21:52.595
we did some experiments. And followed by that quickly, we started to see a lot of these open source,
241
00:21:52.975 --> 00:21:58.620
LLMs coming out where, of course, the parameters are not as high as OpenAI, but they're reasonable
242
00:21:59.000 --> 00:22:04.805
and you can have it a local copy of it. That was a huge deal because, with OpenAI,
243
00:22:05.185 --> 00:22:07.685
even if you are getting the best results,
244
00:22:07.985 --> 00:22:09.915
there are many issues where you have to,
245
00:22:10.880 --> 00:22:13.140
send everything, to OpenAI APIs,
246
00:22:13.520 --> 00:22:16.260
and the performance sometimes was not great.
247
00:22:16.640 --> 00:22:19.300
Whereas, we started to play with Llama, Netformata,
248
00:22:19.895 --> 00:22:21.195
and, then, of course,
249
00:22:21.655 --> 00:22:23.915
at this point, we have 5 or 6,
250
00:22:24.535 --> 00:22:25.035
different,
251
00:22:25.575 --> 00:22:26.075
models
252
00:22:26.535 --> 00:22:27.275
that we
253
00:22:27.800 --> 00:22:49.960
offer in our toolset. And this is given to the users where they can experiment with any of them, and they can fine tune it with their with their data. And once it is fine tuned to their data, they can use any of them. So that's how we present it. We are agnostic to which 1 do you use. But now coming back to your question about which 1 works better for which kind of data. So 1 pattern that we notice is,
254
00:22:50.680 --> 00:23:03.330
the performance, of course, if you're dealing with the unstructured data, chat gpt, that still stays at the top. The performance or the quality is about the best from there for unstructured data. We did 1 more experiment with,
255
00:23:04.210 --> 00:23:07.250
creating natural language interface for our own,
256
00:23:08.050 --> 00:23:13.475
expression language. So in our product, we have an expression language where you can write formulas to do calculations,
257
00:23:14.255 --> 00:23:18.820
and it looks like, formulas that you see in Microsoft Excel. That kind of,
258
00:23:19.760 --> 00:23:22.820
learning, not a whole lot, but still for some users,
259
00:23:23.360 --> 00:23:26.980
it is a bit difficult that, hey. I have to write these formulas and expressions,
260
00:23:27.304 --> 00:23:28.205
to do some calculations.
261
00:23:28.745 --> 00:23:29.885
So we did experiment
262
00:23:30.345 --> 00:23:36.205
over that, scenario where we gave a natural language interface to generate those formulas.
263
00:23:36.809 --> 00:23:38.170
And there we are seeing that,
264
00:23:38.890 --> 00:23:42.990
llama is a pretty good option. It does pretty well. Llama 2 now.
265
00:23:43.290 --> 00:23:45.230
So, depending on the scenario,
266
00:23:45.845 --> 00:23:50.985
it changes at which 1 is gonna be performing better. So what I would recommend is
267
00:23:51.525 --> 00:23:55.410
that experiment with all top 4 or 5. It doesn't take much.
268
00:23:55.810 --> 00:24:01.350
And there are tools like ours available now that are going to be that will give you a playground
269
00:24:01.890 --> 00:24:03.410
where you can experiment with,
270
00:24:04.215 --> 00:24:23.845
fine tuning the 3 or 4 models and see that which 1 works the best. Because it is not at least in our experience, it is not that winner takes all. In certain areas, we see that OpenAI is doing much better than others, but there are some areas where others are doing better than OpenAI. So I would suggest to experiment and see that which 1 is, the best fit
271
00:24:24.145 --> 00:24:25.845
for your kind of data.
272
00:24:26.145 --> 00:24:27.285
And the process
273
00:24:27.745 --> 00:24:29.525
and, the tools of department,
274
00:24:29.990 --> 00:24:35.210
they are they're getting there already. There are many solutions available out there that are going to let you experiment.
275
00:24:35.669 --> 00:24:38.455
Now the other fun piece of
276
00:24:38.755 --> 00:24:41.015
working with AI is that it is nondeterministic,
277
00:24:41.475 --> 00:24:54.080
and so there is the potential for logical errors, logical bugs to come in, and you're not even necessarily going to get the exact same output for the exact same input, especially if you're dealing with successive generations of models.
278
00:24:54.395 --> 00:25:00.255
I'm wondering what are some of the ways that teams need to be thinking about error handling, error identification,
279
00:25:01.410 --> 00:25:05.750
validating the outputs of the AIs before they put it into production, etcetera?
280
00:25:06.130 --> 00:25:06.850
Great question.
281
00:25:07.170 --> 00:25:11.294
This is something that we have been dealing with even in our internal implementations
282
00:25:11.595 --> 00:25:15.534
and, even our in our own own coding that we did for the product.
283
00:25:16.559 --> 00:25:17.059
So,
284
00:25:17.440 --> 00:25:20.740
of course, the stochastic nature of, the predictions
285
00:25:21.440 --> 00:25:24.340
makes it suitable for certain things, but not suitable
286
00:25:24.640 --> 00:25:25.460
for the other.
287
00:25:25.975 --> 00:25:26.475
We
288
00:25:27.095 --> 00:25:29.195
tend to recommend using
289
00:25:29.735 --> 00:25:30.235
AI
290
00:25:30.535 --> 00:25:32.395
where it is not going to be deterministic
291
00:25:32.695 --> 00:25:34.155
only for the design time.
292
00:25:34.799 --> 00:25:40.019
Don't let it be in the run time. That is actually 1 of the principles that we have agreed upon
293
00:25:40.320 --> 00:25:43.220
that if it is at the run time, it is going to cause issues.
294
00:25:43.885 --> 00:25:44.385
And,
295
00:25:45.245 --> 00:25:50.385
the only way possibly to use it in at run time is to have a strong layer
296
00:25:50.685 --> 00:25:51.745
of data validations
297
00:25:52.380 --> 00:25:53.679
that will reject certain,
298
00:25:55.020 --> 00:25:57.760
things done by AI if it is not meeting your standards.
299
00:25:58.220 --> 00:26:05.015
And then it throws it back to you saying that, okay. Hey. Have a look at it. So that is the only way to use AI at run time.
300
00:26:05.555 --> 00:26:08.295
Otherwise, at design time is where we see
301
00:26:08.770 --> 00:26:09.830
a lot of value.
302
00:26:10.530 --> 00:26:12.450
At design time, we see that,
303
00:26:13.170 --> 00:26:13.670
where,
304
00:26:14.370 --> 00:26:15.750
you are making decisions
305
00:26:16.370 --> 00:26:16.870
and
306
00:26:18.265 --> 00:26:18.765
even,
307
00:26:19.465 --> 00:26:19.965
implementations.
308
00:26:21.065 --> 00:26:23.485
The first cut that is created by AI
309
00:26:24.184 --> 00:26:25.404
is way faster
310
00:26:25.940 --> 00:26:30.440
than even an expert of, let's say, 20 years or 25 years, 30 years,
311
00:26:31.220 --> 00:26:36.544
will will come up with. So we see a lot of gains on that front, And the benefit is,
312
00:26:37.345 --> 00:26:40.325
the biggest, I would say benefit is that once the
313
00:26:40.705 --> 00:26:44.860
recommendation is in front of you, you can review it and override it.
314
00:26:45.320 --> 00:26:47.580
So that capability you have to have.
315
00:26:47.960 --> 00:26:50.655
In your process, make sure that you have that built in,
316
00:26:51.535 --> 00:26:57.315
capability to review the work that is done by AI and override it if need be.
317
00:26:57.750 --> 00:26:59.270
So look at AI as your,
318
00:27:00.470 --> 00:27:03.930
assistant who's helping you in doing some work.
319
00:27:04.230 --> 00:27:09.225
And you're not going to trust it blindly. You're going to look at the work done by
320
00:27:09.685 --> 00:27:10.745
AI, review it.
321
00:27:11.125 --> 00:27:19.480
And then if it looks good, sure. No problem at all. It goes to the next step. But if it doesn't, then you have a way to fix it, and then it goes to the next step.
322
00:27:20.259 --> 00:27:22.915
So mostly at design time, we are using it. And,
323
00:27:23.355 --> 00:27:26.414
we do have, some places where we let AI,
324
00:27:27.595 --> 00:27:30.095
handles certain pieces in the at the runtime.
325
00:27:30.500 --> 00:27:35.240
But there, we make sure that you are using some kind of rule based checking
326
00:27:35.540 --> 00:27:36.360
of the results.
327
00:27:36.740 --> 00:27:37.800
So data validation,
328
00:27:38.674 --> 00:27:40.934
the module, that we have that is,
329
00:27:41.635 --> 00:27:43.815
that is a must that you have to apply
330
00:27:44.434 --> 00:27:51.420
after the AI step. Any step that involves AI, after that, you have to have, data quality checks and data validation,
331
00:27:52.200 --> 00:27:53.240
to make sure that,
332
00:27:54.355 --> 00:27:56.695
at runtime, you have your eyes on AI.
333
00:27:57.155 --> 00:28:01.895
That's how I put it. And you mentioned that you wisely don't incorporate
334
00:28:02.275 --> 00:28:07.320
the AI into the actual runtime behavior, but just in the design and implementation
335
00:28:07.700 --> 00:28:09.320
phase. And so for
336
00:28:09.700 --> 00:28:27.825
ongoing maintenance purposes, what do you see as the ongoing role of the AI as you maintain and evolve the different pipelines or try to implement new pipelines that maybe feed off of some of the ones that are already implemented, things like that. Great. So on that front,
337
00:28:28.205 --> 00:28:31.345
the rule is increasing. Actually, I'll take that analogy of assistant.
338
00:28:31.965 --> 00:28:33.585
Your assistant is being trained.
339
00:28:34.125 --> 00:28:34.945
They are doing
340
00:28:35.470 --> 00:28:39.410
certain tasks now, but once they are trained, they can do bigger tasks.
341
00:28:40.350 --> 00:28:43.090
So that rule is also evolving and,
342
00:28:43.665 --> 00:28:44.965
we do see already
343
00:28:46.465 --> 00:28:53.680
some, some tool sets or some some teams working on it. That is where we increase the responsibility of AI
344
00:28:54.060 --> 00:28:54.800
to be
345
00:28:55.660 --> 00:28:57.120
our assistant to do monitoring
346
00:28:57.580 --> 00:28:58.800
of existing flows.
347
00:28:59.585 --> 00:29:04.485
So monitoring is another part where we see a huge role that AI can play.
348
00:29:05.185 --> 00:29:07.605
So here, it is not impacting your real data,
349
00:29:08.299 --> 00:29:10.880
but it is helping you as a user
350
00:29:11.580 --> 00:29:13.279
that that would be your responsibility
351
00:29:13.580 --> 00:29:14.080
otherwise
352
00:29:14.380 --> 00:29:15.360
to look into
353
00:29:16.380 --> 00:29:18.655
your data pipelines, how they're behaving,
354
00:29:19.035 --> 00:29:22.175
what kind of data you're getting, then any errors that you are receiving,
355
00:29:22.795 --> 00:29:28.620
what is the frequency. So it is pretty good on that front. It can tell you that, hey, most likely this is what is going wrong
356
00:29:28.920 --> 00:29:35.715
and you can go and fix it. So data anomalies inside your metadata at runtime, your information that is coming out from runtime.
357
00:29:36.415 --> 00:29:36.895
If,
358
00:29:37.375 --> 00:29:43.740
we can have an eye into that process, that is very useful. And AI can do that for us. So AI is being used
359
00:29:44.360 --> 00:29:44.860
to
360
00:29:45.320 --> 00:29:47.100
help us monitor the entire
361
00:29:47.480 --> 00:29:49.419
workflow that we have deployed at runtime.
362
00:29:50.125 --> 00:29:51.345
So that is 1 area.
363
00:29:51.805 --> 00:29:54.365
And, also for the areas of,
364
00:29:55.165 --> 00:29:57.980
design or implementation that that we're talking about earlier,
365
00:29:58.540 --> 00:29:59.040
there,
366
00:29:59.500 --> 00:30:02.559
the role is getting more and more advanced, more complex.
367
00:30:03.340 --> 00:30:05.840
So we are seeing that if you have
368
00:30:06.415 --> 00:30:08.195
designed certain flows in past
369
00:30:08.575 --> 00:30:10.835
and you were seeing there are certain errors or certain
370
00:30:11.135 --> 00:30:12.995
issues that you're seeing with those flows,
371
00:30:13.350 --> 00:30:16.250
it can detect it. And, also, it can generate
372
00:30:17.030 --> 00:30:17.770
a recommendation
373
00:30:18.070 --> 00:30:21.370
about, hey. Instead of using this flow, how about you try this flow?
374
00:30:21.965 --> 00:30:23.164
So that is,
375
00:30:23.645 --> 00:30:24.145
again,
376
00:30:24.525 --> 00:30:27.105
for design time, but a huge help
377
00:30:27.485 --> 00:30:31.025
because that is kind of troubleshooting. In troubleshooting, it takes time,
378
00:30:31.350 --> 00:30:37.290
and, AI can do that work for you. So in these these 2 areas, I see that, AI's role is gradually increasing.
379
00:30:39.945 --> 00:30:47.005
As more people start using AI for projects, 2 things are clear. It's a rapidly advancing field, and it's tough to navigate.
380
00:30:47.620 --> 00:30:49.960
How can you get the best results for your use case?
381
00:30:50.500 --> 00:30:59.615
Instead of being subjected to a bunch of buzzword bingo, hear directly from pioneers in the developer and data science space on how they use graph tech to build AI powered apps.
382
00:31:00.395 --> 00:31:03.294
Attend the dev and ML talks at NODES 2023,
383
00:31:04.080 --> 00:31:09.220
a free online conference on October 26th featuring some of the brightest minds in tech.
384
00:31:09.760 --> 00:31:12.500
Check out the agenda and register today at neo4j.com
385
00:31:13.615 --> 00:31:14.115
/nodes.
386
00:31:14.735 --> 00:31:15.235
That's
387
00:31:15.535 --> 00:31:16.035
ne0,
388
00:31:16.335 --> 00:31:17.395
the number 4,
389
00:31:17.775 --> 00:31:18.275
j.com/nodes.
390
00:31:23.360 --> 00:31:23.860
For
391
00:31:24.320 --> 00:31:26.420
teams and individuals and organizations
392
00:31:26.800 --> 00:31:33.015
who are considering this introduction of AI into their critical data flows, what are
393
00:31:33.555 --> 00:31:42.870
the typical motivators that you see and the types of questions and concerns that they need to address before they can feel comfortable with actually putting the results into production?
394
00:31:44.049 --> 00:31:46.130
Yeah. So on this front, we are seeing that,
395
00:31:47.570 --> 00:31:48.230
our customers,
396
00:31:48.850 --> 00:31:50.070
ask us this question,
397
00:31:50.615 --> 00:31:55.755
frequently now that how should we approach it? How should we incorporate AI into our solution,
398
00:31:56.455 --> 00:31:57.415
design and,
399
00:31:57.895 --> 00:31:58.395
implementations?
400
00:31:59.650 --> 00:32:00.150
And,
401
00:32:01.010 --> 00:32:02.150
of course, they have,
402
00:32:03.010 --> 00:32:03.510
some,
403
00:32:04.770 --> 00:32:07.990
reservations about it as well that should we be using AI
404
00:32:08.290 --> 00:32:08.950
or not.
405
00:32:09.815 --> 00:32:12.955
So from our side, we always suggest that, take your time.
406
00:32:13.414 --> 00:32:15.835
Come up with a strategy of how you want
407
00:32:16.855 --> 00:32:18.154
to incorporate AI,
408
00:32:19.130 --> 00:32:22.510
into your data solutions, your architecture, or overall,
409
00:32:22.850 --> 00:32:24.990
the organization wide, how you want to,
410
00:32:25.850 --> 00:32:27.230
to start using AI.
411
00:32:27.804 --> 00:32:32.304
And, there are a few things that are important. The first 1 is that you have to identify
412
00:32:33.404 --> 00:32:36.384
the areas or the scenarios in which AI is applicable.
413
00:32:37.070 --> 00:32:47.985
Then there are the issues with the the tooling that you need to have the right tool set available because that can make or break. And then, of course, the training of your resources. That is also very important. They need to be trained properly
414
00:32:48.525 --> 00:32:53.425
in using AI because, if it is not done properly, you're not gonna get any value out of it.
415
00:32:54.230 --> 00:32:55.289
Now coming to,
416
00:32:55.750 --> 00:32:58.570
the reservations or objections to AI and,
417
00:32:58.950 --> 00:33:02.730
there are many reasons for that. The first 1 that comes up is
418
00:33:03.345 --> 00:33:03.845
compliance
419
00:33:04.625 --> 00:33:05.125
explainability.
420
00:33:06.305 --> 00:33:09.605
The problem with the AI is that it's like a black box in many scenarios.
421
00:33:09.985 --> 00:33:14.020
And there are many industries where you have to have complete visibility
422
00:33:14.480 --> 00:33:15.380
into anything
423
00:33:15.920 --> 00:33:17.380
that is happening with your data.
424
00:33:18.320 --> 00:33:18.880
And those
425
00:33:20.225 --> 00:33:24.725
if you're using AI in certain scenarios, it is not going to give you that information.
426
00:33:25.424 --> 00:33:32.270
So you must identify the areas where you can use it without being able to explain what is happening inside this black box.
427
00:33:32.650 --> 00:33:36.030
So it's a kind of a design problem where you look at your,
428
00:33:36.730 --> 00:33:37.230
scenario
429
00:33:37.855 --> 00:33:43.315
and figure out that where you can fit in this black box, and that you are going to be still okay from the compliance perspective,
430
00:33:43.855 --> 00:33:44.355
from,
431
00:33:45.054 --> 00:33:45.554
explainability
432
00:33:45.934 --> 00:33:46.434
perspective.
433
00:33:47.350 --> 00:33:49.750
So that is 1, 1 issue. And,
434
00:33:50.390 --> 00:33:52.410
with the right design, of course, it can be addressed.
435
00:33:52.950 --> 00:33:54.730
And, then, of course,
436
00:33:55.190 --> 00:33:55.850
the second
437
00:33:56.274 --> 00:33:57.335
part would be about
438
00:33:57.955 --> 00:33:59.575
the data itself that you have.
439
00:33:59.875 --> 00:34:03.575
Sometimes data is suitable. Sometimes, it is not suitable for AI.
440
00:34:03.950 --> 00:34:06.690
For simple scenarios, AI is an overkill.
441
00:34:07.310 --> 00:34:08.110
If you have,
442
00:34:08.510 --> 00:34:17.965
a simple source database that you want to move it to your data warehouse, you don't need AI for that. You can simply plug in a standard ETL, and it's going to be much more cost effective.
443
00:34:18.425 --> 00:34:21.565
And it is going to be able to bring your data from source to destination,
444
00:34:22.670 --> 00:34:23.490
much quicker
445
00:34:23.950 --> 00:34:25.410
compared to using AI.
446
00:34:25.710 --> 00:34:28.450
So that's what we'll recommend that look at your scenario.
447
00:34:29.070 --> 00:34:30.690
Do the evaluation about
448
00:34:31.150 --> 00:34:34.135
how you want to use AI, if it is even applicable.
449
00:34:34.435 --> 00:34:39.575
If it is, then where does it go inside your use case? And that's how we go about designing
450
00:34:40.750 --> 00:34:41.890
a tailor made solution
451
00:34:42.190 --> 00:34:43.410
for each of our customers.
452
00:34:44.590 --> 00:34:45.490
In your experience
453
00:34:45.790 --> 00:34:47.730
of working with generative
454
00:34:48.190 --> 00:34:49.650
AI in the ATL context
455
00:34:50.015 --> 00:34:56.835
and onboarding people into these workflows, what are the most interesting or innovative or unexpected ways that you have seen the AI
456
00:34:57.214 --> 00:34:59.075
used in the context of ETL?
457
00:35:00.540 --> 00:35:07.520
Yeah. Interesting, question. So this is, I would say all of it that we have seen or what what I have said so far.
458
00:35:07.954 --> 00:35:11.095
When it started out pretty much every other week, we'll see something
459
00:35:11.795 --> 00:35:13.255
that we have never thought about
460
00:35:13.555 --> 00:35:15.895
being addressed by, generative AI.
461
00:35:16.420 --> 00:35:18.180
So that was a phase for about,
462
00:35:19.380 --> 00:35:20.760
6 months where
463
00:35:21.380 --> 00:35:21.860
every,
464
00:35:22.180 --> 00:35:27.755
couple of weeks, we see something that we never thought could be done, and it was there. So,
465
00:35:29.415 --> 00:35:32.055
of course, it is still going on, but we see this pace kind of,
466
00:35:33.799 --> 00:35:37.900
slowing down on on the new innovations, using generative AI in ETL.
467
00:35:38.200 --> 00:35:40.539
But the most common ones, I would say, is
468
00:35:41.705 --> 00:35:42.845
use of AI
469
00:35:43.625 --> 00:35:44.125
to
470
00:35:44.745 --> 00:35:47.165
get the insight from unstructured data.
471
00:35:47.545 --> 00:35:50.700
And I'll give you 1 example for that. So we had
472
00:35:52.200 --> 00:35:55.500
a use case where we had to get certain answers
473
00:35:56.200 --> 00:35:59.905
from our documents as if you're asking a question that, okay. Hey.
474
00:36:00.365 --> 00:36:02.385
Read this document and tell me
475
00:36:02.765 --> 00:36:04.305
what is the answer to this question.
476
00:36:05.060 --> 00:36:07.700
And, this is a very common pattern in,
477
00:36:08.180 --> 00:36:09.400
data insight gathering.
478
00:36:09.860 --> 00:36:13.240
And we had a solution that, of course, worked pretty reasonably.
479
00:36:15.105 --> 00:36:19.185
But this scenario to implement this scenario, it took us about,
480
00:36:19.665 --> 00:36:25.060
good 6 man months to write the right, solution, build it, test it, and all that.
481
00:36:26.160 --> 00:36:26.480
And,
482
00:36:27.920 --> 00:36:29.140
we started to experiment
483
00:36:29.520 --> 00:36:29.920
with,
484
00:36:30.674 --> 00:36:34.855
OpenAI's APIs. And, of course, we had to to experiment with the prompts and,
485
00:36:35.315 --> 00:36:37.335
when solution when the data comes in,
486
00:36:38.070 --> 00:36:42.890
we had we run into some issues on that front as well. But, eventually, once it was done,
487
00:36:43.830 --> 00:36:48.755
the results were better than what we had earlier, and it was done within 1 week.
488
00:36:49.375 --> 00:36:51.315
So that's the extent of,
489
00:36:52.015 --> 00:36:55.155
savings we are talking about, 6 months versus 1 week.
490
00:36:55.650 --> 00:36:58.550
And solution is even and and results are better. So
491
00:36:59.010 --> 00:37:03.955
when it comes to unstructured data, text, images, and all that, it's beautiful. The solution,
492
00:37:04.335 --> 00:37:08.115
based on generative AI, they are way better than what we had earlier.
493
00:37:08.815 --> 00:37:09.135
And,
494
00:37:09.615 --> 00:37:11.635
then, the next trend was
495
00:37:12.380 --> 00:37:12.880
semantic
496
00:37:13.420 --> 00:37:15.920
mapping. This is another 1 where we saw that
497
00:37:16.300 --> 00:37:17.040
our users
498
00:37:17.500 --> 00:37:18.000
struggled.
499
00:37:18.540 --> 00:37:21.515
Back in the day, we had, basically, auto map
500
00:37:21.994 --> 00:37:23.694
features where we will,
501
00:37:24.954 --> 00:37:26.255
try to figure out,
502
00:37:27.275 --> 00:37:31.290
that, column name in the source, what should it belong to in the
503
00:37:34.810 --> 00:37:36.590
we used to call it smart mapping.
504
00:37:37.450 --> 00:37:39.230
And, the smart mapping
505
00:37:39.655 --> 00:37:46.155
was okay. It used to do, like, I would say, about 60, 70 percent accurate, but still there was a lot of errors.
506
00:37:48.100 --> 00:37:49.000
Coming to picture,
507
00:37:49.460 --> 00:37:57.000
AI again. And, we started to use the semantic mapping and give the context, and now the accuracy goes up to about 95%.
508
00:37:58.214 --> 00:38:01.554
So we have seen kind of, almost like a,
509
00:38:01.894 --> 00:38:04.154
magical results in certain areas.
510
00:38:04.720 --> 00:38:06.020
1 more that,
511
00:38:06.640 --> 00:38:07.440
that is very,
512
00:38:08.240 --> 00:38:11.460
interesting experience, at least, personally for me, was
513
00:38:13.200 --> 00:38:13.700
creating
514
00:38:14.435 --> 00:38:15.415
the data models,
515
00:38:15.875 --> 00:38:16.375
for
516
00:38:16.755 --> 00:38:17.575
data warehousing.
517
00:38:18.035 --> 00:38:26.349
Data warehousing, data modeling is not easy. For practitioners who have been in the field for decades, they still take time. Figuring out what should be,
518
00:38:27.210 --> 00:38:31.950
a fact, what should be a dimension, how they should be connected, and what what kind of,
519
00:38:33.385 --> 00:38:39.165
configurations you wanna do for facts and dimensions, and also for other architectures such as data port and all that.
520
00:38:39.880 --> 00:38:41.340
So we did another experiment
521
00:38:41.720 --> 00:38:42.700
where we took
522
00:38:44.360 --> 00:38:45.580
our transactional database
523
00:38:46.360 --> 00:38:46.860
and
524
00:38:48.365 --> 00:38:53.184
created a data model out of it. So we have a reverse engineering functionality that can create the model automatically.
525
00:38:53.885 --> 00:38:56.625
No problem on that front. Once we have the data model,
526
00:38:57.049 --> 00:39:02.109
we let the AI decide for us that how should you convert this by denormalizing
527
00:39:02.490 --> 00:39:05.069
and creating a star schema for you.
528
00:39:05.375 --> 00:39:08.355
And, again, the results were amazing. It was almost, like, magical.
529
00:39:09.135 --> 00:39:13.555
We, you know, we get the right prompts, we get the right information, and there comes a data model,
530
00:39:14.510 --> 00:39:15.490
that looks like
531
00:39:15.869 --> 00:39:21.730
a perfect star schema that would otherwise have taken several days of iterations and
532
00:39:22.185 --> 00:39:25.725
back and forth with your subject matter experts and the data architect,
533
00:39:26.905 --> 00:39:30.079
was done in a matter of few seconds again. So these are the few,
534
00:39:30.400 --> 00:39:30.900
interesting,
535
00:39:31.680 --> 00:39:35.619
usage or scenarios that I can think of, but there are many.
536
00:39:36.885 --> 00:39:39.545
To summarize, I would say that, wherever,
537
00:39:40.565 --> 00:39:41.785
you have patterns
538
00:39:42.165 --> 00:39:44.185
and patterns have been applied in past,
539
00:39:44.650 --> 00:39:46.430
It is known to the practitioners,
540
00:39:47.609 --> 00:39:48.829
and it is repetitive.
541
00:39:49.530 --> 00:39:52.829
Let AI do the decision making for you, and you will not be
542
00:39:53.855 --> 00:39:56.195
disappointed. It is going to do
543
00:39:56.495 --> 00:39:59.315
pretty good job. At least the first cut is going to be amazing.
544
00:39:59.615 --> 00:40:06.320
And then, of course, you know, as we talked about earlier, you can take that and override it if need be. And in your experience
545
00:40:06.700 --> 00:40:18.115
of working in this space of ETL for so long and the introduction of generative AI as as a solution in this process, what are the most interesting or unexpected or challenging lessons that you've learned?
546
00:40:18.655 --> 00:40:19.155
Yes.
547
00:40:20.480 --> 00:40:20.980
On,
548
00:40:22.080 --> 00:40:24.100
early on, I would say, we
549
00:40:25.040 --> 00:40:27.460
and actually, like like anyone else, we
550
00:40:27.995 --> 00:40:29.695
we thought that it can do,
551
00:40:30.155 --> 00:40:30.975
much more,
552
00:40:32.555 --> 00:40:34.895
in terms of even, looking at the data
553
00:40:35.355 --> 00:40:35.855
and
554
00:40:37.520 --> 00:40:41.600
being more deterministic, I would say. The solutions were not that,
555
00:40:43.120 --> 00:40:44.980
or the results were not that satisfying.
556
00:40:45.600 --> 00:40:46.100
So
557
00:40:46.400 --> 00:40:46.900
we
558
00:40:47.505 --> 00:40:48.305
had to kind of,
559
00:40:48.945 --> 00:40:52.405
take a step back and, redo the all of the all that work.
560
00:40:52.705 --> 00:41:09.095
So it will sometimes make mistakes about simple things. Like, you know, I'm asking for what is the location of a specific field in my file and we'll make a mistake in that. That you can basically do with the naked eyes. You can see that, hey, it is in line number 2 and a 10th character. It can't even figure out that.
561
00:41:09.475 --> 00:41:14.935
It will make mistakes. And if you run it twice, 3 times, 4 times, maybe once it'll give you the right results.
562
00:41:15.660 --> 00:41:18.160
So anything where you're relying on
563
00:41:18.940 --> 00:41:20.000
giving any deterministic
564
00:41:20.300 --> 00:41:20.800
answers,
565
00:41:21.420 --> 00:41:24.755
forget about it. So we decided that wherever,
566
00:41:25.695 --> 00:41:28.994
precision is required, it is not a good fit,
567
00:41:29.910 --> 00:41:33.830
because we're trying with pretty much everything that is out there. We are trying out,
568
00:41:34.950 --> 00:41:37.210
even the the figuring out the locations.
569
00:41:37.590 --> 00:41:40.315
So we have actually an unstructured data extraction,
570
00:41:40.775 --> 00:41:41.275
module
571
00:41:41.895 --> 00:41:42.555
that can,
572
00:41:43.575 --> 00:41:45.035
based on templates, it can
573
00:41:45.734 --> 00:41:47.835
extract data points from your document.
574
00:41:48.450 --> 00:41:48.850
And,
575
00:41:49.250 --> 00:41:51.910
we were trying to build a template using AI.
576
00:41:52.210 --> 00:41:58.785
And for that, you need to get the locations of certain fields and patterns using which you can build, the template.
577
00:41:59.405 --> 00:42:03.025
And it was not a good experience. So we would try to do that and,
578
00:42:03.645 --> 00:42:05.780
every single time you run, you get a different result.
579
00:42:06.740 --> 00:42:09.320
And the whole algorithm will be messed up. So
580
00:42:09.780 --> 00:42:11.320
anything that is, indeterministic,
581
00:42:13.895 --> 00:42:15.575
it's okay to use,
582
00:42:15.895 --> 00:42:18.474
for those scenarios. But where do you expect,
583
00:42:19.734 --> 00:42:20.795
precise results,
584
00:42:21.450 --> 00:42:22.990
it's it's not gonna be a good fit.
585
00:42:23.530 --> 00:42:27.310
So barring that, I think it has been, pretty useful.
586
00:42:27.770 --> 00:42:30.270
But just keep in mind whenever you're designing something,
587
00:42:30.795 --> 00:42:31.755
go make sure that,
588
00:42:33.355 --> 00:42:38.895
you look at it carefully and see that are you expecting precise results? If yes, then be careful.
589
00:42:39.530 --> 00:42:43.790
Otherwise, approximate results. If you're okay with 95 percent if you're okay with 90%,
590
00:42:44.650 --> 00:42:46.830
you're good. But if you want 100%,
591
00:42:47.295 --> 00:42:48.595
please do not use it.
592
00:42:49.295 --> 00:42:54.995
You already mentioned this a little bit, but what are the cases where AI is the wrong choice for ETL applications
593
00:42:55.450 --> 00:42:58.910
and you should just go with just just do it manually, write the code,
594
00:42:59.290 --> 00:43:00.670
use the low code tool?
595
00:43:01.930 --> 00:43:02.430
Yes.
596
00:43:03.065 --> 00:43:03.565
So
597
00:43:05.305 --> 00:43:08.285
apart from what I said already, that is for simple scenarios.
598
00:43:08.664 --> 00:43:10.924
So if you have just a handful of sources,
599
00:43:11.240 --> 00:43:12.460
structured data mostly,
600
00:43:13.240 --> 00:43:15.020
and your destination is
601
00:43:15.320 --> 00:43:16.300
also neatly defined,
602
00:43:16.680 --> 00:43:18.940
ETL is still going to be more cost effective,
603
00:43:19.675 --> 00:43:21.855
and it is going to be actually a better choice.
604
00:43:22.715 --> 00:43:24.095
Whereas if you have,
605
00:43:24.395 --> 00:43:28.015
more complex data, you have unstructured data, you have
606
00:43:29.180 --> 00:43:29.680
scalability
607
00:43:29.980 --> 00:43:32.880
issues, as in your scale can grow,
608
00:43:33.740 --> 00:43:35.360
the volume of data can grow.
609
00:43:36.115 --> 00:43:38.455
And then, your overall data
610
00:43:39.155 --> 00:43:40.615
ecosystem or the entire,
611
00:43:40.995 --> 00:43:41.795
data management,
612
00:43:42.355 --> 00:43:45.170
platform that you're looking at, It has to be more,
613
00:43:45.730 --> 00:43:48.230
able to handle more complexity in future.
614
00:43:49.170 --> 00:43:49.670
Then
615
00:43:50.050 --> 00:43:54.155
we would recommend that you use AI based solutions or start using
616
00:43:54.535 --> 00:43:56.315
AI. But if it is a standard,
617
00:43:56.615 --> 00:43:57.195
a simple,
618
00:43:57.895 --> 00:44:04.640
data pipeline that is going to be building your data warehouse or a data integration use case, it is not gonna be cost effective.
619
00:44:05.340 --> 00:44:09.900
Also, it depends on, other aspects of implementation that is,
620
00:44:10.380 --> 00:44:10.880
how
621
00:44:12.105 --> 00:44:14.525
well trained is your team, how big is your team,
622
00:44:14.905 --> 00:44:16.845
kind of resources you have at your disposal.
623
00:44:17.545 --> 00:44:18.925
If you have a small team,
624
00:44:19.305 --> 00:44:22.099
it is, again, it is not, going to be,
625
00:44:23.200 --> 00:44:23.700
that,
626
00:44:25.040 --> 00:44:26.339
applicable to your scenario.
627
00:44:27.475 --> 00:44:28.195
And, also,
628
00:44:28.915 --> 00:44:32.595
it, depends on your strategy. That is 1 more thing,
629
00:44:32.995 --> 00:44:34.295
that I would like to add.
630
00:44:34.915 --> 00:44:45.855
We have seen some smaller teams, but their leadership, they want to bring in AI. If that is a scenario, definitely go for it. Even if it is an overkill in the beginning, but it's going to help you in future.
631
00:44:46.315 --> 00:44:47.615
So in that scenario,
632
00:44:48.395 --> 00:44:49.775
we do recommend that,
633
00:44:50.395 --> 00:45:00.180
you start using it from the beginning itself so that when the time arrives, you're ready for it. And as you continue to invest in this area, work with customers,
634
00:45:00.480 --> 00:45:12.185
work with your own technology stack, what do you see as the future applications for AI either in ETL specifically, but also in just the broad application of data engineering as a role?
635
00:45:13.830 --> 00:45:16.310
Yeah. This is a, a question that,
636
00:45:16.950 --> 00:45:18.090
that we discuss
637
00:45:18.550 --> 00:45:19.050
routinely.
638
00:45:19.590 --> 00:45:25.555
This is our design meetings. This is what we talk about, where the market is going, where what we should be doing.
639
00:45:25.855 --> 00:45:30.194
And 1 topic that is particularly fascinating is to
640
00:45:30.575 --> 00:45:31.555
let the AI
641
00:45:32.800 --> 00:45:35.060
do the real time or dynamic ETL.
642
00:45:36.080 --> 00:45:36.740
That is
643
00:45:37.760 --> 00:45:40.260
the decision making that we do at design time,
644
00:45:40.825 --> 00:45:42.525
can that happen in real time?
645
00:45:43.225 --> 00:45:47.405
Of course, it comes with, it's all all, caveats and all that, but
646
00:45:47.940 --> 00:45:48.680
it is,
647
00:45:49.220 --> 00:45:51.560
most definitely going in the direction where
648
00:45:52.500 --> 00:45:53.000
AI
649
00:45:53.940 --> 00:45:58.434
can make some decisions about what data streams or what, datasets
650
00:45:59.135 --> 00:46:01.375
can be merged and how they can be,
651
00:46:01.855 --> 00:46:02.355
transformed.
652
00:46:03.480 --> 00:46:04.220
So transformation
653
00:46:04.760 --> 00:46:07.580
tools and all that, maybe it can use your existing tool.
654
00:46:07.960 --> 00:46:10.780
But decision making about how to build
655
00:46:11.244 --> 00:46:13.585
the pipeline, what pipeline makes sense,
656
00:46:14.045 --> 00:46:16.865
that can be given in hands of AI. So kind of dynamic,
657
00:46:17.484 --> 00:46:18.464
ETO pipelines.
658
00:46:19.130 --> 00:46:23.869
This is where I think, the future is going where it can automatically generate those pipelines.
659
00:46:24.569 --> 00:46:26.190
And, it is going to be more
660
00:46:26.569 --> 00:46:27.069
declarative,
661
00:46:27.609 --> 00:46:28.109
where
662
00:46:28.625 --> 00:46:29.905
you can declare that,
663
00:46:30.464 --> 00:46:32.165
instructions given to the AI
664
00:46:32.545 --> 00:46:33.045
that,
665
00:46:33.424 --> 00:46:34.464
hey. If you look at,
666
00:46:35.025 --> 00:46:35.924
any new
667
00:46:36.224 --> 00:46:38.244
datasets coming in for my customers,
668
00:46:38.730 --> 00:46:42.030
They must you must apply this system of records
669
00:46:43.289 --> 00:46:47.869
and put this into this destination. This is instruction given to AI. Now it has to do the design,
670
00:46:48.224 --> 00:46:50.625
the internal tool sets. It may be using,
671
00:46:51.105 --> 00:46:52.805
a low code a low code solution,
672
00:46:53.105 --> 00:46:55.125
but it is the user of that,
673
00:46:55.505 --> 00:47:00.410
that that tool set. And it can make the decision for you about how to build that pipeline automatically.
674
00:47:01.030 --> 00:47:05.625
So this is, something that we we see that it is going to happen. And,
675
00:47:06.565 --> 00:47:07.545
of course, so,
676
00:47:08.005 --> 00:47:12.185
at we are still in early stages. We have done some experiments too in that on that front.
677
00:47:12.725 --> 00:47:13.285
There are some,
678
00:47:15.410 --> 00:47:23.475
obvious issues on that front. Again, we talked about how the decision making, if it goes wrong, what do you do about it? So there's still some questions we have to answer on that front,
679
00:47:24.035 --> 00:47:26.535
but it is definitely going in the direction where,
680
00:47:27.315 --> 00:47:31.200
AI can do a little bit more decision making at a higher level, and it can,
681
00:47:31.839 --> 00:47:34.180
dynamically build the ATL flows for you.
682
00:47:36.400 --> 00:47:44.635
Alright. Are there any other aspects of the applications for generative AI in the ETL workflow or just the overall
683
00:47:45.015 --> 00:47:45.515
ecosystem
684
00:47:45.815 --> 00:47:52.440
of data engineering and the ways that AI is going to impact it that we didn't discuss yet that you'd like to cover before we close out the show?
685
00:47:54.660 --> 00:47:59.285
So 1 area that we see where AI is particularly good at, that is,
686
00:47:59.765 --> 00:48:00.825
anomaly detection.
687
00:48:01.365 --> 00:48:02.805
So and it has been,
688
00:48:03.685 --> 00:48:05.385
not only the generative AI,
689
00:48:06.020 --> 00:48:06.520
even,
690
00:48:07.780 --> 00:48:09.480
the predecessors of AI,
691
00:48:10.020 --> 00:48:15.080
generative AI that is. In past, I would say almost like a decade, they have been used
692
00:48:15.465 --> 00:48:17.165
in certain scenarios like,
693
00:48:17.705 --> 00:48:19.705
fraud detection. So if you give,
694
00:48:20.025 --> 00:48:24.685
a time series data, it can figure out where something is wrong, and it can,
695
00:48:25.420 --> 00:48:26.320
help you with that.
696
00:48:26.780 --> 00:48:37.255
So that specific use case now is being u is applied to ETL as well. That is something that, I I see that it is being, more frequently used.
697
00:48:37.875 --> 00:48:39.975
And, briefly, we touched upon,
698
00:48:41.120 --> 00:48:41.940
in the context
699
00:48:43.760 --> 00:48:45.140
of, the runtime
700
00:48:46.400 --> 00:48:49.140
errors and monitoring of your ETL workflows.
701
00:48:50.075 --> 00:48:54.415
So this is, of course, just 1 part of it, but we can use
702
00:48:54.795 --> 00:49:02.520
AI to detect anything that is going on wrong. So here's my pattern. Here is how we use it. If you see something wrong, let us know.
703
00:49:02.820 --> 00:49:04.600
So it can be your eyes into,
704
00:49:05.140 --> 00:49:05.800
the runtime.
705
00:49:06.474 --> 00:49:06.974
And,
706
00:49:07.435 --> 00:49:09.775
also, of course, for the datasets itself.
707
00:49:10.315 --> 00:49:13.935
So not only the, the runtime, you can have a site process
708
00:49:14.359 --> 00:49:16.300
that can be looking at your incoming datasets.
709
00:49:16.920 --> 00:49:17.740
And beforehand,
710
00:49:18.040 --> 00:49:20.380
in profiling stage, if you see something
711
00:49:21.745 --> 00:49:24.885
drastically wrong or or different from what you expect,
712
00:49:25.265 --> 00:49:25.585
it,
713
00:49:26.385 --> 00:49:27.025
you can,
714
00:49:27.505 --> 00:49:29.765
kind of short circuit the entire pipeline
715
00:49:30.270 --> 00:49:37.250
and handle it in different way. So anomaly detection in the data itself and the metadata, these are the 2 areas where,
716
00:49:37.550 --> 00:49:38.530
I see that
717
00:49:39.105 --> 00:49:40.964
AI can, be a huge help.
718
00:49:41.345 --> 00:49:56.490
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
719
00:49:57.215 --> 00:50:04.810
This area is, changing rapidly and, a lot of tools at this point, they are early beta stage, sometimes alpha stage.
720
00:50:05.290 --> 00:50:10.430
And, new tools are coming pretty much every every week we see something new being launched.
721
00:50:10.890 --> 00:50:13.470
But 1 thing that I would love to see is
722
00:50:14.684 --> 00:50:15.184
more
723
00:50:15.645 --> 00:50:17.005
natural language based,
724
00:50:17.805 --> 00:50:18.305
UX.
725
00:50:19.164 --> 00:50:19.664
So
726
00:50:20.605 --> 00:50:21.585
as of now,
727
00:50:22.299 --> 00:50:27.760
the standard is, again, drag and drop UI. That is a golden standard where, for any ETL,
728
00:50:28.380 --> 00:50:28.880
platform
729
00:50:29.615 --> 00:50:31.715
or all data management tools, basically.
730
00:50:32.095 --> 00:50:35.955
The graphical user interface based tools, the drag and drop is the standard.
731
00:50:36.550 --> 00:50:39.450
I would like to see the natural language being implemented
732
00:50:39.990 --> 00:50:44.810
where you should be able to to speak with your toolset, and it does the work for you.
733
00:50:45.165 --> 00:50:46.525
So that is something that is,
734
00:50:47.085 --> 00:50:50.705
of course, in, in some products, including our product, it is
735
00:50:51.405 --> 00:50:53.025
starting out in certain areas,
736
00:50:53.380 --> 00:50:53.880
But,
737
00:50:54.339 --> 00:50:55.240
that can
738
00:50:55.779 --> 00:51:03.055
be taken to the next level, and I'm expecting that in within the next few months, it should be there. That will be the bridge between the AI,
739
00:51:03.595 --> 00:51:06.415
specifically generative AI, and ETL toolsets.
740
00:51:07.115 --> 00:51:09.295
So that is going to be kind of connecting
741
00:51:09.710 --> 00:51:12.849
the users with the tool sets in a much more meaningful way.
742
00:51:13.150 --> 00:51:27.695
Alright. Well, thank you very much for taking the time today to join me and share the experiences you've had with bringing AI into the ETL workflow. It's definitely a very interesting topic area, definitely 1 that is constantly moving. So I appreciate you,
743
00:51:28.589 --> 00:51:36.552
sharing your perspective on that and helping people get a leg up on that journey. So, thank you again, and I hope you enjoy the rest of your day. My pleasure. Thank you for having me.