WEBVTT
00:00:00.239 --> 00:00:07.519
You're listening to the Data Flowcast, the podcast about HD Airflow and the world of data and AI around it.
00:00:07.679 --> 00:00:14.560
We are excited to invite you to orchestrate every day for a free online conference on September 16th.
00:00:14.960 --> 00:00:17.760
Engineering data from Lyft RAM and WIC.
00:00:18.000 --> 00:00:28.320
I don't care how they are shipping production AI workflows on AI core for the new Astronomer Certified AI Orchestration Fundamental exam.
00:00:28.559 --> 00:00:31.600
Attendees get a discount code for a free certification.
00:00:32.000 --> 00:00:34.159
A$150 value.
00:00:34.399 --> 00:00:39.600
Go to astronomer.link slash data minus flowcast minus OE.
00:00:39.759 --> 00:00:41.119
Link in the show notes.
00:00:41.359 --> 00:00:43.439
Now let's get into the show.
00:00:43.759 --> 00:00:44.320
Alright.
00:00:44.479 --> 00:00:45.359
Hi everybody.
00:00:45.439 --> 00:00:45.920
Welcome.
00:00:46.079 --> 00:00:47.840
Welcome to the Data Flowcast.
00:00:47.920 --> 00:00:54.880
I'm your host, Mark Romati, and today I'm super excited because I'm joined by Marim Ralini, Data Platformat, head out.
00:00:55.359 --> 00:00:58.079
And uh first thing first, how are you doing today?
00:00:58.399 --> 00:01:00.479
I'm doing great, very excited to be here.
00:01:00.560 --> 00:01:05.120
Um, but I've like love looking forward to having this conversation with you again.
00:01:05.519 --> 00:01:08.239
Same, likewise, I'm super happy to have you here.
00:01:08.400 --> 00:01:23.439
Um, so what's gonna what we're gonna discover in today's podcast is really interesting because we'll have a quick tour of how Head Out, which is a travel platform, um, how they run DBT, MLOps, alerting and for detection in Airflow.
00:01:23.599 --> 00:01:28.560
So basically, we're gonna have this tour of the use cases that they have that they run in Airflow.
00:01:28.799 --> 00:01:34.719
And uh maybe Rinalini will tell us more about some technical details and so on, but we'll see.
00:01:34.959 --> 00:01:35.840
Uh okay.
00:01:36.000 --> 00:01:41.920
So, first thing first, can you introduce yourself and tell us what you do at uh Head Out?
00:01:42.400 --> 00:01:42.480
Okay.
00:01:42.879 --> 00:01:45.920
And maybe maybe a little bit of your background as well, if you want.
00:01:46.400 --> 00:01:46.640
Okay.
00:01:46.719 --> 00:01:48.239
Um, so hi, I'm Renalini.
00:01:48.480 --> 00:01:50.959
Um, I am a data platform engineer here at Head Out.
00:01:51.040 --> 00:01:52.560
Head Out is basically a travel platform.
00:01:52.640 --> 00:01:54.959
We try to bring real life experiences to people.
00:01:55.200 --> 00:01:59.120
Um, so here I work on a big breadth of things.
00:01:59.200 --> 00:02:01.120
That's the best part of my job, I feel like.
00:02:01.200 --> 00:02:04.319
Um, I get because it's a lean team, I get to work on everything.
00:02:04.480 --> 00:02:07.840
So I work from DVT to Airflow to MFOps to governance.
00:02:07.920 --> 00:02:08.960
So all of it end-to-end.
00:02:09.280 --> 00:02:10.319
Yeah, that's mostly it.
00:02:10.400 --> 00:02:11.680
Um, I've been here for four years.
00:02:11.759 --> 00:02:13.520
I enjoy it um a lot.
00:02:14.240 --> 00:02:15.199
That's great.
00:02:15.439 --> 00:02:20.560
Um, can you can you tell me more about like what what what exactly is a data platform engineer?
00:02:20.639 --> 00:02:22.240
I guess like your data platform engineer.
00:02:22.319 --> 00:02:27.199
Like, is it is it like your data engineer or just like you have also the DevOps part as well?
00:02:27.280 --> 00:02:30.000
Like what Yeah, it's a combination of both, I would say.
00:02:30.240 --> 00:02:34.479
So um I mostly handle the infra and enable people to do their best work.
00:02:34.639 --> 00:02:46.879
So it includes a bunch of toolings to make sure um people have the best tools available to do their work, um, and in touch also with like um the infra bits, um everything is optimized, everything is running fine.
00:02:46.960 --> 00:02:51.039
Um, and then also exploring newer tools, making sure everything is working fine.
00:02:51.120 --> 00:02:53.360
Um, then also involves a bunch of DevOps.
00:02:53.520 --> 00:02:56.639
We maintain a lot of services internally um for data.
00:02:56.800 --> 00:02:58.719
So yeah, a little bit of everything.
00:02:59.120 --> 00:02:59.680
Okay, okay.
00:02:59.759 --> 00:03:01.840
So you are like a full-stack data engineer.
00:03:02.159 --> 00:03:04.080
Yeah, I'm a jack of all trades.
00:03:04.479 --> 00:03:06.240
Okay, okay, that's interesting.
00:03:06.400 --> 00:03:13.439
Uh, just out of curiosity, um, I guess like, do you do you work with uh um data analysts, data scientists, and so on?
00:03:13.520 --> 00:03:16.400
Like, do you work with different teams like that in your uh yeah?
00:03:16.639 --> 00:03:20.960
Yeah, so um how it works in our team is basically we have a hub and spokes model.
00:03:21.039 --> 00:03:29.360
So the data platform team sets in the center, and all the other data scientists, data um analysts, all these people sit at different spokes, different in within different pods.
00:03:29.520 --> 00:03:36.319
Um so our work is to like kind of um make sure that they are doing their best work, um, help them in whatever way.
00:03:36.479 --> 00:03:38.960
Um, because it's a very lean team, everyone does everything.
00:03:39.039 --> 00:03:42.479
So our all of our analysts, all of our scientists also write their own data models.
00:03:42.560 --> 00:03:50.960
So we need to make sure that we have good enough standards and good enough documentation so that they write code and like it's a well-enough code, um, even like optimization for things.
00:03:51.039 --> 00:03:52.800
So yeah, it's like it sits in the middle.
00:03:52.960 --> 00:03:56.080
We work a lot with um and we try to make their lives easier.
00:03:56.719 --> 00:03:57.520
That's amazing.
00:03:57.680 --> 00:03:58.000
Okay.
00:03:58.319 --> 00:04:16.800
Um I guess the this was not uh uh um in the in the in the questions that we we discussed before, but I I I I'm just wondering like uh is Airflow was already there when you joined Head Out, or uh like did you decide to have Airflow in your company?
00:04:17.519 --> 00:04:23.040
So it was already there when I was there, but like we were doing very smaller things back then.
00:04:23.199 --> 00:04:26.319
Um it has now increased to do a lot more things.
00:04:26.639 --> 00:04:26.959
Oh okay.
00:04:27.120 --> 00:04:38.480
I was just wondering like if you uh you know if you had to go through uh the uh you know the install of Airflow and and also like maybe why did you choose Airflow and so on, but uh it's good to know that it was already there.
00:04:38.639 --> 00:04:42.000
Um okay, I guess like uh another question to you.
00:04:42.079 --> 00:04:46.000
So, what kind of data challenges uh come with running a travel platform?
00:04:46.160 --> 00:04:51.600
Um I think that's actually the first time that I interview someone that it is working for uh a travel platform.
00:04:51.759 --> 00:05:01.199
So I'm really interested to know more about that, uh, the data challenges that you have, and also where does Airflow sit in your uh in your stack?
00:05:01.680 --> 00:05:02.160
Got it.
00:05:02.319 --> 00:05:13.920
Okay, so um we already have like all the issues that come within a general data team of volume, data being wrong, or like the formatting being wrong, different kinds of file formats, different kinds of sources.
00:05:14.079 --> 00:05:15.199
So all of that is there.
00:05:15.360 --> 00:05:36.160
But specifically when you come to travel, there's like a few different things um that um are of a slightly bigger issue for us, especially things like inventory, where it changes very quickly, you need to be on top of things, you need to um run a lot of these things very quickly because a lot of your forecasting, a lot of your how much do you purchase, how much do you um not purchase, how much of it is going out.
00:05:36.240 --> 00:05:38.639
Um a lot of revenue sits on top of that, right?
00:05:38.800 --> 00:05:40.560
Um so inventory is a big thing.
00:05:40.720 --> 00:05:52.079
Um, then um for real-time use case, especially like fraud, uh, you need to do a lot of real-time analytics, you need to do a lot of graph analytics, um, you need to process a lot of things very quickly in real time.
00:05:52.319 --> 00:05:53.600
Uh, so that is one thing.
00:05:53.759 --> 00:06:06.160
We also have a very big performance marketing heavy um domain travel, uh, where like a lot of our um like we have a lot of our customers coming in from campaigns um that we run online.
00:06:06.319 --> 00:06:14.319
So that is a big data uh for us to manage um to maintain attribution um so that your money is actually going in the correct place where it should be going.
00:06:14.480 --> 00:06:20.879
So those are some things um that are slightly different, like slightly different problem statements uh from like a general industry standpoint.
00:06:21.040 --> 00:06:25.439
Um, Airflow sits at like, imagine like a nervous system of it all.
00:06:25.519 --> 00:06:27.279
It's basically anywhere that we need.
00:06:27.360 --> 00:06:34.959
So all the use cases that I told before, um, anything that we need to run, also like DVT, all the transformations happen through Airflow.
00:06:35.120 --> 00:06:45.199
Like it's a tie-in for every kind of different system that we have from ingestion to kind of um serving some data to transformation, all of it just sits in Airflow.
00:06:45.839 --> 00:06:46.160
Okay.
00:06:46.639 --> 00:06:49.600
Um did you mention real time at some point?
00:06:49.920 --> 00:06:50.560
Yes.
00:06:51.040 --> 00:06:55.279
So how do you use Airflow with some real-time?
00:06:56.000 --> 00:07:00.000
So real-time-ish orcha, I gotcha.
00:07:00.319 --> 00:07:07.360
Um for some things you need to run it like at a two, three-minute interval, maybe a minute interval, five-minute interval, those.
00:07:07.519 --> 00:07:11.920
Um for more real-time things we use Flink, um, but like for real-time-ish things.
00:07:12.000 --> 00:07:14.319
Um, no, that that's interesting.
00:07:14.480 --> 00:07:24.959
Um, yeah, I mean, I I was just wondering because like like you're not the first person that I hear about uh real-time use cases, and and also it was more like okay, two, two, two, three minutes.
00:07:25.439 --> 00:07:28.399
Um you also mentioned dbt.
00:07:28.639 --> 00:07:33.360
I mean, obviously dbt is a very, very popular um tool uh today.
00:07:33.680 --> 00:07:39.040
But can you tell us more about like what is your dbt setup with Airflow?
00:07:39.120 --> 00:07:42.480
Um, and uh you know how you uh interact with it.
00:07:42.959 --> 00:07:43.519
Got it.
00:07:43.680 --> 00:07:48.079
Um so our uh setup in dbt is a write audit publish setup.
00:07:48.160 --> 00:07:57.519
Um we basically believe in that um stale data, slightly stale data is better than wrong data because a lot of our decisions are taken because of from data.
00:07:57.600 --> 00:07:59.439
Like most of our teams are very data-driven.
00:07:59.519 --> 00:08:02.720
All the decisions are go through um and are validated on top of data.
00:08:02.800 --> 00:08:12.720
Now, if you give a wrong data that looks slightly correct, but might be wrong in a lot of ways, that does not um that would not lead to good results um if that gets in the pipeline.
00:08:12.879 --> 00:08:21.519
So we went ahead with the right audit audit publish setup uh where we firstly basically pass through the manifest, um, see what tasks we want to build.
00:08:21.600 --> 00:08:26.240
We will task groups per model, we'll run the model actually first in audit.
00:08:26.319 --> 00:08:27.759
We'll run tests on top of it.
00:08:27.839 --> 00:08:31.279
Um, if it passes, we replace the production model with that data.
00:08:31.439 --> 00:08:35.679
If it doesn't, we um let people know so that um it gets fixed.
00:08:35.759 --> 00:08:42.320
Um, slightly we have SLAs for all those things, um, so that we'd never have wrong data in production essentially.
00:08:43.120 --> 00:08:44.639
That's actually very interesting.
00:08:44.879 --> 00:08:56.000
Um because the write audit publish pattern is is not that I mean, to be honest, I I heard a quite a lot of people saying that they don't know about this pattern, uh, and I think it's a very useful one.
00:08:56.159 --> 00:09:02.399
So just like in a very nutshell, um, can you just tell us a little bit more on how like what is exactly this pattern?
00:09:02.480 --> 00:09:03.440
Like how does it work?
00:09:03.919 --> 00:09:18.159
Um so basically right audit publish, or you might also have heard of like Broogle deployment, where you firstly deploy a thing, um, you test things, you test everything on top of it, and then if everything works fine on that audit instance or like that audit model.
00:09:18.240 --> 00:09:21.600
So in case of dbt, we'll suppose build a bookings model.
00:09:21.759 --> 00:09:25.519
Uh, we'll run XYZ text on top of it that we've defined in DBT.
00:09:25.600 --> 00:09:28.480
If all tests tests run fine, we know the data is correct.
00:09:28.559 --> 00:09:34.320
Um, and then we replace the production model that is user-facing and everyone in our company uses with that specific model.
00:09:34.480 --> 00:09:37.360
If it's not, then older data continues to stay in production.
00:09:37.519 --> 00:09:45.840
Um, so basically, you write it, you audit um by running tests, and then you replace it in production uh with the correct data if your tests run for correctly.
00:09:46.000 --> 00:09:52.480
Okay, so so this pattern is really about avoiding having like some bad data in your production environment.
00:09:52.799 --> 00:09:53.360
Okay, okay.
00:09:53.519 --> 00:10:06.240
Uh it's uh actually, uh I mean, like you know, for the people that are listening to us and that don't know about this pattern, I highly recommend you to look into it because it's it's not a I mean it's not a hard pattern to uh to put in place, but it's a very efficient one.
00:10:06.960 --> 00:10:08.399
Um great.
00:10:08.559 --> 00:10:17.039
Uh but now like how how I mean I'm curious to know how do you interact with dbt from Airflow?
00:10:17.440 --> 00:10:22.240
Sorry, from Airflow, because you know with Airflow, you have different ways for interacting with dbt.
00:10:22.559 --> 00:10:26.000
Uh you can go with the bash operator and just run some dbt commands.
00:10:26.159 --> 00:10:29.360
Uh you can also use like the dbt provider.
00:10:29.519 --> 00:10:34.080
Um and there is also another way, but uh can you tell us more about your way?
00:10:34.159 --> 00:10:36.080
Like how do you interact with with dbt?
00:10:36.720 --> 00:10:40.559
Uh we use like a slightly different way, which many people might not agree with.
00:10:40.639 --> 00:10:54.320
We use like a Python operator uh because uh we have like a certain level of customizations we have there, and we have like some alerting, some things that we send our send to our warehouse to see, okay, this task ran is much time.
00:10:54.399 --> 00:10:54.960
Did it fail?
00:10:55.120 --> 00:10:57.759
If the test failed, how many um rows it failed for?
00:10:57.919 --> 00:11:01.600
We have a lot of logging and um auditing for all those data.
00:11:01.759 --> 00:11:04.080
Um, so that is why we went ahead with a Python operator.
00:11:04.159 --> 00:11:06.320
It like it allows you to do a lot more things.
00:11:06.480 --> 00:11:09.279
Uh and that's in the main um five.
00:11:09.360 --> 00:11:11.360
We went ahead with Python operator, yes.
00:11:11.840 --> 00:11:19.360
Okay, and so this this Python operator, like is it um so it's like like so so you have this this custom operator, is that correct?
00:11:19.600 --> 00:11:20.879
Where you interact with dbt?
00:11:21.200 --> 00:11:32.639
Okay, but does it give you like the um because you know, like one issue that people have with uh with dbt and airflow is that they don't have the full picture of their of their uh dbt model, right?
00:11:32.720 --> 00:11:43.679
Like when they look at their tag, they just see like one task, and then uh you know you have your dbt model behind this task, but you you don't see you know the the staging models and so on.
00:11:44.000 --> 00:11:46.159
Uh is it something that you solved?
00:11:46.559 --> 00:12:04.799
Yeah, so basically it's the same as like we build out the full lineage, we read the manifest, um, and then every um different model is a task, and within like it's the task group, and then audit is one task, tests are another task um that run after audit, and then we have a replace task.
00:12:05.200 --> 00:12:05.759
Oh, okay.
00:12:05.840 --> 00:12:07.759
And so how this is generated?
00:12:07.919 --> 00:12:09.120
Like, do you have like a task?
00:12:09.600 --> 00:12:20.159
We read the manifest file, and then we have like uh kind of like everything that dbt builds, uh, we kind of build it in how we have all those rules of partitions and all of those written custom.
00:12:20.480 --> 00:12:23.360
Okay, and it is uh like what reads the manifest file?
00:12:23.440 --> 00:12:25.120
Is it a script like a Python script?
00:12:25.360 --> 00:12:28.240
Uh no, just the dbt manifest file um that dbt creates.
00:12:28.320 --> 00:12:43.679
So dbt, whenever you run a dbt compile uh or a dbt um run, so it'll create a manifest file which will give you an information of all the different models, all the tasks, um the actual code that should be running, what are the columns, what are the column descriptions, um, if there's partition set.
00:12:43.919 --> 00:12:49.279
So it'll give you a holistic information of what needs to be done to them, like what transformation needs to be made to that model.
00:12:49.600 --> 00:12:53.440
Um, basically use all of that logic, apply it, and then we run it.
00:12:53.600 --> 00:12:55.440
So we run it in two different projects.
00:12:55.519 --> 00:13:04.879
So we run audit in one specific GCP project and then replace in another GCP project because of some governance issue where we did not want like the complete thing to be in one project.
00:13:04.960 --> 00:13:09.919
Uh, we just wanted the final output data that we want to show to users in that specific project.
00:13:10.320 --> 00:13:10.879
Okay, okay.
00:13:11.120 --> 00:13:14.080
And just to be clear, like what reads this manifest file?
00:13:14.159 --> 00:13:16.159
Is it like a script, a Python script?
00:13:16.240 --> 00:13:19.200
Is it a task that generates the DAG with the manifest?
00:13:19.840 --> 00:13:21.440
Um it's a just a JSON file.
00:13:21.600 --> 00:13:30.240
Um, manifest file that's like the dbt um generated manifest file, it's a JSON file, and then we read through that and create like dynamic tasks um depending on what is there.
00:13:30.720 --> 00:13:30.799
Okay.
00:13:31.279 --> 00:13:32.879
Okay, okay, got it, got it, got it.
00:13:33.039 --> 00:13:37.440
Uh it's interesting because that your your uh implementation looks very similar to uh Cosmos.
00:13:38.080 --> 00:13:45.519
Uh I mean we discussed about that, but uh uh you you told me that you you had implemented that before even Cosmos existed.
00:13:45.840 --> 00:13:50.799
Um but like for people that don't know about Cosmos, it's an open source project, you can use it for free.
00:13:50.879 --> 00:13:55.200
Uh, I highly recommend you to look into it because you will save a lot of time.
00:13:55.360 --> 00:13:59.679
Um it's just the best way to integrate um dbt with with Airflow.
00:13:59.840 --> 00:14:04.720
You will have the full picture of your uh of your dbt models directly in Airflow, exactly as you uh explained.
00:14:06.559 --> 00:14:14.960
This episode is brought to you by Astronomer, the team behind Astro, a managed Airflow platform built for data engineering teams.
00:14:15.279 --> 00:14:19.200
From better pipelines to training models and wiring up agents.
00:14:19.360 --> 00:14:25.440
Astro handles the infrastructure upgrades and scaling so you can focus on building.
00:14:25.679 --> 00:14:27.919
Check it out at astronomer.io.
00:14:29.919 --> 00:14:31.200
Um great.
00:14:32.240 --> 00:14:43.279
Now, uh, you know, I mean, obviously with AI, I mean AI is super popular right now, but um, before AI, actually, uh many people were talking about ML, like machine learning.
00:14:43.519 --> 00:14:47.360
Um machine learning is still a thing, by the way, for people that wonder.
00:14:47.519 --> 00:14:49.679
It's still something that works well.
00:14:50.159 --> 00:14:56.159
Um I think you have some ML use cases uh that you run with Airflow.
00:14:56.320 --> 00:14:58.000
Can you tell us a little bit more about that?
00:14:58.080 --> 00:15:01.919
Because you know, for some people it's not obvious that you can use Airflow for uh ML.
00:15:02.639 --> 00:15:07.759
So can you yeah, can you tell me more about so we like have like a few different ML use cases?
00:15:07.840 --> 00:15:14.480
Some of them we run through Airflow uh specifically because it's like it's very easy to set up, it's very quick to because everyone is familiar.
00:15:14.559 --> 00:15:16.480
Um they can very easily set up things.
00:15:16.559 --> 00:15:22.480
Um, so there's different parts of ML ops um that we run on Airflow.
00:15:22.720 --> 00:15:29.440
Um so feature store, we have feast as our feature store, a lot of things that flow into feature store, the things that we want to track, um etc.
00:15:29.679 --> 00:15:35.759
Like they go through Airflow things that need to like on a scheduled interval, we send to a feature store that we want to track, that is done.
00:15:35.919 --> 00:15:43.120
Uh, for some model training um and model inference, uh, we still use Airflow as our orchestrator, that's the main orchestrator.
00:15:43.200 --> 00:15:48.399
Um, everything runs through there, like models, um, our models are very easy to train, very quick to run.
00:15:48.559 --> 00:15:51.200
Um, then we also have like testing on top of it, right?
00:15:51.279 --> 00:15:56.080
So you have data derived um and feature drift that can happen when you run all these airflow models.
00:15:56.240 --> 00:15:59.200
So we have very extensive metric tracking there.
00:15:59.279 --> 00:16:05.360
Um, and then we do calculations on top of it to see, okay, if there was any change, if we want to retune the model, etc.
00:16:05.600 --> 00:16:12.080
Um, and then because most models are then retrained at X interval, Airflow makes it very easy to kind of do it on a schedule.
00:16:12.240 --> 00:16:23.919
Um, same for inference for things that we have uh static um static recommendations for um or like static output for where like you just generate it one, like you generate a candidate set.
00:16:24.000 --> 00:16:26.320
It's again very easy to do it 130 through Airflow.
00:16:26.480 --> 00:16:32.960
So that is um there are some those are some ways where we um kind of use Airflow in our MLM system.
00:16:33.600 --> 00:16:34.720
This is super interesting.
00:16:34.799 --> 00:16:48.080
Um, I guess like when you say uh training your machine learning models, like um do you actually use to train your model or do you do you have like a different tool that you know to to train your model on on the data?
00:16:48.399 --> 00:16:55.120
So for most of these use cases, like the small recommendation ranking, all these things, we do like all the training on Airflow itself.
00:16:55.600 --> 00:16:56.879
Okay, okay, got it.
00:16:57.120 --> 00:17:05.279
And uh do you have any like did you have any performance issues related to uh running Airflow with ML?
00:17:05.599 --> 00:17:11.279
Uh so not specifically this ML likes for like the things that we run right now, it's fine.
00:17:11.440 --> 00:17:29.359
Uh, but like for some bigger things, uh, we have like realized that memory and all those things, like it gets very expensive to do it um on our current, and it's easier to just like have another machine where you do all these things um and then push it uh rather than do it on Airflow because like continuously having that resource did not make sense.
00:17:29.599 --> 00:17:30.000
Right.
00:17:30.240 --> 00:17:31.200
Okay, okay.
00:17:31.440 --> 00:17:39.680
Um okay, I guess like you know, when things go well, this is great, but as we know, right, in our world, many things go wrong first.
00:17:40.240 --> 00:17:45.839
Uh how do you know that things go wrong in your uh you know in your Airflow instance?
00:17:46.000 --> 00:17:53.279
Like, do you use the usual um you know alerting mechanism in Airflow, or do you did you create your own framework on top of Airflow?
00:17:53.519 --> 00:17:59.519
Like, can you tell us more about how do you ensure that whenever something goes wrong in your Airflow instance, you know about it?
00:18:00.000 --> 00:18:04.880
So there's a few different ways in we in which we track um how our Airflow instance is doing.
00:18:05.039 --> 00:18:12.880
Um so first is we just have moved a lot of it to Grafana, where we have all of the metrics that we care about and the things that we care about.
00:18:13.039 --> 00:18:17.039
Um, and also task level metrics where okay, it failed how many times?
00:18:17.200 --> 00:18:22.000
What were like if it was like a general issue, who was the person who changed it, all those things.
00:18:22.079 --> 00:18:25.599
So that is our central thing where we um track everything through Grafana.
00:18:25.680 --> 00:18:29.200
There's a few alerts built on top of Grafana um that give us information.
00:18:29.359 --> 00:18:31.920
Um the second way is at like a task level.
00:18:32.079 --> 00:18:36.079
So uh we have like all the failure hooks, um success hooks.
00:18:36.160 --> 00:18:44.000
So all for all failure hooks, we have uh we've made sure that okay, so we have owners in our DAGs, uh, we have tags in our DAGs for um teams.
00:18:44.319 --> 00:18:48.720
We also check through GitHub who was the last person who changed that file.
00:18:48.880 --> 00:19:00.720
So uh whenever uh a specific task fails, um, it'll send an alert on Slack that this is a person who last changed it, this is the commit that last changed it, this is the log link, and um, these are like the errors.
00:19:00.799 --> 00:19:05.680
So it's very very easy to then like kind of converse on Slack and then okay, who is fixing it?
00:19:05.839 --> 00:19:06.559
Can you fix it?
00:19:06.720 --> 00:19:08.880
Also, very easy to kind of have all the links there.
00:19:09.039 --> 00:19:13.519
So you have the commit link, so you can just directly see, okay, do you do I want to revert this PR?
00:19:13.759 --> 00:19:17.039
Or um if it's something else, and you can directly go check the logs.
00:19:17.599 --> 00:19:19.200
Wait, this is super interesting.
00:19:19.359 --> 00:19:22.559
Like, like how how how does that work exactly?
00:19:22.640 --> 00:19:26.000
Like, like like how do you how do you do that?
00:19:26.319 --> 00:19:31.200
Like if you can enter a little bit more in the technical details, like how how did you implement this?
00:19:31.759 --> 00:19:37.440
So um few like firstly, like if a task fails, all of the tasks will go to Slack.
00:19:37.599 --> 00:19:39.119
Um that's like a failure hook.
00:19:39.359 --> 00:19:39.599
Sorry.
00:19:39.759 --> 00:19:39.839
Okay.
00:19:40.640 --> 00:19:41.759
No, no, go ahead, go ahead, sorry.
00:19:42.240 --> 00:19:43.839
Um, so there's a failure hook there.
00:19:44.000 --> 00:19:46.880
Now on the failure hook, we have written a specific function.
00:19:47.039 --> 00:19:51.680
Now I know uh what DAG is failing, so I know what kind of file is failing.
00:19:51.839 --> 00:20:02.480
Um on GitHub because it's linked one-on-one, uh, because I can from Airflow get like what file um is the actual task failing in, what is the task and all those other information who's the author?
00:20:02.640 --> 00:20:05.200
Um, so I can then go back checking GitHub.
00:20:05.279 --> 00:20:07.680
Okay, what is the last commit on this particular file?
00:20:07.759 --> 00:20:11.920
Um, and I know my teammates like Slack IDs and emails and all those things.
00:20:12.000 --> 00:20:12.079
Okay.
00:20:12.319 --> 00:20:14.480
So just stitch all of it together.
00:20:14.559 --> 00:20:17.680
Um, you get Airflow logs, you get GitHub uh commit URLs.
00:20:17.759 --> 00:20:20.640
So yeah, um, and then you just stitch it and send it to Slack.
00:20:20.960 --> 00:20:22.559
And all of that is automated?
00:20:22.880 --> 00:20:23.920
It's all automated.
00:20:24.000 --> 00:20:30.000
Um, so because like user emails and all those things you can directly get, um, and then um Git commits, etc.
00:20:30.400 --> 00:20:33.519
With GitHub API, it's very easy to um get all this information.
00:20:34.319 --> 00:20:34.720
Awesome.
00:20:34.880 --> 00:20:35.359
That's great.
00:20:35.440 --> 00:20:38.720
That's the first one that I hear uh that I hear that this is this is cool.
00:20:39.119 --> 00:21:09.599
Um so um I think that we discussed uh, you know, I mean uh previously, and I think is very common for people to have, maybe a little bit less today, but um I think you had some zombie tasks maybe in the past, I'm not sure if you still have uh but can you tell us more about how did you deal with this painful issue that I know many people had or at least had once uh with their with their own.
00:21:10.160 --> 00:21:11.519
That that was a very long day.
00:21:12.240 --> 00:21:13.920
Hang on, then again.
00:21:14.079 --> 00:21:17.599
So basically, uh we had like a few uh zombie tasks there.
00:21:17.759 --> 00:21:20.480
Um usually at Lowell Celery would kick them out.
00:21:20.559 --> 00:21:24.079
Um, it is all okay, but that specific instance, sorry.
00:21:24.640 --> 00:21:32.960
So sorry, very quickly, can you define what what a zombie task is for those who um so a zombie task is basically a task where like um it has ended.
00:21:33.039 --> 00:21:38.559
Um on celery thinks it hasn't ended, but it has actually ended, it's not really running, but celery thinks it's running.
00:21:38.640 --> 00:21:43.359
Um so there's a misma mismatch in status, um, and then it causes a lot of issues.
00:21:43.440 --> 00:21:51.119
So for us specifically, what it caused was we weren't able to identify uh for a you know that it's running and it kept running in the background, right?
00:21:51.200 --> 00:21:55.359
Um not because it did not happen to us before, so we were not tracking it that often.
00:21:55.519 --> 00:21:59.279
Okay, this is something that's running, or like this is something that we need to take care of.
00:21:59.359 --> 00:22:10.400
So the That time when it happened, what happened was it led to DV um size increasing because like it kept on adding logs, it kept on adding run states, etc., in the DV.
00:22:10.480 --> 00:22:14.960
Now, DV very quickly, in like a few hours, um, changed the scale.
00:22:15.039 --> 00:22:18.319
Um, and then it was causing locking issues on the DV.
00:22:18.400 --> 00:22:23.440
Uh, and because of locking issues, no new tasks were getting um added to the queue.
00:22:23.599 --> 00:22:25.680
Nothing was processing, it was all stuck.
00:22:26.000 --> 00:22:35.359
And then it could also not cancel because even through API, even through um UI, because it doesn't know that the task is there, because according to it, the task is not there.
00:22:35.519 --> 00:22:37.680
Um, so yeah, that was a big pain.
00:22:37.839 --> 00:22:41.440
Now we have added like a bunch of alerting on top of it also.
00:22:41.599 --> 00:22:50.319
Uh where now we track okay, what is the status, what is the task, how much longer has been it running, um, like memory and all those things.
00:22:50.400 --> 00:22:59.839
So, like the Grafana and all those alerting and metrics came after this because we realized we need to build more exhaustive systems um to be able to monitor um air.
00:23:00.319 --> 00:23:06.880
Okay, so so what allows you to spot those zombie tasks today are the alerts that you defined in Grafana.
00:23:07.519 --> 00:23:10.319
Grafana plus also just um from the API.
00:23:10.400 --> 00:23:13.200
So um in the Airflow API, you get like a lot of information, right?
00:23:13.279 --> 00:23:15.759
Um, like how much tasks it's running, like all of those things.
00:23:15.839 --> 00:23:18.559
So we use that also in um addition to Grafana.
00:23:18.960 --> 00:23:19.440
That's cool.
00:23:19.519 --> 00:23:24.160
Yeah, I mean I'm sure that people will uh will uh reflect a lot on what you what you just said.
00:23:24.640 --> 00:23:31.839
Um all right, one last thing because I have to ask you, uh, you know, um AI is everywhere.
00:23:32.079 --> 00:23:32.319
Yes.
00:23:32.480 --> 00:23:38.400
So I have to ask you, uh, how do you use AI today in your in your um in your company?
00:23:38.720 --> 00:23:40.559
Just if you can tell us more about that.
00:23:40.960 --> 00:23:41.359
Yes.
00:23:41.599 --> 00:23:44.559
So we use AI very extensively in our company now.
00:23:44.640 --> 00:23:46.960
Like a lot of non-tech people also create commits.
00:23:47.039 --> 00:23:51.200
Um, they have a lot of changes making the developer velocity, has changed a lot.
00:23:51.359 --> 00:24:02.079
Specifically in the data team, um, we try to augment it, but also because data is a very sensitive thing, where like if AI hallucinates, there is a lot of issues, so you need to double check everything.
00:24:02.240 --> 00:24:08.319
Um, we were building the self-serve um system, and then it was hallucinating columns that did not exist.
00:24:08.400 --> 00:24:11.119
So you need to be also careful, careful about it.
00:24:11.359 --> 00:24:16.319
Uh specifically, um coming to like coding um in data.
00:24:16.480 --> 00:24:24.480
Um, in the start, we looked at things um where like people would create um PRs with um plot code or like any other agent.
00:24:24.640 --> 00:24:29.519
Uh, but the thing was that was it was correct Python code, but it was not very airflow.
00:24:29.680 --> 00:24:36.160
Um, like it would not follow it, like it would look like it's a fine Python code, but it's like it doesn't follow airflow conventions.
00:24:36.240 --> 00:24:40.720
Like there would be things like where like um connections are created outside of tasks.
00:24:40.799 --> 00:24:44.480
Um every time a DAG or bar, it'll create a connection.
00:24:44.720 --> 00:24:52.799
Um so we realized we already have a lot of uh systems and um recommendations in place when you write DAGs.
00:24:53.039 --> 00:24:55.759
There's always like this things like um good recommendation.
00:24:55.839 --> 00:24:58.720
We also have internal things that this is the pattern that we follow.
00:24:58.880 --> 00:25:05.519
Um so we took a two-step process here where firstly we encoded all of this as like a skill file.
00:25:05.599 --> 00:25:13.519
Like if you want to develop, you have these um resources that this is the pattern that we follow, this these are the guidelines, patterns, anti-patterns, everything.
00:25:13.599 --> 00:25:16.400
Like we try to make it as exhaustive as we can.
00:25:16.559 --> 00:25:20.400
Um, so that whenever you're developing code, it goes in the right direction.
00:25:20.640 --> 00:25:32.880
But actually, I missed this, but before that, you need to clean up your current current code because if it learns wrong patterns from your current code or current DAGs, like it's it's very difficult.
00:25:33.039 --> 00:25:36.160
Um, it'll always create like not great code.
00:25:36.400 --> 00:25:37.519
Uh that's right.
00:25:37.759 --> 00:25:44.240
So so we cleaned it up, we created like a new system, and then we also have like a reviewer bot on top of it.
00:25:44.319 --> 00:25:48.960
So there's a reviewer bot that reviews every PR in the company, tech, not like everything.
00:25:49.119 --> 00:25:58.799
Uh, so we added specific uh instructions there that even if someone misses anything, even when they're building it through the skill, it's able to catch um these smaller things now.
00:25:58.960 --> 00:26:02.079
Things like memory leaks are very difficult to spot.
00:26:02.319 --> 00:26:05.119
So all those things you have added, like specific um edition.
00:26:05.440 --> 00:26:06.559
It's still a work in progress.
00:26:06.720 --> 00:26:13.519
There's still sometimes where we miss things and like uh things need to be modified, but again, like it's just an iterative process.
00:26:13.680 --> 00:26:16.319
Like if something goes wrong, we try to fix it the next time.
00:26:16.799 --> 00:26:21.279
But this is another AI that we've used the yes.
00:26:21.759 --> 00:26:22.319
Oh, okay.
00:26:22.400 --> 00:26:28.160
So it's like you have like you generate the code with AI and then you have another AI that checks whatever has been generated.
00:26:28.400 --> 00:26:29.119
Yes, yes.
00:26:29.440 --> 00:26:29.920
Okay, okay.
00:26:30.000 --> 00:26:31.680
So it's like AI as a judge, right?
00:26:31.759 --> 00:26:32.960
I mean kind of, yes.
00:26:33.119 --> 00:26:40.559
So it does like first level of um checking for things, so that it's easier for like the reviewers to then go ahead and actually win the reviewing code.
00:26:40.720 --> 00:26:43.519
Um, most of the obvious mistakes are fixed before then.
00:26:43.839 --> 00:26:44.160
Right.
00:26:44.480 --> 00:26:59.599
I think you you said something super uh super important is that um is that when you if you if you like if you decide to uh create a skill out of your code, then you better make sure that your code actually is great.
00:26:59.839 --> 00:27:12.319
Uh and I think maybe this is one of the things that AI um bring you know brought back to life is the documentation as well, like ensuring that you document your uh your code properly.
00:27:12.799 --> 00:27:18.400
Uh, you know, it it I mean today it's it's super important because obviously AI reads all of that.
00:27:19.039 --> 00:27:23.359
Um yeah, I mean I think this is uh this is super interesting.
00:27:23.440 --> 00:27:28.240
Uh this is super interesting the way that you are using uh AI and and again as you said, right?
00:27:28.400 --> 00:27:35.200
AI can uh hallucinate uh quite a lot, so you better be careful with uh whatever you generate uh with it.
00:27:35.839 --> 00:27:45.920
Well, uh I must say I'm very I'm very shocked because it's already uh over, and to be honest, that went super fast.
00:27:47.359 --> 00:27:48.559
That went super fast.
00:27:48.720 --> 00:27:52.079
Uh I'm sure that people will enjoy a lot uh this episode.
00:27:52.240 --> 00:27:56.480
You obviously explained a lot of very interesting things and uh in details.
00:27:56.559 --> 00:27:58.160
Uh so thank you so much for that.
00:27:58.319 --> 00:28:08.799
Uh but before leaving, I have to ask uh what is the best way for uh folks to get in touch with you if they want to ask questions, maybe or you know if they want to learn more about uh head out.
00:28:09.519 --> 00:28:13.039
Yeah, um, you can just uh connect with me on LinkedIn and maybe Blue Sky.
00:28:13.279 --> 00:28:16.720
I don't use it as often, but you can shoot me at my probably replacing a few days.
00:28:17.839 --> 00:28:18.960
Awesome, awesome.
00:28:19.119 --> 00:28:21.359
Well, thank you so much again, Minalini.
00:28:21.440 --> 00:28:23.279
Um, hope to see you again for another episode.
00:28:23.440 --> 00:28:24.079
Who knows?
00:28:24.240 --> 00:28:26.559
Uh but otherwise, I wish you the best.
00:28:26.720 --> 00:28:27.920
Take care and uh see you.
00:28:28.640 --> 00:28:28.960
You do.
00:28:29.200 --> 00:28:29.920
Thank you so much for having me.
00:28:30.000 --> 00:28:30.880
It was a very nice chat.
00:28:31.279 --> 00:28:31.519
Thank you.
00:28:31.599 --> 00:28:31.839
See you.
00:28:32.799 --> 00:28:32.960
Bye.
00:28:34.400 --> 00:28:37.359
Thank you for listening to the Data Flowcast.
00:28:37.519 --> 00:28:40.960
Check the show notes for links to everything mentioned today.
00:28:41.119 --> 00:28:45.599
If you're enjoying the podcast, subscribe so you never miss an episode.
00:28:46.000 --> 00:28:48.079
And leave us a five-star review.
00:00:00.239 --> 00:00:07.519
You're listening to the Data Flowcast, the podcast about HD Airflow and the world of data and AI around it.
00:00:07.679 --> 00:00:14.560
We are excited to invite you to orchestrate every day for a free online conference on September 16th.
00:00:14.960 --> 00:00:17.760
Engineering data from Lyft RAM and WIC.
00:00:18.000 --> 00:00:28.320
I don't care how they are shipping production AI workflows on AI core for the new Astronomer Certified AI Orchestration Fundamental exam.
00:00:28.559 --> 00:00:31.600
Attendees get a discount code for a free certification.
00:00:32.000 --> 00:00:34.159
A$150 value.
00:00:34.399 --> 00:00:39.600
Go to astronomer.link slash data minus flowcast minus OE.
00:00:39.759 --> 00:00:41.119
Link in the show notes.
00:00:41.359 --> 00:00:43.439
Now let's get into the show.
00:00:43.759 --> 00:00:44.320
Alright.
00:00:44.479 --> 00:00:45.359
Hi everybody.
00:00:45.439 --> 00:00:45.920
Welcome.
00:00:46.079 --> 00:00:47.840
Welcome to the Data Flowcast.
00:00:47.920 --> 00:00:54.880
I'm your host, Mark Romati, and today I'm super excited because I'm joined by Marim Ralini, Data Platformat, head out.
00:00:55.359 --> 00:00:58.079
And uh first thing first, how are you doing today?
00:00:58.399 --> 00:01:00.479
I'm doing great, very excited to be here.
00:01:00.560 --> 00:01:05.120
Um, but I've like love looking forward to having this conversation with you again.
00:01:05.519 --> 00:01:08.239
Same, likewise, I'm super happy to have you here.
00:01:08.400 --> 00:01:23.439
Um, so what's gonna what we're gonna discover in today's podcast is really interesting because we'll have a quick tour of how Head Out, which is a travel platform, um, how they run DBT, MLOps, alerting and for detection in Airflow.
00:01:23.599 --> 00:01:28.560
So basically, we're gonna have this tour of the use cases that they have that they run in Airflow.
00:01:28.799 --> 00:01:34.719
And uh maybe Rinalini will tell us more about some technical details and so on, but we'll see.
00:01:34.959 --> 00:01:35.840
Uh okay.
00:01:36.000 --> 00:01:41.920
So, first thing first, can you introduce yourself and tell us what you do at uh Head Out?
00:01:42.400 --> 00:01:42.480
Okay.
00:01:42.879 --> 00:01:45.920
And maybe maybe a little bit of your background as well, if you want.
00:01:46.400 --> 00:01:46.640
Okay.
00:01:46.719 --> 00:01:48.239
Um, so hi, I'm Renalini.
00:01:48.480 --> 00:01:50.959
Um, I am a data platform engineer here at Head Out.
00:01:51.040 --> 00:01:52.560
Head Out is basically a travel platform.
00:01:52.640 --> 00:01:54.959
We try to bring real life experiences to people.
00:01:55.200 --> 00:01:59.120
Um, so here I work on a big breadth of things.
00:01:59.200 --> 00:02:01.120
That's the best part of my job, I feel like.
00:02:01.200 --> 00:02:04.319
Um, I get because it's a lean team, I get to work on everything.
00:02:04.480 --> 00:02:07.840
So I work from DVT to Airflow to MFOps to governance.
00:02:07.920 --> 00:02:08.960
So all of it end-to-end.
00:02:09.280 --> 00:02:10.319
Yeah, that's mostly it.
00:02:10.400 --> 00:02:11.680
Um, I've been here for four years.
00:02:11.759 --> 00:02:13.520
I enjoy it um a lot.
00:02:14.240 --> 00:02:15.199
That's great.
00:02:15.439 --> 00:02:20.560
Um, can you can you tell me more about like what what what exactly is a data platform engineer?
00:02:20.639 --> 00:02:22.240
I guess like your data platform engineer.
00:02:22.319 --> 00:02:27.199
Like, is it is it like your data engineer or just like you have also the DevOps part as well?
00:02:27.280 --> 00:02:30.000
Like what Yeah, it's a combination of both, I would say.
00:02:30.240 --> 00:02:34.479
So um I mostly handle the infra and enable people to do their best work.
00:02:34.639 --> 00:02:46.879
So it includes a bunch of toolings to make sure um people have the best tools available to do their work, um, and in touch also with like um the infra bits, um everything is optimized, everything is running fine.
00:02:46.960 --> 00:02:51.039
Um, and then also exploring newer tools, making sure everything is working fine.
00:02:51.120 --> 00:02:53.360
Um, then also involves a bunch of DevOps.
00:02:53.520 --> 00:02:56.639
We maintain a lot of services internally um for data.
00:02:56.800 --> 00:02:58.719
So yeah, a little bit of everything.
00:02:59.120 --> 00:02:59.680
Okay, okay.
00:02:59.759 --> 00:03:01.840
So you are like a full-stack data engineer.
00:03:02.159 --> 00:03:04.080
Yeah, I'm a jack of all trades.
00:03:04.479 --> 00:03:06.240
Okay, okay, that's interesting.
00:03:06.400 --> 00:03:13.439
Uh, just out of curiosity, um, I guess like, do you do you work with uh um data analysts, data scientists, and so on?
00:03:13.520 --> 00:03:16.400
Like, do you work with different teams like that in your uh yeah?
00:03:16.639 --> 00:03:20.960
Yeah, so um how it works in our team is basically we have a hub and spokes model.
00:03:21.039 --> 00:03:29.360
So the data platform team sets in the center, and all the other data scientists, data um analysts, all these people sit at different spokes, different in within different pods.
00:03:29.520 --> 00:03:36.319
Um so our work is to like kind of um make sure that they are doing their best work, um, help them in whatever way.
00:03:36.479 --> 00:03:38.960
Um, because it's a very lean team, everyone does everything.
00:03:39.039 --> 00:03:42.479
So our all of our analysts, all of our scientists also write their own data models.
00:03:42.560 --> 00:03:50.960
So we need to make sure that we have good enough standards and good enough documentation so that they write code and like it's a well-enough code, um, even like optimization for things.
00:03:51.039 --> 00:03:52.800
So yeah, it's like it sits in the middle.
00:03:52.960 --> 00:03:56.080
We work a lot with um and we try to make their lives easier.
00:03:56.719 --> 00:03:57.520
That's amazing.
00:03:57.680 --> 00:03:58.000
Okay.
00:03:58.319 --> 00:04:16.800
Um I guess the this was not uh uh um in the in the in the questions that we we discussed before, but I I I I'm just wondering like uh is Airflow was already there when you joined Head Out, or uh like did you decide to have Airflow in your company?
00:04:17.519 --> 00:04:23.040
So it was already there when I was there, but like we were doing very smaller things back then.
00:04:23.199 --> 00:04:26.319
Um it has now increased to do a lot more things.
00:04:26.639 --> 00:04:26.959
Oh okay.
00:04:27.120 --> 00:04:38.480
I was just wondering like if you uh you know if you had to go through uh the uh you know the install of Airflow and and also like maybe why did you choose Airflow and so on, but uh it's good to know that it was already there.
00:04:38.639 --> 00:04:42.000
Um okay, I guess like uh another question to you.
00:04:42.079 --> 00:04:46.000
So, what kind of data challenges uh come with running a travel platform?
00:04:46.160 --> 00:04:51.600
Um I think that's actually the first time that I interview someone that it is working for uh a travel platform.
00:04:51.759 --> 00:05:01.199
So I'm really interested to know more about that, uh, the data challenges that you have, and also where does Airflow sit in your uh in your stack?
00:05:01.680 --> 00:05:02.160
Got it.
00:05:02.319 --> 00:05:13.920
Okay, so um we already have like all the issues that come within a general data team of volume, data being wrong, or like the formatting being wrong, different kinds of file formats, different kinds of sources.
00:05:14.079 --> 00:05:15.199
So all of that is there.
00:05:15.360 --> 00:05:36.160
But specifically when you come to travel, there's like a few different things um that um are of a slightly bigger issue for us, especially things like inventory, where it changes very quickly, you need to be on top of things, you need to um run a lot of these things very quickly because a lot of your forecasting, a lot of your how much do you purchase, how much do you um not purchase, how much of it is going out.
00:05:36.240 --> 00:05:38.639
Um a lot of revenue sits on top of that, right?
00:05:38.800 --> 00:05:40.560
Um so inventory is a big thing.
00:05:40.720 --> 00:05:52.079
Um, then um for real-time use case, especially like fraud, uh, you need to do a lot of real-time analytics, you need to do a lot of graph analytics, um, you need to process a lot of things very quickly in real time.
00:05:52.319 --> 00:05:53.600
Uh, so that is one thing.
00:05:53.759 --> 00:06:06.160
We also have a very big performance marketing heavy um domain travel, uh, where like a lot of our um like we have a lot of our customers coming in from campaigns um that we run online.
00:06:06.319 --> 00:06:14.319
So that is a big data uh for us to manage um to maintain attribution um so that your money is actually going in the correct place where it should be going.
00:06:14.480 --> 00:06:20.879
So those are some things um that are slightly different, like slightly different problem statements uh from like a general industry standpoint.
00:06:21.040 --> 00:06:25.439
Um, Airflow sits at like, imagine like a nervous system of it all.
00:06:25.519 --> 00:06:27.279
It's basically anywhere that we need.
00:06:27.360 --> 00:06:34.959
So all the use cases that I told before, um, anything that we need to run, also like DVT, all the transformations happen through Airflow.
00:06:35.120 --> 00:06:45.199
Like it's a tie-in for every kind of different system that we have from ingestion to kind of um serving some data to transformation, all of it just sits in Airflow.
00:06:45.839 --> 00:06:46.160
Okay.
00:06:46.639 --> 00:06:49.600
Um did you mention real time at some point?
00:06:49.920 --> 00:06:50.560
Yes.
00:06:51.040 --> 00:06:55.279
So how do you use Airflow with some real-time?
00:06:56.000 --> 00:07:00.000
So real-time-ish orcha, I gotcha.
00:07:00.319 --> 00:07:07.360
Um for some things you need to run it like at a two, three-minute interval, maybe a minute interval, five-minute interval, those.
00:07:07.519 --> 00:07:11.920
Um for more real-time things we use Flink, um, but like for real-time-ish things.
00:07:12.000 --> 00:07:14.319
Um, no, that that's interesting.
00:07:14.480 --> 00:07:24.959
Um, yeah, I mean, I I was just wondering because like like you're not the first person that I hear about uh real-time use cases, and and also it was more like okay, two, two, two, three minutes.
00:07:25.439 --> 00:07:28.399
Um you also mentioned dbt.
00:07:28.639 --> 00:07:33.360
I mean, obviously dbt is a very, very popular um tool uh today.
00:07:33.680 --> 00:07:39.040
But can you tell us more about like what is your dbt setup with Airflow?
00:07:39.120 --> 00:07:42.480
Um, and uh you know how you uh interact with it.
00:07:42.959 --> 00:07:43.519
Got it.
00:07:43.680 --> 00:07:48.079
Um so our uh setup in dbt is a write audit publish setup.
00:07:48.160 --> 00:07:57.519
Um we basically believe in that um stale data, slightly stale data is better than wrong data because a lot of our decisions are taken because of from data.
00:07:57.600 --> 00:07:59.439
Like most of our teams are very data-driven.
00:07:59.519 --> 00:08:02.720
All the decisions are go through um and are validated on top of data.
00:08:02.800 --> 00:08:12.720
Now, if you give a wrong data that looks slightly correct, but might be wrong in a lot of ways, that does not um that would not lead to good results um if that gets in the pipeline.
00:08:12.879 --> 00:08:21.519
So we went ahead with the right audit audit publish setup uh where we firstly basically pass through the manifest, um, see what tasks we want to build.
00:08:21.600 --> 00:08:26.240
We will task groups per model, we'll run the model actually first in audit.
00:08:26.319 --> 00:08:27.759
We'll run tests on top of it.
00:08:27.839 --> 00:08:31.279
Um, if it passes, we replace the production model with that data.
00:08:31.439 --> 00:08:35.679
If it doesn't, we um let people know so that um it gets fixed.
00:08:35.759 --> 00:08:42.320
Um, slightly we have SLAs for all those things, um, so that we'd never have wrong data in production essentially.
00:08:43.120 --> 00:08:44.639
That's actually very interesting.
00:08:44.879 --> 00:08:56.000
Um because the write audit publish pattern is is not that I mean, to be honest, I I heard a quite a lot of people saying that they don't know about this pattern, uh, and I think it's a very useful one.
00:08:56.159 --> 00:09:02.399
So just like in a very nutshell, um, can you just tell us a little bit more on how like what is exactly this pattern?
00:09:02.480 --> 00:09:03.440
Like how does it work?
00:09:03.919 --> 00:09:18.159
Um so basically right audit publish, or you might also have heard of like Broogle deployment, where you firstly deploy a thing, um, you test things, you test everything on top of it, and then if everything works fine on that audit instance or like that audit model.
00:09:18.240 --> 00:09:21.600
So in case of dbt, we'll suppose build a bookings model.
00:09:21.759 --> 00:09:25.519
Uh, we'll run XYZ text on top of it that we've defined in DBT.
00:09:25.600 --> 00:09:28.480
If all tests tests run fine, we know the data is correct.
00:09:28.559 --> 00:09:34.320
Um, and then we replace the production model that is user-facing and everyone in our company uses with that specific model.
00:09:34.480 --> 00:09:37.360
If it's not, then older data continues to stay in production.
00:09:37.519 --> 00:09:45.840
Um, so basically, you write it, you audit um by running tests, and then you replace it in production uh with the correct data if your tests run for correctly.
00:09:46.000 --> 00:09:52.480
Okay, so so this pattern is really about avoiding having like some bad data in your production environment.
00:09:52.799 --> 00:09:53.360
Okay, okay.
00:09:53.519 --> 00:10:06.240
Uh it's uh actually, uh I mean, like you know, for the people that are listening to us and that don't know about this pattern, I highly recommend you to look into it because it's it's not a I mean it's not a hard pattern to uh to put in place, but it's a very efficient one.
00:10:06.960 --> 00:10:08.399
Um great.
00:10:08.559 --> 00:10:17.039
Uh but now like how how I mean I'm curious to know how do you interact with dbt from Airflow?
00:10:17.440 --> 00:10:22.240
Sorry, from Airflow, because you know with Airflow, you have different ways for interacting with dbt.
00:10:22.559 --> 00:10:26.000
Uh you can go with the bash operator and just run some dbt commands.
00:10:26.159 --> 00:10:29.360
Uh you can also use like the dbt provider.
00:10:29.519 --> 00:10:34.080
Um and there is also another way, but uh can you tell us more about your way?
00:10:34.159 --> 00:10:36.080
Like how do you interact with with dbt?
00:10:36.720 --> 00:10:40.559
Uh we use like a slightly different way, which many people might not agree with.
00:10:40.639 --> 00:10:54.320
We use like a Python operator uh because uh we have like a certain level of customizations we have there, and we have like some alerting, some things that we send our send to our warehouse to see, okay, this task ran is much time.
00:10:54.399 --> 00:10:54.960
Did it fail?
00:10:55.120 --> 00:10:57.759
If the test failed, how many um rows it failed for?
00:10:57.919 --> 00:11:01.600
We have a lot of logging and um auditing for all those data.
00:11:01.759 --> 00:11:04.080
Um, so that is why we went ahead with a Python operator.
00:11:04.159 --> 00:11:06.320
It like it allows you to do a lot more things.
00:11:06.480 --> 00:11:09.279
Uh and that's in the main um five.
00:11:09.360 --> 00:11:11.360
We went ahead with Python operator, yes.
00:11:11.840 --> 00:11:19.360
Okay, and so this this Python operator, like is it um so it's like like so so you have this this custom operator, is that correct?
00:11:19.600 --> 00:11:20.879
Where you interact with dbt?
00:11:21.200 --> 00:11:32.639
Okay, but does it give you like the um because you know, like one issue that people have with uh with dbt and airflow is that they don't have the full picture of their of their uh dbt model, right?
00:11:32.720 --> 00:11:43.679
Like when they look at their tag, they just see like one task, and then uh you know you have your dbt model behind this task, but you you don't see you know the the staging models and so on.
00:11:44.000 --> 00:11:46.159
Uh is it something that you solved?
00:11:46.559 --> 00:12:04.799
Yeah, so basically it's the same as like we build out the full lineage, we read the manifest, um, and then every um different model is a task, and within like it's the task group, and then audit is one task, tests are another task um that run after audit, and then we have a replace task.
00:12:05.200 --> 00:12:05.759
Oh, okay.
00:12:05.840 --> 00:12:07.759
And so how this is generated?
00:12:07.919 --> 00:12:09.120
Like, do you have like a task?
00:12:09.600 --> 00:12:20.159
We read the manifest file, and then we have like uh kind of like everything that dbt builds, uh, we kind of build it in how we have all those rules of partitions and all of those written custom.
00:12:20.480 --> 00:12:23.360
Okay, and it is uh like what reads the manifest file?
00:12:23.440 --> 00:12:25.120
Is it a script like a Python script?
00:12:25.360 --> 00:12:28.240
Uh no, just the dbt manifest file um that dbt creates.
00:12:28.320 --> 00:12:43.679
So dbt, whenever you run a dbt compile uh or a dbt um run, so it'll create a manifest file which will give you an information of all the different models, all the tasks, um the actual code that should be running, what are the columns, what are the column descriptions, um, if there's partition set.
00:12:43.919 --> 00:12:49.279
So it'll give you a holistic information of what needs to be done to them, like what transformation needs to be made to that model.
00:12:49.600 --> 00:12:53.440
Um, basically use all of that logic, apply it, and then we run it.
00:12:53.600 --> 00:12:55.440
So we run it in two different projects.
00:12:55.519 --> 00:13:04.879
So we run audit in one specific GCP project and then replace in another GCP project because of some governance issue where we did not want like the complete thing to be in one project.
00:13:04.960 --> 00:13:09.919
Uh, we just wanted the final output data that we want to show to users in that specific project.
00:13:10.320 --> 00:13:10.879
Okay, okay.
00:13:11.120 --> 00:13:14.080
And just to be clear, like what reads this manifest file?
00:13:14.159 --> 00:13:16.159
Is it like a script, a Python script?
00:13:16.240 --> 00:13:19.200
Is it a task that generates the DAG with the manifest?
00:13:19.840 --> 00:13:21.440
Um it's a just a JSON file.
00:13:21.600 --> 00:13:30.240
Um, manifest file that's like the dbt um generated manifest file, it's a JSON file, and then we read through that and create like dynamic tasks um depending on what is there.
00:13:30.720 --> 00:13:30.799
Okay.
00:13:31.279 --> 00:13:32.879
Okay, okay, got it, got it, got it.
00:13:33.039 --> 00:13:37.440
Uh it's interesting because that your your uh implementation looks very similar to uh Cosmos.
00:13:38.080 --> 00:13:45.519
Uh I mean we discussed about that, but uh uh you you told me that you you had implemented that before even Cosmos existed.
00:13:45.840 --> 00:13:50.799
Um but like for people that don't know about Cosmos, it's an open source project, you can use it for free.
00:13:50.879 --> 00:13:55.200
Uh, I highly recommend you to look into it because you will save a lot of time.
00:13:55.360 --> 00:13:59.679
Um it's just the best way to integrate um dbt with with Airflow.
00:13:59.840 --> 00:14:04.720
You will have the full picture of your uh of your dbt models directly in Airflow, exactly as you uh explained.
00:14:06.559 --> 00:14:14.960
This episode is brought to you by Astronomer, the team behind Astro, a managed Airflow platform built for data engineering teams.
00:14:15.279 --> 00:14:19.200
From better pipelines to training models and wiring up agents.
00:14:19.360 --> 00:14:25.440
Astro handles the infrastructure upgrades and scaling so you can focus on building.
00:14:25.679 --> 00:14:27.919
Check it out at astronomer.io.
00:14:29.919 --> 00:14:31.200
Um great.
00:14:32.240 --> 00:14:43.279
Now, uh, you know, I mean, obviously with AI, I mean AI is super popular right now, but um, before AI, actually, uh many people were talking about ML, like machine learning.
00:14:43.519 --> 00:14:47.360
Um machine learning is still a thing, by the way, for people that wonder.
00:14:47.519 --> 00:14:49.679
It's still something that works well.
00:14:50.159 --> 00:14:56.159
Um I think you have some ML use cases uh that you run with Airflow.
00:14:56.320 --> 00:14:58.000
Can you tell us a little bit more about that?
00:14:58.080 --> 00:15:01.919
Because you know, for some people it's not obvious that you can use Airflow for uh ML.
00:15:02.639 --> 00:15:07.759
So can you yeah, can you tell me more about so we like have like a few different ML use cases?
00:15:07.840 --> 00:15:14.480
Some of them we run through Airflow uh specifically because it's like it's very easy to set up, it's very quick to because everyone is familiar.
00:15:14.559 --> 00:15:16.480
Um they can very easily set up things.
00:15:16.559 --> 00:15:22.480
Um, so there's different parts of ML ops um that we run on Airflow.
00:15:22.720 --> 00:15:29.440
Um so feature store, we have feast as our feature store, a lot of things that flow into feature store, the things that we want to track, um etc.
00:15:29.679 --> 00:15:35.759
Like they go through Airflow things that need to like on a scheduled interval, we send to a feature store that we want to track, that is done.
00:15:35.919 --> 00:15:43.120
Uh, for some model training um and model inference, uh, we still use Airflow as our orchestrator, that's the main orchestrator.
00:15:43.200 --> 00:15:48.399
Um, everything runs through there, like models, um, our models are very easy to train, very quick to run.
00:15:48.559 --> 00:15:51.200
Um, then we also have like testing on top of it, right?
00:15:51.279 --> 00:15:56.080
So you have data derived um and feature drift that can happen when you run all these airflow models.
00:15:56.240 --> 00:15:59.200
So we have very extensive metric tracking there.
00:15:59.279 --> 00:16:05.360
Um, and then we do calculations on top of it to see, okay, if there was any change, if we want to retune the model, etc.
00:16:05.600 --> 00:16:12.080
Um, and then because most models are then retrained at X interval, Airflow makes it very easy to kind of do it on a schedule.
00:16:12.240 --> 00:16:23.919
Um, same for inference for things that we have uh static um static recommendations for um or like static output for where like you just generate it one, like you generate a candidate set.
00:16:24.000 --> 00:16:26.320
It's again very easy to do it 130 through Airflow.
00:16:26.480 --> 00:16:32.960
So that is um there are some those are some ways where we um kind of use Airflow in our MLM system.
00:16:33.600 --> 00:16:34.720
This is super interesting.
00:16:34.799 --> 00:16:48.080
Um, I guess like when you say uh training your machine learning models, like um do you actually use to train your model or do you do you have like a different tool that you know to to train your model on on the data?
00:16:48.399 --> 00:16:55.120
So for most of these use cases, like the small recommendation ranking, all these things, we do like all the training on Airflow itself.
00:16:55.600 --> 00:16:56.879
Okay, okay, got it.
00:16:57.120 --> 00:17:05.279
And uh do you have any like did you have any performance issues related to uh running Airflow with ML?
00:17:05.599 --> 00:17:11.279
Uh so not specifically this ML likes for like the things that we run right now, it's fine.
00:17:11.440 --> 00:17:29.359
Uh, but like for some bigger things, uh, we have like realized that memory and all those things, like it gets very expensive to do it um on our current, and it's easier to just like have another machine where you do all these things um and then push it uh rather than do it on Airflow because like continuously having that resource did not make sense.
00:17:29.599 --> 00:17:30.000
Right.
00:17:30.240 --> 00:17:31.200
Okay, okay.
00:17:31.440 --> 00:17:39.680
Um okay, I guess like you know, when things go well, this is great, but as we know, right, in our world, many things go wrong first.
00:17:40.240 --> 00:17:45.839
Uh how do you know that things go wrong in your uh you know in your Airflow instance?
00:17:46.000 --> 00:17:53.279
Like, do you use the usual um you know alerting mechanism in Airflow, or do you did you create your own framework on top of Airflow?
00:17:53.519 --> 00:17:59.519
Like, can you tell us more about how do you ensure that whenever something goes wrong in your Airflow instance, you know about it?
00:18:00.000 --> 00:18:04.880
So there's a few different ways in we in which we track um how our Airflow instance is doing.
00:18:05.039 --> 00:18:12.880
Um so first is we just have moved a lot of it to Grafana, where we have all of the metrics that we care about and the things that we care about.
00:18:13.039 --> 00:18:17.039
Um, and also task level metrics where okay, it failed how many times?
00:18:17.200 --> 00:18:22.000
What were like if it was like a general issue, who was the person who changed it, all those things.
00:18:22.079 --> 00:18:25.599
So that is our central thing where we um track everything through Grafana.
00:18:25.680 --> 00:18:29.200
There's a few alerts built on top of Grafana um that give us information.
00:18:29.359 --> 00:18:31.920
Um the second way is at like a task level.
00:18:32.079 --> 00:18:36.079
So uh we have like all the failure hooks, um success hooks.
00:18:36.160 --> 00:18:44.000
So all for all failure hooks, we have uh we've made sure that okay, so we have owners in our DAGs, uh, we have tags in our DAGs for um teams.
00:18:44.319 --> 00:18:48.720
We also check through GitHub who was the last person who changed that file.
00:18:48.880 --> 00:19:00.720
So uh whenever uh a specific task fails, um, it'll send an alert on Slack that this is a person who last changed it, this is the commit that last changed it, this is the log link, and um, these are like the errors.
00:19:00.799 --> 00:19:05.680
So it's very very easy to then like kind of converse on Slack and then okay, who is fixing it?
00:19:05.839 --> 00:19:06.559
Can you fix it?
00:19:06.720 --> 00:19:08.880
Also, very easy to kind of have all the links there.
00:19:09.039 --> 00:19:13.519
So you have the commit link, so you can just directly see, okay, do you do I want to revert this PR?
00:19:13.759 --> 00:19:17.039
Or um if it's something else, and you can directly go check the logs.
00:19:17.599 --> 00:19:19.200
Wait, this is super interesting.
00:19:19.359 --> 00:19:22.559
Like, like how how how does that work exactly?
00:19:22.640 --> 00:19:26.000
Like, like like how do you how do you do that?
00:19:26.319 --> 00:19:31.200
Like if you can enter a little bit more in the technical details, like how how did you implement this?
00:19:31.759 --> 00:19:37.440
So um few like firstly, like if a task fails, all of the tasks will go to Slack.
00:19:37.599 --> 00:19:39.119
Um that's like a failure hook.
00:19:39.359 --> 00:19:39.599
Sorry.
00:19:39.759 --> 00:19:39.839
Okay.
00:19:40.640 --> 00:19:41.759
No, no, go ahead, go ahead, sorry.
00:19:42.240 --> 00:19:43.839
Um, so there's a failure hook there.
00:19:44.000 --> 00:19:46.880
Now on the failure hook, we have written a specific function.
00:19:47.039 --> 00:19:51.680
Now I know uh what DAG is failing, so I know what kind of file is failing.
00:19:51.839 --> 00:20:02.480
Um on GitHub because it's linked one-on-one, uh, because I can from Airflow get like what file um is the actual task failing in, what is the task and all those other information who's the author?
00:20:02.640 --> 00:20:05.200
Um, so I can then go back checking GitHub.
00:20:05.279 --> 00:20:07.680
Okay, what is the last commit on this particular file?
00:20:07.759 --> 00:20:11.920
Um, and I know my teammates like Slack IDs and emails and all those things.
00:20:12.000 --> 00:20:12.079
Okay.
00:20:12.319 --> 00:20:14.480
So just stitch all of it together.
00:20:14.559 --> 00:20:17.680
Um, you get Airflow logs, you get GitHub uh commit URLs.
00:20:17.759 --> 00:20:20.640
So yeah, um, and then you just stitch it and send it to Slack.
00:20:20.960 --> 00:20:22.559
And all of that is automated?
00:20:22.880 --> 00:20:23.920
It's all automated.
00:20:24.000 --> 00:20:30.000
Um, so because like user emails and all those things you can directly get, um, and then um Git commits, etc.
00:20:30.400 --> 00:20:33.519
With GitHub API, it's very easy to um get all this information.
00:20:34.319 --> 00:20:34.720
Awesome.
00:20:34.880 --> 00:20:35.359
That's great.
00:20:35.440 --> 00:20:38.720
That's the first one that I hear uh that I hear that this is this is cool.
00:20:39.119 --> 00:21:09.599
Um so um I think that we discussed uh, you know, I mean uh previously, and I think is very common for people to have, maybe a little bit less today, but um I think you had some zombie tasks maybe in the past, I'm not sure if you still have uh but can you tell us more about how did you deal with this painful issue that I know many people had or at least had once uh with their with their own.
00:21:10.160 --> 00:21:11.519
That that was a very long day.
00:21:12.240 --> 00:21:13.920
Hang on, then again.
00:21:14.079 --> 00:21:17.599
So basically, uh we had like a few uh zombie tasks there.
00:21:17.759 --> 00:21:20.480
Um usually at Lowell Celery would kick them out.
00:21:20.559 --> 00:21:24.079
Um, it is all okay, but that specific instance, sorry.
00:21:24.640 --> 00:21:32.960
So sorry, very quickly, can you define what what a zombie task is for those who um so a zombie task is basically a task where like um it has ended.
00:21:33.039 --> 00:21:38.559
Um on celery thinks it hasn't ended, but it has actually ended, it's not really running, but celery thinks it's running.
00:21:38.640 --> 00:21:43.359
Um so there's a misma mismatch in status, um, and then it causes a lot of issues.
00:21:43.440 --> 00:21:51.119
So for us specifically, what it caused was we weren't able to identify uh for a you know that it's running and it kept running in the background, right?
00:21:51.200 --> 00:21:55.359
Um not because it did not happen to us before, so we were not tracking it that often.
00:21:55.519 --> 00:21:59.279
Okay, this is something that's running, or like this is something that we need to take care of.
00:21:59.359 --> 00:22:10.400
So the That time when it happened, what happened was it led to DV um size increasing because like it kept on adding logs, it kept on adding run states, etc., in the DV.
00:22:10.480 --> 00:22:14.960
Now, DV very quickly, in like a few hours, um, changed the scale.
00:22:15.039 --> 00:22:18.319
Um, and then it was causing locking issues on the DV.
00:22:18.400 --> 00:22:23.440
Uh, and because of locking issues, no new tasks were getting um added to the queue.
00:22:23.599 --> 00:22:25.680
Nothing was processing, it was all stuck.
00:22:26.000 --> 00:22:35.359
And then it could also not cancel because even through API, even through um UI, because it doesn't know that the task is there, because according to it, the task is not there.
00:22:35.519 --> 00:22:37.680
Um, so yeah, that was a big pain.
00:22:37.839 --> 00:22:41.440
Now we have added like a bunch of alerting on top of it also.
00:22:41.599 --> 00:22:50.319
Uh where now we track okay, what is the status, what is the task, how much longer has been it running, um, like memory and all those things.
00:22:50.400 --> 00:22:59.839
So, like the Grafana and all those alerting and metrics came after this because we realized we need to build more exhaustive systems um to be able to monitor um air.
00:23:00.319 --> 00:23:06.880
Okay, so so what allows you to spot those zombie tasks today are the alerts that you defined in Grafana.
00:23:07.519 --> 00:23:10.319
Grafana plus also just um from the API.
00:23:10.400 --> 00:23:13.200
So um in the Airflow API, you get like a lot of information, right?
00:23:13.279 --> 00:23:15.759
Um, like how much tasks it's running, like all of those things.
00:23:15.839 --> 00:23:18.559
So we use that also in um addition to Grafana.
00:23:18.960 --> 00:23:19.440
That's cool.
00:23:19.519 --> 00:23:24.160
Yeah, I mean I'm sure that people will uh will uh reflect a lot on what you what you just said.
00:23:24.640 --> 00:23:31.839
Um all right, one last thing because I have to ask you, uh, you know, um AI is everywhere.
00:23:32.079 --> 00:23:32.319
Yes.
00:23:32.480 --> 00:23:38.400
So I have to ask you, uh, how do you use AI today in your in your um in your company?
00:23:38.720 --> 00:23:40.559
Just if you can tell us more about that.
00:23:40.960 --> 00:23:41.359
Yes.
00:23:41.599 --> 00:23:44.559
So we use AI very extensively in our company now.
00:23:44.640 --> 00:23:46.960
Like a lot of non-tech people also create commits.
00:23:47.039 --> 00:23:51.200
Um, they have a lot of changes making the developer velocity, has changed a lot.
00:23:51.359 --> 00:24:02.079
Specifically in the data team, um, we try to augment it, but also because data is a very sensitive thing, where like if AI hallucinates, there is a lot of issues, so you need to double check everything.
00:24:02.240 --> 00:24:08.319
Um, we were building the self-serve um system, and then it was hallucinating columns that did not exist.
00:24:08.400 --> 00:24:11.119
So you need to be also careful, careful about it.
00:24:11.359 --> 00:24:16.319
Uh specifically, um coming to like coding um in data.
00:24:16.480 --> 00:24:24.480
Um, in the start, we looked at things um where like people would create um PRs with um plot code or like any other agent.
00:24:24.640 --> 00:24:29.519
Uh, but the thing was that was it was correct Python code, but it was not very airflow.
00:24:29.680 --> 00:24:36.160
Um, like it would not follow it, like it would look like it's a fine Python code, but it's like it doesn't follow airflow conventions.
00:24:36.240 --> 00:24:40.720
Like there would be things like where like um connections are created outside of tasks.
00:24:40.799 --> 00:24:44.480
Um every time a DAG or bar, it'll create a connection.
00:24:44.720 --> 00:24:52.799
Um so we realized we already have a lot of uh systems and um recommendations in place when you write DAGs.
00:24:53.039 --> 00:24:55.759
There's always like this things like um good recommendation.
00:24:55.839 --> 00:24:58.720
We also have internal things that this is the pattern that we follow.
00:24:58.880 --> 00:25:05.519
Um so we took a two-step process here where firstly we encoded all of this as like a skill file.
00:25:05.599 --> 00:25:13.519
Like if you want to develop, you have these um resources that this is the pattern that we follow, this these are the guidelines, patterns, anti-patterns, everything.
00:25:13.599 --> 00:25:16.400
Like we try to make it as exhaustive as we can.
00:25:16.559 --> 00:25:20.400
Um, so that whenever you're developing code, it goes in the right direction.
00:25:20.640 --> 00:25:32.880
But actually, I missed this, but before that, you need to clean up your current current code because if it learns wrong patterns from your current code or current DAGs, like it's it's very difficult.
00:25:33.039 --> 00:25:36.160
Um, it'll always create like not great code.
00:25:36.400 --> 00:25:37.519
Uh that's right.
00:25:37.759 --> 00:25:44.240
So so we cleaned it up, we created like a new system, and then we also have like a reviewer bot on top of it.
00:25:44.319 --> 00:25:48.960
So there's a reviewer bot that reviews every PR in the company, tech, not like everything.
00:25:49.119 --> 00:25:58.799
Uh, so we added specific uh instructions there that even if someone misses anything, even when they're building it through the skill, it's able to catch um these smaller things now.
00:25:58.960 --> 00:26:02.079
Things like memory leaks are very difficult to spot.
00:26:02.319 --> 00:26:05.119
So all those things you have added, like specific um edition.
00:26:05.440 --> 00:26:06.559
It's still a work in progress.
00:26:06.720 --> 00:26:13.519
There's still sometimes where we miss things and like uh things need to be modified, but again, like it's just an iterative process.
00:26:13.680 --> 00:26:16.319
Like if something goes wrong, we try to fix it the next time.
00:26:16.799 --> 00:26:21.279
But this is another AI that we've used the yes.
00:26:21.759 --> 00:26:22.319
Oh, okay.
00:26:22.400 --> 00:26:28.160
So it's like you have like you generate the code with AI and then you have another AI that checks whatever has been generated.
00:26:28.400 --> 00:26:29.119
Yes, yes.
00:26:29.440 --> 00:26:29.920
Okay, okay.
00:26:30.000 --> 00:26:31.680
So it's like AI as a judge, right?
00:26:31.759 --> 00:26:32.960
I mean kind of, yes.
00:26:33.119 --> 00:26:40.559
So it does like first level of um checking for things, so that it's easier for like the reviewers to then go ahead and actually win the reviewing code.
00:26:40.720 --> 00:26:43.519
Um, most of the obvious mistakes are fixed before then.
00:26:43.839 --> 00:26:44.160
Right.
00:26:44.480 --> 00:26:59.599
I think you you said something super uh super important is that um is that when you if you if you like if you decide to uh create a skill out of your code, then you better make sure that your code actually is great.
00:26:59.839 --> 00:27:12.319
Uh and I think maybe this is one of the things that AI um bring you know brought back to life is the documentation as well, like ensuring that you document your uh your code properly.
00:27:12.799 --> 00:27:18.400
Uh, you know, it it I mean today it's it's super important because obviously AI reads all of that.
00:27:19.039 --> 00:27:23.359
Um yeah, I mean I think this is uh this is super interesting.
00:27:23.440 --> 00:27:28.240
Uh this is super interesting the way that you are using uh AI and and again as you said, right?
00:27:28.400 --> 00:27:35.200
AI can uh hallucinate uh quite a lot, so you better be careful with uh whatever you generate uh with it.
00:27:35.839 --> 00:27:45.920
Well, uh I must say I'm very I'm very shocked because it's already uh over, and to be honest, that went super fast.
00:27:47.359 --> 00:27:48.559
That went super fast.
00:27:48.720 --> 00:27:52.079
Uh I'm sure that people will enjoy a lot uh this episode.
00:27:52.240 --> 00:27:56.480
You obviously explained a lot of very interesting things and uh in details.
00:27:56.559 --> 00:27:58.160
Uh so thank you so much for that.
00:27:58.319 --> 00:28:08.799
Uh but before leaving, I have to ask uh what is the best way for uh folks to get in touch with you if they want to ask questions, maybe or you know if they want to learn more about uh head out.
00:28:09.519 --> 00:28:13.039
Yeah, um, you can just uh connect with me on LinkedIn and maybe Blue Sky.
00:28:13.279 --> 00:28:16.720
I don't use it as often, but you can shoot me at my probably replacing a few days.
00:28:17.839 --> 00:28:18.960
Awesome, awesome.
00:28:19.119 --> 00:28:21.359
Well, thank you so much again, Minalini.
00:28:21.440 --> 00:28:23.279
Um, hope to see you again for another episode.
00:28:23.440 --> 00:28:24.079
Who knows?
00:28:24.240 --> 00:28:26.559
Uh but otherwise, I wish you the best.
00:28:26.720 --> 00:28:27.920
Take care and uh see you.
00:28:28.640 --> 00:28:28.960
You do.
00:28:29.200 --> 00:28:29.920
Thank you so much for having me.
00:28:30.000 --> 00:28:30.880
It was a very nice chat.
00:28:31.279 --> 00:28:31.519
Thank you.
00:28:31.599 --> 00:28:31.839
See you.
00:28:32.799 --> 00:28:32.960
Bye.
00:28:34.400 --> 00:28:37.359
Thank you for listening to the Data Flowcast.
00:28:37.519 --> 00:28:40.960
Check the show notes for links to everything mentioned today.
00:28:41.119 --> 00:28:45.599
If you're enjoying the podcast, subscribe so you never miss an episode.
00:28:46.000 --> 00:28:48.079
And leave us a five-star review.
