WEBVTT
00:00:00.320 --> 00:00:07.360
You're listening to the data flowcast, a data podcast about Airflow and the world of data and AI around it.
00:00:07.759 --> 00:00:18.879
We are excited to invite you to orchestrate everything for a free online conference on September 19th engineering data from the left around and wait air out there.
00:00:28.559 --> 00:00:31.679
Attendees get a discount code for a free certification.
00:00:32.000 --> 00:00:34.240
A$150 value.
00:00:34.399 --> 00:00:39.439
Go to astronomer.link slash data minus flowcast minus OE.
00:00:39.759 --> 00:00:41.200
Link in the show notes.
00:00:41.359 --> 00:00:43.200
Now let's get into the show.
00:00:43.600 --> 00:00:44.399
Hi everyone.
00:00:44.560 --> 00:00:46.240
Welcome to the Data Flowcast.
00:00:46.399 --> 00:00:54.320
I'm your host, Kenton Dannis, and today I'm joined by Komuda Srinivasa, who is a senior data architect at ATC Drivetrain.
00:00:54.640 --> 00:00:56.000
Komuda, how are you?
00:00:56.479 --> 00:00:57.920
I'm pretty good, Kenton.
00:00:58.079 --> 00:00:58.880
How about you?
00:00:59.280 --> 00:00:59.840
I'm good.
00:01:00.000 --> 00:01:01.600
Yeah, thanks for asking.
00:01:01.759 --> 00:01:03.280
It's really great to have you on.
00:01:03.359 --> 00:01:05.040
I'm excited to dig into it.
00:01:05.200 --> 00:01:14.959
So just for a bit of a teaser for today's podcast, we're going to discuss self-healing airflow platforms and AI-assisted pipeline recovery.
00:01:15.280 --> 00:01:16.400
Very hot topic.
00:01:16.560 --> 00:01:19.840
I think everybody in the industry is thinking about this right now.
00:01:19.920 --> 00:01:22.879
So I'm sure this will be very helpful to our listeners.
00:01:23.280 --> 00:01:26.799
Before we get into all of that, I want to start with a little bit of background.
00:01:26.959 --> 00:01:29.439
So tell me first about the company that you work for.
00:01:29.519 --> 00:01:31.680
What does ATC Drivetrain do?
00:01:32.239 --> 00:01:34.000
Oh, thank you for having me.
00:01:34.640 --> 00:01:36.560
That's the first thing I want to say.
00:01:36.799 --> 00:01:39.200
So I work for ATC Drivetrain.
00:01:39.359 --> 00:01:42.640
ATC Drivetrain is a remanufacturing uh industry.
00:01:42.879 --> 00:01:49.760
So I think you would know about old engines where like they take up the parts and build a new engine.
00:01:49.920 --> 00:01:50.959
That's what we do.
00:01:51.200 --> 00:01:55.040
And we do like the trucks, the cars, the bigger trucks.
00:01:55.280 --> 00:02:01.680
So ATC kind of operates in the automotive remanufacturing, so powertrain products.
00:02:01.760 --> 00:02:06.480
So we have major four major facilities across the US and also Europe.
00:02:06.959 --> 00:02:17.120
Uh this is like a very complex production environment where data comes from manufacturing equipment, short-floor applications, quality systems, and supply chain processes.
00:02:17.439 --> 00:02:22.319
So the whole scenario is like data reliability is extremely important.
00:02:22.479 --> 00:02:31.919
If an F a delay or an incorrect pipeline in the whole ATC dry train can have a big production visibility or quality decisions going bad.
00:02:32.080 --> 00:02:34.000
Yeah, that's where I work for.
00:02:34.400 --> 00:02:35.759
Yeah, very cool.
00:02:36.080 --> 00:02:44.479
I think it's probably the case that a lot of people don't think a lot about where the parts of their car come from.
00:02:46.159 --> 00:02:51.199
But it has a big impact on their day-to-day lives, or at least it does for those of us in the US.
00:02:51.360 --> 00:02:59.120
Uh so yeah, really cool to hear about that and the role that data plays in the manufacturing of those parts.
00:02:59.439 --> 00:03:01.520
I would love to dig more into that.
00:03:01.599 --> 00:03:07.759
So tell me a little bit more about what your team does specifically and then how Airflow fits into that data management.
00:03:08.080 --> 00:03:13.759
Um, so our whole IT team, we connect operational manufacturing data across the whole Shopflow systems.
00:03:14.159 --> 00:03:18.000
So Shopflow is basically where you have like a live manufacturing going on.
00:03:18.639 --> 00:03:22.479
They take off the screws, they do everything, small, small parts, everything.
00:03:22.639 --> 00:03:27.599
So we have like enterprises database, cloud platforms, you know, analytic tools.
00:03:27.680 --> 00:03:36.879
So we use Airflow has orchestration for like ETLs and ELT jobs, and we do data quality checks and production reporting and file processing.
00:03:37.120 --> 00:03:42.719
We operate almost on thousands of DACs and across different schedules and business processes.
00:03:42.960 --> 00:03:49.120
So that's how a whole team works on like manufacturing data, so systems, and enterprises data systems.
00:03:49.280 --> 00:03:49.919
Yeah.
00:03:50.800 --> 00:03:51.439
Okay, great.
00:03:51.520 --> 00:03:55.759
And then it sounds like so Airflow is kind of your central orchestrator within all of that.
00:03:56.479 --> 00:04:02.159
Yeah, uh, so we have all the workflows, like into centralized platform, data platform, which is Airflow.
00:04:02.240 --> 00:04:07.360
Uh it gives us a common way to manage dependencies, scheduling, uh, monitoring, auditability.
00:04:07.520 --> 00:04:13.360
So even a small percentage of failures, everything workload is taken by Airflow.
00:04:13.439 --> 00:04:16.879
So that's our main centralized management uh dependency system.
00:04:16.959 --> 00:04:17.519
Yeah.
00:04:17.920 --> 00:04:18.720
Awesome.
00:04:18.959 --> 00:04:20.639
That's great background.
00:04:20.879 --> 00:04:26.800
And now I want to dig into this self-healing pipeline use case that I mentioned at the front.
00:04:26.959 --> 00:04:36.079
This is something that I keep hearing more and more about, and it's something that Astronomer is uh very interested in as well and is implementing as part of our product.
00:04:36.240 --> 00:04:38.480
Uh so I want to hear about how you've done that.
00:04:38.560 --> 00:04:46.480
And maybe first we can talk about just kind of conceptually what does self-healing mean in the context of your setup?
00:04:47.279 --> 00:04:52.720
So when we really talk about self-healing, it means like let's say you have done a small pipeline.
00:04:52.879 --> 00:05:00.879
Uh, can that pipeline identify a known failure or a condition, whether it's still safe or like it can recover automatically?
00:05:01.040 --> 00:05:06.240
So, for example, you are performing an approved recovery action, you need to validate the results.
00:05:06.399 --> 00:05:11.759
So, continuous processing without you know having a person, like for example, I'm a senior data architect.
00:05:11.920 --> 00:05:16.800
If you have a pipeline which can just self-heal, you have that in this word itself.
00:05:16.959 --> 00:05:23.759
You have a bandage and you just put a bandage and it heals itself, you don't need like you know, kind of extra stitches or something.
00:05:23.920 --> 00:05:26.160
That's what I would call it as self-heal.
00:05:26.319 --> 00:05:37.759
So without intervention of any person, uh a pipeline is identifying a problem and determining if it is like you can recover automatically and having an action plan and validating the results.
00:05:37.920 --> 00:05:40.079
That's the whole part of self-healing.
00:05:40.399 --> 00:05:44.720
It doesn't mean like you know, giving an AI unrestricted control flow.
00:05:44.959 --> 00:05:54.399
So our approach in ATC drivetrain is like you have a deterministic uh automation, you have policies and AI like you know assisted diagnosis.
00:05:54.720 --> 00:05:58.959
For example, we had a transient database connectivity delay.
00:05:59.040 --> 00:06:03.839
So there was like upstream dependency, a temporary, you know, some API went down.
00:06:04.160 --> 00:06:15.439
So what we did was like we had set up a policy saying that if an API goes down or a resource capacity goes down, you can recover by yourself automatically.
00:06:15.600 --> 00:06:20.800
If there's like uh unexpected schema changes, no unexpected schema changes happening.
00:06:20.959 --> 00:06:27.360
So it did check the data quality, security, and business logics which we had set up instead of escalating to a person.
00:06:27.439 --> 00:06:30.240
We did get a mail like, hey, this went down.
00:06:30.399 --> 00:06:36.879
So it we I took it in in, Airflow took it in and just like recovered on itself.
00:06:37.120 --> 00:06:50.240
So it's not about eliminating the engineers from the process, it's just you know, a person is like fixing the same stuff again and again, our recovery work so that Airflow can do it on its own.
00:06:50.399 --> 00:06:52.399
So that's what is self-healing.
00:06:52.560 --> 00:07:03.680
It heals on itself uh instead of putting a burden on the engineer, which is on or concentrating on, which is really genuinely need concentration on, for example, projects or any investigations they need to do.
00:07:04.399 --> 00:07:24.800
Yeah, it's certainly a much better experience for the engineers to you know log on and get a notification that hey, there was an error and now it is solved, and here's how it was solved, or now there's a PR that does solve it that's open for your review, as opposed to just a red square, and then you have to start from nothing to figure it out.
00:07:24.959 --> 00:07:27.360
So yeah, that's very cool.
00:07:27.680 --> 00:07:33.600
And tell me a little bit more about how you've implemented this from an architecture standpoint.
00:07:33.759 --> 00:07:36.319
So, how does the system detect the failure?
00:07:36.399 --> 00:07:40.879
And then what by what mechanism is it doing that recovery?
00:07:41.600 --> 00:07:45.920
So when we talk about architecture, first is like self-healing or anything, right?
00:07:46.160 --> 00:07:49.519
So it should have like we had like five layers, I would say.
00:07:49.680 --> 00:07:51.759
First in first is the observability layer.
00:07:51.839 --> 00:07:57.680
Like even a person is usually is like if there's a failure, what do what happened?
00:07:57.839 --> 00:07:58.879
What are the locks?
00:07:59.040 --> 00:08:11.360
So we had airflow task, which calls back and collects like the failure task, logs, error messages, you know, metadata, and what was the dependency state, history, execution time, you know, infrastructure signals.
00:08:11.600 --> 00:08:15.839
This gives the idea of like to the A4, like, hey, something went wrong.
00:08:15.920 --> 00:08:18.160
I'm observing it, I'm collecting the data.
00:08:18.319 --> 00:08:22.959
So, second is like you have the use huge data, but you need to classify it.
00:08:23.120 --> 00:08:27.279
So the system compares like incident with like you know, what were the failures?
00:08:27.439 --> 00:08:34.240
Let's say we usually use Airflow only for something which has already repeated, happened.
00:08:34.399 --> 00:08:41.840
We are not exposing AI with that whole control, like, hey, this is a new incident, you just don't want to have it to take a decision and it does happen.
00:08:42.000 --> 00:08:43.279
So classification.
00:08:43.440 --> 00:08:52.000
So it goes to the library of known failure patterns, it includes like you know connections, timeouts, you know, upstream data missing or like you know, resource limit.
00:08:52.159 --> 00:08:57.440
So it just compares with the data quality failures or any files, then comes the policy layer.
00:08:57.600 --> 00:09:03.600
We would have set up with a bunch of policies saying, hey, this was a failure, but you need to check all of this.
00:09:03.759 --> 00:09:12.080
If anything is changed, make sure there's automatic recovery permitted, go back to that category, get the confidence level.
00:09:12.159 --> 00:09:14.720
Uh, you had this number of previous attempts.
00:09:14.960 --> 00:09:23.120
If it is not done by you, like you know, approved recovery actions, whatever you go through, and you match each and every policy layer.
00:09:23.360 --> 00:09:25.120
And the fourth comes the recovery.
00:09:25.200 --> 00:09:38.960
So, depending on the pattern, the confidence level, you try to go off back off, maybe re-establish a connection, you know, rerun a task or like a resource management or processing from a safe point, you can do that.
00:09:39.120 --> 00:09:47.759
But if something happens or you're not sure, just you know, send a mail to the engineer or trigger a big alarm or something.
00:09:48.000 --> 00:09:50.320
So that's like the four layers.
00:09:50.480 --> 00:09:59.279
Once you have validated, sorry, you have recovered, you have gone through the policy, you need a validation layer that if the recovery actually has happened and succeeded.
00:09:59.440 --> 00:10:07.440
So you put some tests which goes through like rogue forms, time stamp, data quality rules, or downstream dependencies.
00:10:07.679 --> 00:10:13.919
So if you have that validation done, so the whole thing, the architecture is this five layers.
00:10:14.080 --> 00:10:15.440
So the system stops.
00:10:15.600 --> 00:10:26.320
Let's say you escalate an incident and you did like a recovery, the validation is done that the system has started, then your whole complete architecture depends on that.
00:10:26.559 --> 00:10:32.000
So I would say observe, classify, policy, and recovery and validate.
00:10:32.159 --> 00:10:36.720
So these are the five architecture pieces for a failure and recovery on its own.
00:10:36.799 --> 00:10:37.360
Yeah.
00:10:37.759 --> 00:10:39.039
Yeah, that makes sense.
00:10:39.200 --> 00:10:43.840
And tell me a little more about the AI piece of the recovery loop.
00:10:43.919 --> 00:10:53.679
So you're using the AI to help figure out how to solve whatever the issue is, and then decide whether it needs to be routed to a human first or not.
00:10:53.919 --> 00:10:57.039
Kind of what has been your experience working with it in that way?
00:10:57.120 --> 00:11:05.039
And has it required a lot of tuning, or do you have any, I guess, built-in mechanism for kind of improvement in that over time?
00:11:06.480 --> 00:11:10.879
Yes, I would say like AI primarily helps with classification and context gathering.
00:11:10.960 --> 00:11:19.919
You know, for example, let's say that AFlow does have a lot of context, but you need AI to help it like you know, classify it, gather it, and recommend limitations.
00:11:20.080 --> 00:11:22.320
So, for example, I have a startup called Resetly.
00:11:22.639 --> 00:11:26.639
So we have like a lot of data, especially from financial data.
00:11:26.799 --> 00:11:31.759
So traditional runes remain effective when there's like you know, repeated failures or something.
00:11:32.000 --> 00:11:41.039
Let's say uh you AI becomes more useful when the error is like less structured, and you need to have like consider all the signals together.
00:11:41.120 --> 00:11:52.879
Let's say you have like a thousand or through two thousand locks, and you need to identify the likely root cost and compare with the failure with historical incidents or you know, recommend the next uproot actions.
00:11:53.679 --> 00:11:59.279
AI is the best one to do that, it should not bypass the operational controls.
00:11:59.519 --> 00:12:04.480
Uh, the AI should can propose that an issue resembles you know a transient failure.
00:12:04.639 --> 00:12:08.960
For example, let's say you have a policy layer, whether a retry is safe or permitted.
00:12:09.039 --> 00:12:14.240
Let's say you you retry it like four times, but is was is it safe to retry again or permitted?
00:12:14.399 --> 00:12:18.799
So it depends on the high confidence or low risk, you know, the action can be automatic.
00:12:18.960 --> 00:12:22.480
So AI comes into picture with with a lot of these cases.
00:12:22.639 --> 00:12:35.519
So AI prepares the diagnosis, recommended steps for the engineer, it gives us the speed, like you know, while maintaining the production governance or AI policies, especially when you're dealing with financial data and also the automatic automated, which is like a live data.
00:12:35.759 --> 00:12:37.519
Both of them have their own risk.
00:12:37.600 --> 00:12:45.840
And AI uh stands behind to help airflow to rely on like some of the decisions, gathering approved actions.
00:12:46.080 --> 00:12:54.639
But most of the things AI does is like it recommends rather than uh just sitting there and you know just taking the own decision.
00:12:54.799 --> 00:12:59.039
That would be the best case of scenario of like AI fit into the recovery loop.
00:12:59.440 --> 00:13:10.720
You should have AI placed in the back set, helping you out with all of the stuff, but you should have your approval steps, even for a recovery or a human involved in that.
00:13:10.799 --> 00:13:11.440
Yeah.
00:13:12.960 --> 00:13:14.799
Yeah, that's really helpful.
00:13:14.960 --> 00:13:16.720
And I love the example that you gave.
00:13:16.879 --> 00:13:31.360
I'm curious if you have another example you can talk about around like a failure that you might have had before you had the self-healing in place and kind of how that's improved now with the new architecture.
00:13:31.759 --> 00:13:38.960
Uh, for example, when Recetly came into picture, like I have a Recetly app, which is like takes into Xpense app basically.
00:13:39.279 --> 00:13:48.240
So we had like a lot of people started using it and it had a failure because it was the currency exchange, just giving an example.
00:13:48.399 --> 00:13:54.000
And A4 Airflow triggered like, hey, this is currency exchange and the rows are not matching up.
00:13:54.320 --> 00:14:07.919
So we had done an AI policy saying, like, let's say you get an in dollars or something, uh, you need to trigger to change it to Euros or pounds if the person is traveling when he is on or something.
00:14:08.159 --> 00:14:11.120
So it triggered, but it did not work.
00:14:11.360 --> 00:14:20.879
So Airflow recommended the steps, AI recommended the steps, which saved us almost downtime of four hours, which would have costed us almost 120k.
00:14:21.440 --> 00:14:28.240
So wow, yeah, so the DAX, all the formulas, you know, there's a lot of stuff which happens.
00:14:28.320 --> 00:14:41.679
We we say, like, hey, it's an expense app, and uh, we have like the point of sales connected to you know, you buy something and which is like 45 dollars, but it was showing like 450 dollars uh 450 euros pounds, which had like transit in it.
00:14:41.759 --> 00:14:44.480
So AI recommended that it happened at 12 a.m.
00:14:44.639 --> 00:14:54.639
So all of us were asleep or something, it would get a big warning that self-healing with the AI saying, like you need to convert this, this number of rows are not matching.
00:14:54.799 --> 00:14:57.919
It saved us a lot of money, especially production time.
00:14:58.000 --> 00:15:02.080
Uh I mean not production, I would say downtime, and also the money-wise.
00:15:02.240 --> 00:15:02.799
Yeah.
00:15:03.120 --> 00:15:05.919
Yeah, that's an amazing success story.
00:15:07.200 --> 00:15:12.720
How often do you feel like it gets the right answer in terms of the recovery?
00:15:12.879 --> 00:15:15.440
Like, what is the success rate overall?
00:15:15.919 --> 00:15:16.960
Uh it depends.
00:15:17.120 --> 00:15:22.399
So when we talk about recovery time or recovery success, it depends on the confidence level.
00:15:22.639 --> 00:15:32.720
For when you're setting up an AI, I'm saying, like, you know, everyone uses AI, there's a downtime, everyone, you know, this suddenly there's a browser which goes down, let's say.
00:15:32.879 --> 00:15:36.080
So Airflow has something called as confidence levels.
00:15:36.240 --> 00:15:44.240
Let's say if your failure falls in between like 10 to 25 percent, the success rate is almost 99%.
00:15:44.960 --> 00:15:49.360
If it is a higher risk, it may go below, you know, kind of 50%.
00:15:49.679 --> 00:15:59.519
Also, it depends on the confidence level, but I would say 95% of the failure modes it fixes, which can be like a small API or any stuff, it just fixes on its own.
00:15:59.600 --> 00:16:00.000
Yeah.
00:16:02.159 --> 00:16:02.320
Yeah.
00:16:02.480 --> 00:16:06.240
Uh I would say like 95% of the time, small, small errors, it just takes care.
00:16:07.360 --> 00:16:07.919
That's amazing.
00:16:08.000 --> 00:16:10.080
That must have saved your team so much time.
00:16:10.960 --> 00:16:22.720
That's absolutely this episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:16:22.960 --> 00:16:33.120
From batch pipelines to training models and wiring up agents, Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
00:16:33.360 --> 00:16:35.759
Check it out at astronomer.io.
00:16:38.559 --> 00:16:39.120
Cool.
00:16:39.279 --> 00:16:44.480
Well, I would love to also talk about AI-assisted troubleshooting.
00:16:44.799 --> 00:16:50.480
So maybe in the case where you do need a little more human involvement to figure out what's going on.
00:16:50.720 --> 00:16:54.159
You mentioned that you're running EarthLow at quite a high scale.
00:16:54.320 --> 00:16:57.120
So I think you told me you have thousands of DAGs.
00:16:57.679 --> 00:17:06.480
So when when you're working with that sort of volume, how does AI help someone find the root cause of a failure?
00:17:07.359 --> 00:17:16.799
So when I talk about like a lot of stuff, when so for an engineer, I'm just saying you trigger a failure, you just go into the locks.
00:17:16.960 --> 00:17:22.559
Let's say uh a piece of information doesn't give so much context.
00:17:22.799 --> 00:17:26.480
So when you need a lot of connection and it takes it saves a lot of time.
00:17:26.559 --> 00:17:34.240
Let's say for me as a senior data architect, I sit down with automative the whole table, database, or airflow or something, right?
00:17:34.480 --> 00:17:42.079
So when you have in this transaction locks, which is like thousands of rows, AI gives us the contextual locks.
00:17:42.240 --> 00:17:45.759
Sometimes you don't understand, like you're not an expert in everything.
00:17:46.000 --> 00:17:50.559
AI does give you, like, you know, for example, let's say something like 404.
00:17:50.799 --> 00:17:52.160
I will be like, what's that?
00:17:52.240 --> 00:17:54.480
I'm gonna Google it, I have to spend a lot of time.
00:17:54.640 --> 00:18:05.279
So AI does give you contextual locks information, which gets from a different, you know, it just not one connection like DTSQL01 or a SQL or any of the database.
00:18:05.599 --> 00:18:10.160
So it gathers all the information and gives you like, hey, you may need to check this connection.
00:18:10.240 --> 00:18:11.599
This API may have failed.
00:18:11.759 --> 00:18:17.119
It's triggering a lot of upstreams or downstreams data loss or something.
00:18:17.440 --> 00:18:21.680
So the biggest challenge is not detecting something which has failed.
00:18:21.759 --> 00:18:25.680
So it gives the visibility and also recommends action plans.
00:18:26.000 --> 00:18:30.480
You may not have that in mind, oh, I may do this, it will fix this API.
00:18:30.559 --> 00:18:40.720
So uh let's say 27 task failures is connected to some one unavailable you know source database, which the AI is checking, which I can't check, like sometimes I may forget it.
00:18:40.880 --> 00:18:48.000
So the whole scenario is that AI gives us visibility, observability, recommended actions.
00:18:48.160 --> 00:18:54.960
So even with thousands of lines, connections, and everything, AI sits in the middle to have a recommendation system.
00:18:55.440 --> 00:18:59.519
So that's helps a lot with especially with this big environment.
00:18:59.680 --> 00:19:11.039
And especially airflows, you have like a manual scheduling or any scheduling, AI sits back of the airflow and helps with all the stuff, which is like recovery, self-healing.
00:19:11.200 --> 00:19:13.440
That's the center of it, yeah, I would say.
00:19:14.000 --> 00:19:15.039
Yeah, absolutely.
00:19:15.279 --> 00:19:23.920
At Astronomer, we have built some of this into our own product where we have our data engineering agent, which is called Auto, can investigate failures for you.
00:19:24.000 --> 00:19:40.160
And the whole purpose is that you can just click a button and it will do it, and you don't ever have to open the logs, which I, as you said, I think is so helpful the amount of time that I have spent trying to figure out like what an error message in the log means, or even trying to find like the right part in the logs.
00:19:40.240 --> 00:19:48.880
Like sometimes the thing that generates the error is not really the actual issue, it's like multiple steps up or down, and it's just cascaded.
00:19:49.039 --> 00:19:54.559
And the AI is like super fast at figuring out, like, oh, this is the actual problem.
00:19:54.720 --> 00:19:55.839
Like, this is great.
00:19:56.160 --> 00:20:12.799
Yes, I mean agents, especially nowadays, is like when you try to use agents in a very specific way, it helps you like a lot save a lot of money, I would say, in terms of money, the time, and not having the stressful day for the engineer rather than doing more projects, more productivity.
00:20:12.960 --> 00:20:15.200
And the first thing is no downtime.
00:20:15.279 --> 00:20:25.759
So, yeah, yeah, definitely opens the door to doing a lot more, it's just like net productive work as opposed to, like you said, lots of Googling of errors.
00:20:26.079 --> 00:20:27.759
Yes, that's absolutely true.
00:20:27.920 --> 00:20:35.839
I've seen my uh network protocols, people like especially shutting the door and like going on like, hey, we can use this, you know, autopilot or something.
00:20:36.000 --> 00:20:49.680
If you own the you know AI agents, you don't need to train the you know some other resources or something, but if you're owning the AI agents, it helps you to keep the company conserved information within itself and help you out.
00:20:49.839 --> 00:20:51.279
That's one of the things here.
00:20:51.680 --> 00:20:52.799
Yeah, totally.
00:20:53.200 --> 00:21:05.440
Well, speaking of agents, I on a slightly different topic, still related to AI and Airflow, but I understand that you are also working with AI agents that run through Airflow.
00:21:05.680 --> 00:21:07.519
Another very popular topic.
00:21:07.599 --> 00:21:10.640
Uh, we do this on my team at Astronomer as well.
00:21:10.799 --> 00:21:15.920
But I'm curious to learn more about how you are using Airflow for agent orchestration.
00:21:16.480 --> 00:21:20.799
Um, we do have like four agents running uh in ATC TriTrain.
00:21:20.960 --> 00:21:23.680
I also have two agents running for Recitly actually.
00:21:23.920 --> 00:21:32.000
So we treat um AI agents as governed workflow components rather than independent process running without like any any boundaries.
00:21:32.160 --> 00:21:37.759
So Airflow is responsible, you know, scheduling, state management, retires, timeouts, logging, and everything.
00:21:38.079 --> 00:21:51.200
So for us in ATC TriTrend or in Ricitly, uh, an agent perform a boundary task such as like classifying an incident, summarizing logs, checking metadata, or like you know, recommendation in a plan or validating.
00:21:51.440 --> 00:21:59.759
So, for example, I have in recitly one agent just analyzes the failure because uh recitly needs to be live, especially.
00:22:00.079 --> 00:22:04.319
Especially for point of sale POS systems, so it can't go down all the time.
00:22:04.480 --> 00:22:18.720
And also we have second agent running all the time because you have this historical incidents happening all the time with the item not matching with the bill and giving wrong, you know, the item amount or something.
00:22:18.960 --> 00:22:33.119
So we have like two running there, but in ATC we have like another one running all the time, which is the data quality, because data quality and data is the most heart of all this automotive industry.
00:22:33.200 --> 00:22:38.880
So Airflow coordinates those steps and records those inputs and outputs with the agent with the help of the agent.
00:22:38.960 --> 00:22:43.599
So whole agent behavior is more of like absorbable and reproducible.
00:22:43.680 --> 00:22:44.799
We control those tools.
00:22:44.880 --> 00:22:58.160
So uh it's it acts up on the human approval, uh, and if there's a sensitive step coming through, it stops execution based on the confidence level or a validation fails, then it triggers us.
00:22:58.319 --> 00:23:06.960
So agent is giving us a contextual reasoning, and airflow is giving the operation discipline for the whole um 82 drive train.
00:23:07.119 --> 00:23:08.480
I would say interesting.
00:23:08.640 --> 00:23:11.920
So it's just like a smarter data quality check.
00:23:12.240 --> 00:23:14.640
Yes, it has some reasoning behind it.
00:23:14.799 --> 00:23:16.000
Yeah, that's really cool.
00:23:16.160 --> 00:23:18.799
I haven't actually heard of somebody doing that yet.
00:23:19.200 --> 00:23:19.519
Yep.
00:23:19.759 --> 00:23:23.599
Uh, this was actually a project brought by someone from Europe.
00:23:23.759 --> 00:23:30.160
Actually, he uh like he eliminated like a lot of work of QE engineers.
00:23:30.480 --> 00:23:32.960
So that was like this agent.
00:23:33.039 --> 00:23:44.400
Uh, we may think it's a small data quality agent, but especially working in ATC drivetrend, we have like almost, I would say 17,000 rows of data every hour, like literally flowing.
00:23:44.480 --> 00:23:48.400
So data quality is like it may be a work of like six to seven people.
00:23:48.480 --> 00:23:53.359
So that agent is saving a lot of money any downtime or any triggers or anything.
00:23:53.440 --> 00:23:53.920
Yeah.
00:23:54.240 --> 00:23:55.680
Yeah, absolutely.
00:23:56.000 --> 00:23:57.599
Well, very interesting.
00:23:57.920 --> 00:23:59.039
Uh cool.
00:23:59.119 --> 00:24:01.759
Well, yeah, thank you for going through all of this.
00:24:01.920 --> 00:24:13.839
It's been, like I said, this is a really hot topic within the Airflow community right now, and it's always helpful for uh folks to get to hear how somebody is implementing this stuff.
00:24:14.720 --> 00:24:20.880
I mean, I've heard like everyone talk about this AI agents, you know, something's wrong or something's correct.
00:24:20.960 --> 00:24:25.119
You know, some people say it's saved on their side, some people say like it's out of control.
00:24:25.279 --> 00:24:28.400
It depends on how we use put boundaries and how you use it.
00:24:28.559 --> 00:24:34.720
And even for Airflow, Airflow saves a lot of time for especially for recovery or you know, capabilities.
00:24:34.880 --> 00:24:38.880
It's it's more of like the smaller team, the larger team, how to use it.
00:24:38.960 --> 00:24:40.319
That's all at the end of the day.
00:24:40.799 --> 00:24:41.279
Absolutely.
00:24:41.440 --> 00:24:46.160
All about the implementation, and as you mentioned, very importantly the guardrails in this case.
00:24:46.480 --> 00:24:46.880
Yeah.
00:24:47.119 --> 00:24:47.440
Yeah.
00:24:48.640 --> 00:24:48.960
Cool.
00:24:49.039 --> 00:24:58.640
Well, I think that's a good place to pivot to my standard final question, uh, which is what would you most like to see from development of the Airflow project in the near future?
00:24:59.119 --> 00:25:07.039
Uh, one of the stronger is like Airflow should continue on stronger capabilities around observability, event-driven orchestration.
00:25:07.200 --> 00:25:30.319
And also, I would ask them to more focus on the AI agent governance because for AI agent, it's uh I would like to standard as ways to track prompts, tool calls, you know, model versions, approvals, confidence, maybe more on agent outputs as a part of the task history that would make easier, you know, the for the agent workflows to be easier to audit it, operate safely for AFLO.
00:25:30.400 --> 00:25:37.039
So I would want it to be more flexible, even-driven execution, you know, for dynamic log running workflows.
00:25:37.200 --> 00:25:43.839
And I mean, AFLO is already a powerful orchestrator, like it's a very based on governance and everything.
00:25:44.000 --> 00:25:52.319
But I would have more on like AI assisted workflows to make it more important for especially like enterprise AI operations, which we use.
00:25:52.559 --> 00:25:53.920
So maybe that.
00:25:54.240 --> 00:25:55.039
Yeah, for sure.
00:25:55.200 --> 00:26:01.599
Sounds like some requests for future development in the common AI provider, which I know is moving forward quite quickly right now.
00:26:01.680 --> 00:26:05.279
They keep adding it, it feels like they add new functionality to that every day.
00:26:05.440 --> 00:26:06.640
But yeah, that's true.
00:26:06.720 --> 00:26:14.319
Like it's hard to track, also, you know, like they just do some few releases and it's really hard, but they're doing like very quickly, also.
00:26:14.400 --> 00:26:14.640
Yep.
00:26:14.799 --> 00:26:15.759
Yeah, yeah.
00:26:16.079 --> 00:26:16.480
Cool.
00:26:16.640 --> 00:26:17.920
Well, I love that answer.
00:26:18.000 --> 00:26:20.240
Um, Kamuda, thank you so much for going through that.
00:26:20.319 --> 00:26:21.680
It's been really great speaking with you.
00:26:21.839 --> 00:26:24.880
Just to close this out, what is the best way for folks to get in touch with you?
00:26:25.200 --> 00:26:28.400
Uh, they can reach out to me through LinkedIn, uh, Kumudashrinivas.
00:26:28.799 --> 00:26:37.519
So that's my whole linked it, or they can just shoot me a mail at Kumudashunivas and Gmail.com if you have any questions or you want to reach out for anything else.
00:26:37.599 --> 00:26:37.920
Yeah.
00:26:38.240 --> 00:26:38.559
Great.
00:26:38.640 --> 00:26:40.640
We can throw links to those in the show notes.
00:26:40.799 --> 00:26:41.680
Well, thank you again.
00:26:41.759 --> 00:26:43.440
This has been a great conversation.
00:26:44.079 --> 00:26:47.839
Thank you so much for having me here, and it was a wonderful conversation with you.
00:26:47.920 --> 00:26:48.240
Yeah.
00:26:48.640 --> 00:26:51.599
Thank you for listening to the Data Flowcast.
00:26:51.759 --> 00:26:55.200
Check the show notes for links to everything mentioned today.
00:26:55.359 --> 00:26:59.920
If you're enjoying the podcast, subscribe so you never miss an episode.
00:27:00.160 --> 00:27:02.480
And leave us a five star review.
00:00:00.320 --> 00:00:07.360
You're listening to the data flowcast, a data podcast about Airflow and the world of data and AI around it.
00:00:07.759 --> 00:00:18.879
We are excited to invite you to orchestrate everything for a free online conference on September 19th engineering data from the left around and wait air out there.
00:00:28.559 --> 00:00:31.679
Attendees get a discount code for a free certification.
00:00:32.000 --> 00:00:34.240
A$150 value.
00:00:34.399 --> 00:00:39.439
Go to astronomer.link slash data minus flowcast minus OE.
00:00:39.759 --> 00:00:41.200
Link in the show notes.
00:00:41.359 --> 00:00:43.200
Now let's get into the show.
00:00:43.600 --> 00:00:44.399
Hi everyone.
00:00:44.560 --> 00:00:46.240
Welcome to the Data Flowcast.
00:00:46.399 --> 00:00:54.320
I'm your host, Kenton Dannis, and today I'm joined by Komuda Srinivasa, who is a senior data architect at ATC Drivetrain.
00:00:54.640 --> 00:00:56.000
Komuda, how are you?
00:00:56.479 --> 00:00:57.920
I'm pretty good, Kenton.
00:00:58.079 --> 00:00:58.880
How about you?
00:00:59.280 --> 00:00:59.840
I'm good.
00:01:00.000 --> 00:01:01.600
Yeah, thanks for asking.
00:01:01.759 --> 00:01:03.280
It's really great to have you on.
00:01:03.359 --> 00:01:05.040
I'm excited to dig into it.
00:01:05.200 --> 00:01:14.959
So just for a bit of a teaser for today's podcast, we're going to discuss self-healing airflow platforms and AI-assisted pipeline recovery.
00:01:15.280 --> 00:01:16.400
Very hot topic.
00:01:16.560 --> 00:01:19.840
I think everybody in the industry is thinking about this right now.
00:01:19.920 --> 00:01:22.879
So I'm sure this will be very helpful to our listeners.
00:01:23.280 --> 00:01:26.799
Before we get into all of that, I want to start with a little bit of background.
00:01:26.959 --> 00:01:29.439
So tell me first about the company that you work for.
00:01:29.519 --> 00:01:31.680
What does ATC Drivetrain do?
00:01:32.239 --> 00:01:34.000
Oh, thank you for having me.
00:01:34.640 --> 00:01:36.560
That's the first thing I want to say.
00:01:36.799 --> 00:01:39.200
So I work for ATC Drivetrain.
00:01:39.359 --> 00:01:42.640
ATC Drivetrain is a remanufacturing uh industry.
00:01:42.879 --> 00:01:49.760
So I think you would know about old engines where like they take up the parts and build a new engine.
00:01:49.920 --> 00:01:50.959
That's what we do.
00:01:51.200 --> 00:01:55.040
And we do like the trucks, the cars, the bigger trucks.
00:01:55.280 --> 00:02:01.680
So ATC kind of operates in the automotive remanufacturing, so powertrain products.
00:02:01.760 --> 00:02:06.480
So we have major four major facilities across the US and also Europe.
00:02:06.959 --> 00:02:17.120
Uh this is like a very complex production environment where data comes from manufacturing equipment, short-floor applications, quality systems, and supply chain processes.
00:02:17.439 --> 00:02:22.319
So the whole scenario is like data reliability is extremely important.
00:02:22.479 --> 00:02:31.919
If an F a delay or an incorrect pipeline in the whole ATC dry train can have a big production visibility or quality decisions going bad.
00:02:32.080 --> 00:02:34.000
Yeah, that's where I work for.
00:02:34.400 --> 00:02:35.759
Yeah, very cool.
00:02:36.080 --> 00:02:44.479
I think it's probably the case that a lot of people don't think a lot about where the parts of their car come from.
00:02:46.159 --> 00:02:51.199
But it has a big impact on their day-to-day lives, or at least it does for those of us in the US.
00:02:51.360 --> 00:02:59.120
Uh so yeah, really cool to hear about that and the role that data plays in the manufacturing of those parts.
00:02:59.439 --> 00:03:01.520
I would love to dig more into that.
00:03:01.599 --> 00:03:07.759
So tell me a little bit more about what your team does specifically and then how Airflow fits into that data management.
00:03:08.080 --> 00:03:13.759
Um, so our whole IT team, we connect operational manufacturing data across the whole Shopflow systems.
00:03:14.159 --> 00:03:18.000
So Shopflow is basically where you have like a live manufacturing going on.
00:03:18.639 --> 00:03:22.479
They take off the screws, they do everything, small, small parts, everything.
00:03:22.639 --> 00:03:27.599
So we have like enterprises database, cloud platforms, you know, analytic tools.
00:03:27.680 --> 00:03:36.879
So we use Airflow has orchestration for like ETLs and ELT jobs, and we do data quality checks and production reporting and file processing.
00:03:37.120 --> 00:03:42.719
We operate almost on thousands of DACs and across different schedules and business processes.
00:03:42.960 --> 00:03:49.120
So that's how a whole team works on like manufacturing data, so systems, and enterprises data systems.
00:03:49.280 --> 00:03:49.919
Yeah.
00:03:50.800 --> 00:03:51.439
Okay, great.
00:03:51.520 --> 00:03:55.759
And then it sounds like so Airflow is kind of your central orchestrator within all of that.
00:03:56.479 --> 00:04:02.159
Yeah, uh, so we have all the workflows, like into centralized platform, data platform, which is Airflow.
00:04:02.240 --> 00:04:07.360
Uh it gives us a common way to manage dependencies, scheduling, uh, monitoring, auditability.
00:04:07.520 --> 00:04:13.360
So even a small percentage of failures, everything workload is taken by Airflow.
00:04:13.439 --> 00:04:16.879
So that's our main centralized management uh dependency system.
00:04:16.959 --> 00:04:17.519
Yeah.
00:04:17.920 --> 00:04:18.720
Awesome.
00:04:18.959 --> 00:04:20.639
That's great background.
00:04:20.879 --> 00:04:26.800
And now I want to dig into this self-healing pipeline use case that I mentioned at the front.
00:04:26.959 --> 00:04:36.079
This is something that I keep hearing more and more about, and it's something that Astronomer is uh very interested in as well and is implementing as part of our product.
00:04:36.240 --> 00:04:38.480
Uh so I want to hear about how you've done that.
00:04:38.560 --> 00:04:46.480
And maybe first we can talk about just kind of conceptually what does self-healing mean in the context of your setup?
00:04:47.279 --> 00:04:52.720
So when we really talk about self-healing, it means like let's say you have done a small pipeline.
00:04:52.879 --> 00:05:00.879
Uh, can that pipeline identify a known failure or a condition, whether it's still safe or like it can recover automatically?
00:05:01.040 --> 00:05:06.240
So, for example, you are performing an approved recovery action, you need to validate the results.
00:05:06.399 --> 00:05:11.759
So, continuous processing without you know having a person, like for example, I'm a senior data architect.
00:05:11.920 --> 00:05:16.800
If you have a pipeline which can just self-heal, you have that in this word itself.
00:05:16.959 --> 00:05:23.759
You have a bandage and you just put a bandage and it heals itself, you don't need like you know, kind of extra stitches or something.
00:05:23.920 --> 00:05:26.160
That's what I would call it as self-heal.
00:05:26.319 --> 00:05:37.759
So without intervention of any person, uh a pipeline is identifying a problem and determining if it is like you can recover automatically and having an action plan and validating the results.
00:05:37.920 --> 00:05:40.079
That's the whole part of self-healing.
00:05:40.399 --> 00:05:44.720
It doesn't mean like you know, giving an AI unrestricted control flow.
00:05:44.959 --> 00:05:54.399
So our approach in ATC drivetrain is like you have a deterministic uh automation, you have policies and AI like you know assisted diagnosis.
00:05:54.720 --> 00:05:58.959
For example, we had a transient database connectivity delay.
00:05:59.040 --> 00:06:03.839
So there was like upstream dependency, a temporary, you know, some API went down.
00:06:04.160 --> 00:06:15.439
So what we did was like we had set up a policy saying that if an API goes down or a resource capacity goes down, you can recover by yourself automatically.
00:06:15.600 --> 00:06:20.800
If there's like uh unexpected schema changes, no unexpected schema changes happening.
00:06:20.959 --> 00:06:27.360
So it did check the data quality, security, and business logics which we had set up instead of escalating to a person.
00:06:27.439 --> 00:06:30.240
We did get a mail like, hey, this went down.
00:06:30.399 --> 00:06:36.879
So it we I took it in in, Airflow took it in and just like recovered on itself.
00:06:37.120 --> 00:06:50.240
So it's not about eliminating the engineers from the process, it's just you know, a person is like fixing the same stuff again and again, our recovery work so that Airflow can do it on its own.
00:06:50.399 --> 00:06:52.399
So that's what is self-healing.
00:06:52.560 --> 00:07:03.680
It heals on itself uh instead of putting a burden on the engineer, which is on or concentrating on, which is really genuinely need concentration on, for example, projects or any investigations they need to do.
00:07:04.399 --> 00:07:24.800
Yeah, it's certainly a much better experience for the engineers to you know log on and get a notification that hey, there was an error and now it is solved, and here's how it was solved, or now there's a PR that does solve it that's open for your review, as opposed to just a red square, and then you have to start from nothing to figure it out.
00:07:24.959 --> 00:07:27.360
So yeah, that's very cool.
00:07:27.680 --> 00:07:33.600
And tell me a little bit more about how you've implemented this from an architecture standpoint.
00:07:33.759 --> 00:07:36.319
So, how does the system detect the failure?
00:07:36.399 --> 00:07:40.879
And then what by what mechanism is it doing that recovery?
00:07:41.600 --> 00:07:45.920
So when we talk about architecture, first is like self-healing or anything, right?
00:07:46.160 --> 00:07:49.519
So it should have like we had like five layers, I would say.
00:07:49.680 --> 00:07:51.759
First in first is the observability layer.
00:07:51.839 --> 00:07:57.680
Like even a person is usually is like if there's a failure, what do what happened?
00:07:57.839 --> 00:07:58.879
What are the locks?
00:07:59.040 --> 00:08:11.360
So we had airflow task, which calls back and collects like the failure task, logs, error messages, you know, metadata, and what was the dependency state, history, execution time, you know, infrastructure signals.
00:08:11.600 --> 00:08:15.839
This gives the idea of like to the A4, like, hey, something went wrong.
00:08:15.920 --> 00:08:18.160
I'm observing it, I'm collecting the data.
00:08:18.319 --> 00:08:22.959
So, second is like you have the use huge data, but you need to classify it.
00:08:23.120 --> 00:08:27.279
So the system compares like incident with like you know, what were the failures?
00:08:27.439 --> 00:08:34.240
Let's say we usually use Airflow only for something which has already repeated, happened.
00:08:34.399 --> 00:08:41.840
We are not exposing AI with that whole control, like, hey, this is a new incident, you just don't want to have it to take a decision and it does happen.
00:08:42.000 --> 00:08:43.279
So classification.
00:08:43.440 --> 00:08:52.000
So it goes to the library of known failure patterns, it includes like you know connections, timeouts, you know, upstream data missing or like you know, resource limit.
00:08:52.159 --> 00:08:57.440
So it just compares with the data quality failures or any files, then comes the policy layer.
00:08:57.600 --> 00:09:03.600
We would have set up with a bunch of policies saying, hey, this was a failure, but you need to check all of this.
00:09:03.759 --> 00:09:12.080
If anything is changed, make sure there's automatic recovery permitted, go back to that category, get the confidence level.
00:09:12.159 --> 00:09:14.720
Uh, you had this number of previous attempts.
00:09:14.960 --> 00:09:23.120
If it is not done by you, like you know, approved recovery actions, whatever you go through, and you match each and every policy layer.
00:09:23.360 --> 00:09:25.120
And the fourth comes the recovery.
00:09:25.200 --> 00:09:38.960
So, depending on the pattern, the confidence level, you try to go off back off, maybe re-establish a connection, you know, rerun a task or like a resource management or processing from a safe point, you can do that.
00:09:39.120 --> 00:09:47.759
But if something happens or you're not sure, just you know, send a mail to the engineer or trigger a big alarm or something.
00:09:48.000 --> 00:09:50.320
So that's like the four layers.
00:09:50.480 --> 00:09:59.279
Once you have validated, sorry, you have recovered, you have gone through the policy, you need a validation layer that if the recovery actually has happened and succeeded.
00:09:59.440 --> 00:10:07.440
So you put some tests which goes through like rogue forms, time stamp, data quality rules, or downstream dependencies.
00:10:07.679 --> 00:10:13.919
So if you have that validation done, so the whole thing, the architecture is this five layers.
00:10:14.080 --> 00:10:15.440
So the system stops.
00:10:15.600 --> 00:10:26.320
Let's say you escalate an incident and you did like a recovery, the validation is done that the system has started, then your whole complete architecture depends on that.
00:10:26.559 --> 00:10:32.000
So I would say observe, classify, policy, and recovery and validate.
00:10:32.159 --> 00:10:36.720
So these are the five architecture pieces for a failure and recovery on its own.
00:10:36.799 --> 00:10:37.360
Yeah.
00:10:37.759 --> 00:10:39.039
Yeah, that makes sense.
00:10:39.200 --> 00:10:43.840
And tell me a little more about the AI piece of the recovery loop.
00:10:43.919 --> 00:10:53.679
So you're using the AI to help figure out how to solve whatever the issue is, and then decide whether it needs to be routed to a human first or not.
00:10:53.919 --> 00:10:57.039
Kind of what has been your experience working with it in that way?
00:10:57.120 --> 00:11:05.039
And has it required a lot of tuning, or do you have any, I guess, built-in mechanism for kind of improvement in that over time?
00:11:06.480 --> 00:11:10.879
Yes, I would say like AI primarily helps with classification and context gathering.
00:11:10.960 --> 00:11:19.919
You know, for example, let's say that AFlow does have a lot of context, but you need AI to help it like you know, classify it, gather it, and recommend limitations.
00:11:20.080 --> 00:11:22.320
So, for example, I have a startup called Resetly.
00:11:22.639 --> 00:11:26.639
So we have like a lot of data, especially from financial data.
00:11:26.799 --> 00:11:31.759
So traditional runes remain effective when there's like you know, repeated failures or something.
00:11:32.000 --> 00:11:41.039
Let's say uh you AI becomes more useful when the error is like less structured, and you need to have like consider all the signals together.
00:11:41.120 --> 00:11:52.879
Let's say you have like a thousand or through two thousand locks, and you need to identify the likely root cost and compare with the failure with historical incidents or you know, recommend the next uproot actions.
00:11:53.679 --> 00:11:59.279
AI is the best one to do that, it should not bypass the operational controls.
00:11:59.519 --> 00:12:04.480
Uh, the AI should can propose that an issue resembles you know a transient failure.
00:12:04.639 --> 00:12:08.960
For example, let's say you have a policy layer, whether a retry is safe or permitted.
00:12:09.039 --> 00:12:14.240
Let's say you you retry it like four times, but is was is it safe to retry again or permitted?
00:12:14.399 --> 00:12:18.799
So it depends on the high confidence or low risk, you know, the action can be automatic.
00:12:18.960 --> 00:12:22.480
So AI comes into picture with with a lot of these cases.
00:12:22.639 --> 00:12:35.519
So AI prepares the diagnosis, recommended steps for the engineer, it gives us the speed, like you know, while maintaining the production governance or AI policies, especially when you're dealing with financial data and also the automatic automated, which is like a live data.
00:12:35.759 --> 00:12:37.519
Both of them have their own risk.
00:12:37.600 --> 00:12:45.840
And AI uh stands behind to help airflow to rely on like some of the decisions, gathering approved actions.
00:12:46.080 --> 00:12:54.639
But most of the things AI does is like it recommends rather than uh just sitting there and you know just taking the own decision.
00:12:54.799 --> 00:12:59.039
That would be the best case of scenario of like AI fit into the recovery loop.
00:12:59.440 --> 00:13:10.720
You should have AI placed in the back set, helping you out with all of the stuff, but you should have your approval steps, even for a recovery or a human involved in that.
00:13:10.799 --> 00:13:11.440
Yeah.
00:13:12.960 --> 00:13:14.799
Yeah, that's really helpful.
00:13:14.960 --> 00:13:16.720
And I love the example that you gave.
00:13:16.879 --> 00:13:31.360
I'm curious if you have another example you can talk about around like a failure that you might have had before you had the self-healing in place and kind of how that's improved now with the new architecture.
00:13:31.759 --> 00:13:38.960
Uh, for example, when Recetly came into picture, like I have a Recetly app, which is like takes into Xpense app basically.
00:13:39.279 --> 00:13:48.240
So we had like a lot of people started using it and it had a failure because it was the currency exchange, just giving an example.
00:13:48.399 --> 00:13:54.000
And A4 Airflow triggered like, hey, this is currency exchange and the rows are not matching up.
00:13:54.320 --> 00:14:07.919
So we had done an AI policy saying, like, let's say you get an in dollars or something, uh, you need to trigger to change it to Euros or pounds if the person is traveling when he is on or something.
00:14:08.159 --> 00:14:11.120
So it triggered, but it did not work.
00:14:11.360 --> 00:14:20.879
So Airflow recommended the steps, AI recommended the steps, which saved us almost downtime of four hours, which would have costed us almost 120k.
00:14:21.440 --> 00:14:28.240
So wow, yeah, so the DAX, all the formulas, you know, there's a lot of stuff which happens.
00:14:28.320 --> 00:14:41.679
We we say, like, hey, it's an expense app, and uh, we have like the point of sales connected to you know, you buy something and which is like 45 dollars, but it was showing like 450 dollars uh 450 euros pounds, which had like transit in it.
00:14:41.759 --> 00:14:44.480
So AI recommended that it happened at 12 a.m.
00:14:44.639 --> 00:14:54.639
So all of us were asleep or something, it would get a big warning that self-healing with the AI saying, like you need to convert this, this number of rows are not matching.
00:14:54.799 --> 00:14:57.919
It saved us a lot of money, especially production time.
00:14:58.000 --> 00:15:02.080
Uh I mean not production, I would say downtime, and also the money-wise.
00:15:02.240 --> 00:15:02.799
Yeah.
00:15:03.120 --> 00:15:05.919
Yeah, that's an amazing success story.
00:15:07.200 --> 00:15:12.720
How often do you feel like it gets the right answer in terms of the recovery?
00:15:12.879 --> 00:15:15.440
Like, what is the success rate overall?
00:15:15.919 --> 00:15:16.960
Uh it depends.
00:15:17.120 --> 00:15:22.399
So when we talk about recovery time or recovery success, it depends on the confidence level.
00:15:22.639 --> 00:15:32.720
For when you're setting up an AI, I'm saying, like, you know, everyone uses AI, there's a downtime, everyone, you know, this suddenly there's a browser which goes down, let's say.
00:15:32.879 --> 00:15:36.080
So Airflow has something called as confidence levels.
00:15:36.240 --> 00:15:44.240
Let's say if your failure falls in between like 10 to 25 percent, the success rate is almost 99%.
00:15:44.960 --> 00:15:49.360
If it is a higher risk, it may go below, you know, kind of 50%.
00:15:49.679 --> 00:15:59.519
Also, it depends on the confidence level, but I would say 95% of the failure modes it fixes, which can be like a small API or any stuff, it just fixes on its own.
00:15:59.600 --> 00:16:00.000
Yeah.
00:16:02.159 --> 00:16:02.320
Yeah.
00:16:02.480 --> 00:16:06.240
Uh I would say like 95% of the time, small, small errors, it just takes care.
00:16:07.360 --> 00:16:07.919
That's amazing.
00:16:08.000 --> 00:16:10.080
That must have saved your team so much time.
00:16:10.960 --> 00:16:22.720
That's absolutely this episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:16:22.960 --> 00:16:33.120
From batch pipelines to training models and wiring up agents, Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
00:16:33.360 --> 00:16:35.759
Check it out at astronomer.io.
00:16:38.559 --> 00:16:39.120
Cool.
00:16:39.279 --> 00:16:44.480
Well, I would love to also talk about AI-assisted troubleshooting.
00:16:44.799 --> 00:16:50.480
So maybe in the case where you do need a little more human involvement to figure out what's going on.
00:16:50.720 --> 00:16:54.159
You mentioned that you're running EarthLow at quite a high scale.
00:16:54.320 --> 00:16:57.120
So I think you told me you have thousands of DAGs.
00:16:57.679 --> 00:17:06.480
So when when you're working with that sort of volume, how does AI help someone find the root cause of a failure?
00:17:07.359 --> 00:17:16.799
So when I talk about like a lot of stuff, when so for an engineer, I'm just saying you trigger a failure, you just go into the locks.
00:17:16.960 --> 00:17:22.559
Let's say uh a piece of information doesn't give so much context.
00:17:22.799 --> 00:17:26.480
So when you need a lot of connection and it takes it saves a lot of time.
00:17:26.559 --> 00:17:34.240
Let's say for me as a senior data architect, I sit down with automative the whole table, database, or airflow or something, right?
00:17:34.480 --> 00:17:42.079
So when you have in this transaction locks, which is like thousands of rows, AI gives us the contextual locks.
00:17:42.240 --> 00:17:45.759
Sometimes you don't understand, like you're not an expert in everything.
00:17:46.000 --> 00:17:50.559
AI does give you, like, you know, for example, let's say something like 404.
00:17:50.799 --> 00:17:52.160
I will be like, what's that?
00:17:52.240 --> 00:17:54.480
I'm gonna Google it, I have to spend a lot of time.
00:17:54.640 --> 00:18:05.279
So AI does give you contextual locks information, which gets from a different, you know, it just not one connection like DTSQL01 or a SQL or any of the database.
00:18:05.599 --> 00:18:10.160
So it gathers all the information and gives you like, hey, you may need to check this connection.
00:18:10.240 --> 00:18:11.599
This API may have failed.
00:18:11.759 --> 00:18:17.119
It's triggering a lot of upstreams or downstreams data loss or something.
00:18:17.440 --> 00:18:21.680
So the biggest challenge is not detecting something which has failed.
00:18:21.759 --> 00:18:25.680
So it gives the visibility and also recommends action plans.
00:18:26.000 --> 00:18:30.480
You may not have that in mind, oh, I may do this, it will fix this API.
00:18:30.559 --> 00:18:40.720
So uh let's say 27 task failures is connected to some one unavailable you know source database, which the AI is checking, which I can't check, like sometimes I may forget it.
00:18:40.880 --> 00:18:48.000
So the whole scenario is that AI gives us visibility, observability, recommended actions.
00:18:48.160 --> 00:18:54.960
So even with thousands of lines, connections, and everything, AI sits in the middle to have a recommendation system.
00:18:55.440 --> 00:18:59.519
So that's helps a lot with especially with this big environment.
00:18:59.680 --> 00:19:11.039
And especially airflows, you have like a manual scheduling or any scheduling, AI sits back of the airflow and helps with all the stuff, which is like recovery, self-healing.
00:19:11.200 --> 00:19:13.440
That's the center of it, yeah, I would say.
00:19:14.000 --> 00:19:15.039
Yeah, absolutely.
00:19:15.279 --> 00:19:23.920
At Astronomer, we have built some of this into our own product where we have our data engineering agent, which is called Auto, can investigate failures for you.
00:19:24.000 --> 00:19:40.160
And the whole purpose is that you can just click a button and it will do it, and you don't ever have to open the logs, which I, as you said, I think is so helpful the amount of time that I have spent trying to figure out like what an error message in the log means, or even trying to find like the right part in the logs.
00:19:40.240 --> 00:19:48.880
Like sometimes the thing that generates the error is not really the actual issue, it's like multiple steps up or down, and it's just cascaded.
00:19:49.039 --> 00:19:54.559
And the AI is like super fast at figuring out, like, oh, this is the actual problem.
00:19:54.720 --> 00:19:55.839
Like, this is great.
00:19:56.160 --> 00:20:12.799
Yes, I mean agents, especially nowadays, is like when you try to use agents in a very specific way, it helps you like a lot save a lot of money, I would say, in terms of money, the time, and not having the stressful day for the engineer rather than doing more projects, more productivity.
00:20:12.960 --> 00:20:15.200
And the first thing is no downtime.
00:20:15.279 --> 00:20:25.759
So, yeah, yeah, definitely opens the door to doing a lot more, it's just like net productive work as opposed to, like you said, lots of Googling of errors.
00:20:26.079 --> 00:20:27.759
Yes, that's absolutely true.
00:20:27.920 --> 00:20:35.839
I've seen my uh network protocols, people like especially shutting the door and like going on like, hey, we can use this, you know, autopilot or something.
00:20:36.000 --> 00:20:49.680
If you own the you know AI agents, you don't need to train the you know some other resources or something, but if you're owning the AI agents, it helps you to keep the company conserved information within itself and help you out.
00:20:49.839 --> 00:20:51.279
That's one of the things here.
00:20:51.680 --> 00:20:52.799
Yeah, totally.
00:20:53.200 --> 00:21:05.440
Well, speaking of agents, I on a slightly different topic, still related to AI and Airflow, but I understand that you are also working with AI agents that run through Airflow.
00:21:05.680 --> 00:21:07.519
Another very popular topic.
00:21:07.599 --> 00:21:10.640
Uh, we do this on my team at Astronomer as well.
00:21:10.799 --> 00:21:15.920
But I'm curious to learn more about how you are using Airflow for agent orchestration.
00:21:16.480 --> 00:21:20.799
Um, we do have like four agents running uh in ATC TriTrain.
00:21:20.960 --> 00:21:23.680
I also have two agents running for Recitly actually.
00:21:23.920 --> 00:21:32.000
So we treat um AI agents as governed workflow components rather than independent process running without like any any boundaries.
00:21:32.160 --> 00:21:37.759
So Airflow is responsible, you know, scheduling, state management, retires, timeouts, logging, and everything.
00:21:38.079 --> 00:21:51.200
So for us in ATC TriTrend or in Ricitly, uh, an agent perform a boundary task such as like classifying an incident, summarizing logs, checking metadata, or like you know, recommendation in a plan or validating.
00:21:51.440 --> 00:21:59.759
So, for example, I have in recitly one agent just analyzes the failure because uh recitly needs to be live, especially.
00:22:00.079 --> 00:22:04.319
Especially for point of sale POS systems, so it can't go down all the time.
00:22:04.480 --> 00:22:18.720
And also we have second agent running all the time because you have this historical incidents happening all the time with the item not matching with the bill and giving wrong, you know, the item amount or something.
00:22:18.960 --> 00:22:33.119
So we have like two running there, but in ATC we have like another one running all the time, which is the data quality, because data quality and data is the most heart of all this automotive industry.
00:22:33.200 --> 00:22:38.880
So Airflow coordinates those steps and records those inputs and outputs with the agent with the help of the agent.
00:22:38.960 --> 00:22:43.599
So whole agent behavior is more of like absorbable and reproducible.
00:22:43.680 --> 00:22:44.799
We control those tools.
00:22:44.880 --> 00:22:58.160
So uh it's it acts up on the human approval, uh, and if there's a sensitive step coming through, it stops execution based on the confidence level or a validation fails, then it triggers us.
00:22:58.319 --> 00:23:06.960
So agent is giving us a contextual reasoning, and airflow is giving the operation discipline for the whole um 82 drive train.
00:23:07.119 --> 00:23:08.480
I would say interesting.
00:23:08.640 --> 00:23:11.920
So it's just like a smarter data quality check.
00:23:12.240 --> 00:23:14.640
Yes, it has some reasoning behind it.
00:23:14.799 --> 00:23:16.000
Yeah, that's really cool.
00:23:16.160 --> 00:23:18.799
I haven't actually heard of somebody doing that yet.
00:23:19.200 --> 00:23:19.519
Yep.
00:23:19.759 --> 00:23:23.599
Uh, this was actually a project brought by someone from Europe.
00:23:23.759 --> 00:23:30.160
Actually, he uh like he eliminated like a lot of work of QE engineers.
00:23:30.480 --> 00:23:32.960
So that was like this agent.
00:23:33.039 --> 00:23:44.400
Uh, we may think it's a small data quality agent, but especially working in ATC drivetrend, we have like almost, I would say 17,000 rows of data every hour, like literally flowing.
00:23:44.480 --> 00:23:48.400
So data quality is like it may be a work of like six to seven people.
00:23:48.480 --> 00:23:53.359
So that agent is saving a lot of money any downtime or any triggers or anything.
00:23:53.440 --> 00:23:53.920
Yeah.
00:23:54.240 --> 00:23:55.680
Yeah, absolutely.
00:23:56.000 --> 00:23:57.599
Well, very interesting.
00:23:57.920 --> 00:23:59.039
Uh cool.
00:23:59.119 --> 00:24:01.759
Well, yeah, thank you for going through all of this.
00:24:01.920 --> 00:24:13.839
It's been, like I said, this is a really hot topic within the Airflow community right now, and it's always helpful for uh folks to get to hear how somebody is implementing this stuff.
00:24:14.720 --> 00:24:20.880
I mean, I've heard like everyone talk about this AI agents, you know, something's wrong or something's correct.
00:24:20.960 --> 00:24:25.119
You know, some people say it's saved on their side, some people say like it's out of control.
00:24:25.279 --> 00:24:28.400
It depends on how we use put boundaries and how you use it.
00:24:28.559 --> 00:24:34.720
And even for Airflow, Airflow saves a lot of time for especially for recovery or you know, capabilities.
00:24:34.880 --> 00:24:38.880
It's it's more of like the smaller team, the larger team, how to use it.
00:24:38.960 --> 00:24:40.319
That's all at the end of the day.
00:24:40.799 --> 00:24:41.279
Absolutely.
00:24:41.440 --> 00:24:46.160
All about the implementation, and as you mentioned, very importantly the guardrails in this case.
00:24:46.480 --> 00:24:46.880
Yeah.
00:24:47.119 --> 00:24:47.440
Yeah.
00:24:48.640 --> 00:24:48.960
Cool.
00:24:49.039 --> 00:24:58.640
Well, I think that's a good place to pivot to my standard final question, uh, which is what would you most like to see from development of the Airflow project in the near future?
00:24:59.119 --> 00:25:07.039
Uh, one of the stronger is like Airflow should continue on stronger capabilities around observability, event-driven orchestration.
00:25:07.200 --> 00:25:30.319
And also, I would ask them to more focus on the AI agent governance because for AI agent, it's uh I would like to standard as ways to track prompts, tool calls, you know, model versions, approvals, confidence, maybe more on agent outputs as a part of the task history that would make easier, you know, the for the agent workflows to be easier to audit it, operate safely for AFLO.
00:25:30.400 --> 00:25:37.039
So I would want it to be more flexible, even-driven execution, you know, for dynamic log running workflows.
00:25:37.200 --> 00:25:43.839
And I mean, AFLO is already a powerful orchestrator, like it's a very based on governance and everything.
00:25:44.000 --> 00:25:52.319
But I would have more on like AI assisted workflows to make it more important for especially like enterprise AI operations, which we use.
00:25:52.559 --> 00:25:53.920
So maybe that.
00:25:54.240 --> 00:25:55.039
Yeah, for sure.
00:25:55.200 --> 00:26:01.599
Sounds like some requests for future development in the common AI provider, which I know is moving forward quite quickly right now.
00:26:01.680 --> 00:26:05.279
They keep adding it, it feels like they add new functionality to that every day.
00:26:05.440 --> 00:26:06.640
But yeah, that's true.
00:26:06.720 --> 00:26:14.319
Like it's hard to track, also, you know, like they just do some few releases and it's really hard, but they're doing like very quickly, also.
00:26:14.400 --> 00:26:14.640
Yep.
00:26:14.799 --> 00:26:15.759
Yeah, yeah.
00:26:16.079 --> 00:26:16.480
Cool.
00:26:16.640 --> 00:26:17.920
Well, I love that answer.
00:26:18.000 --> 00:26:20.240
Um, Kamuda, thank you so much for going through that.
00:26:20.319 --> 00:26:21.680
It's been really great speaking with you.
00:26:21.839 --> 00:26:24.880
Just to close this out, what is the best way for folks to get in touch with you?
00:26:25.200 --> 00:26:28.400
Uh, they can reach out to me through LinkedIn, uh, Kumudashrinivas.
00:26:28.799 --> 00:26:37.519
So that's my whole linked it, or they can just shoot me a mail at Kumudashunivas and Gmail.com if you have any questions or you want to reach out for anything else.
00:26:37.599 --> 00:26:37.920
Yeah.
00:26:38.240 --> 00:26:38.559
Great.
00:26:38.640 --> 00:26:40.640
We can throw links to those in the show notes.
00:26:40.799 --> 00:26:41.680
Well, thank you again.
00:26:41.759 --> 00:26:43.440
This has been a great conversation.
00:26:44.079 --> 00:26:47.839
Thank you so much for having me here, and it was a wonderful conversation with you.
00:26:47.920 --> 00:26:48.240
Yeah.
00:26:48.640 --> 00:26:51.599
Thank you for listening to the Data Flowcast.
00:26:51.759 --> 00:26:55.200
Check the show notes for links to everything mentioned today.
00:26:55.359 --> 00:26:59.920
If you're enjoying the podcast, subscribe so you never miss an episode.
00:27:00.160 --> 00:27:02.480
And leave us a five star review.
