WEBVTT
00:00:26.480 --> 00:00:28.320
$150 value.
00:00:28.399 --> 00:00:35.600
Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
00:00:35.759 --> 00:00:38.000
Now let's get into the show.
00:00:38.399 --> 00:00:39.520
Okay, hi everyone.
00:00:39.679 --> 00:00:41.359
Welcome to the Data Flowcast.
00:00:41.439 --> 00:00:47.359
I'm your host, Kenton Dannis, and today I'm joined by Chiyonk Jane, who is a senior ML engineer at Visa.
00:00:47.759 --> 00:00:49.359
Chiyonk, how are you doing?
00:00:49.840 --> 00:00:50.640
I'm doing good.
00:00:50.719 --> 00:00:51.280
How are you?
00:00:51.600 --> 00:00:52.159
I'm good.
00:00:52.320 --> 00:00:53.600
It's great to have you on.
00:00:53.679 --> 00:00:55.600
I'm excited to dig into it.
00:00:55.759 --> 00:01:02.159
Uh for today's podcast, we're going to discuss orchestrating local LLMs for sensitive data pipelines with Airflow.
00:01:02.399 --> 00:01:05.680
Of course, this is a very popular topic right now.
00:01:06.239 --> 00:01:12.959
Lots of talk within the Airflow community about using Airflow for AI orchestration of various kinds.
00:01:13.040 --> 00:01:16.159
So I'm really excited to get your perspective on that.
00:01:16.480 --> 00:01:21.439
Before we dive into all of the details, maybe let's start with a little bit of background.
00:01:21.599 --> 00:01:28.480
Tell me about yourself and you know how you got into ML engineering and what you do with Airflow and data pipelines.
00:01:29.280 --> 00:01:29.680
Sure.
00:01:29.920 --> 00:01:36.079
So uh right now I am a senior ML engineer working on inference and LLM system in payments.
00:01:36.480 --> 00:01:40.079
A lot of my time goes in uh serving with Onyx runtime.
00:01:40.239 --> 00:01:43.280
Some some of this there is rust in the serving path.
00:01:43.439 --> 00:01:48.640
But lately, a lot of my time and interest has been going in agentic tooling with Landgraph.
00:01:48.959 --> 00:01:55.840
But I exactly didn't come here with a mindset of building pipelines or something around that.
00:01:56.079 --> 00:02:00.239
I came in through modeling, not just uh uh not just infrastructure.
00:02:00.400 --> 00:02:06.239
Um my first job out of school was uh uh NLP, uh NLP engineer in pharma.
00:02:06.560 --> 00:02:13.520
Uh I built a tool with Spacey, which had which had medical terminology dictionary and compared drug side effects.
00:02:13.680 --> 00:02:22.560
Then I went into risk modeling, then I did uh masters in computer science and spent a few years in clinical machine learning.
00:02:22.719 --> 00:02:30.800
Then I led a five time a five uh five people team building re-admission risk model for electronic health system.
00:02:31.599 --> 00:02:51.680
And that's exactly where my uh my mind shifted, and that's exactly the job that turned me into a pipeline person because we pulled data from uh different hospitals, some were running on Epic, some were running on someone running on Cerner, and there were different formats, different delivery time.
00:02:52.159 --> 00:03:08.560
And I realized I was able to write the core model logic in a few months or um five or six weeks, but I spent a lot more time than expected uh with uh airflow dash, data quality, and backsfill.
00:03:08.879 --> 00:03:19.680
And that's where it clicked to me that uh model is just one task, and there let's say there are 40 tasks, model is just one task, and there are 39 others that I need to look into.
00:03:20.159 --> 00:03:22.240
So that's that's how it came here.
00:03:22.800 --> 00:03:37.599
Yeah, that's a great summary, I think, of one of the challenges with AI orchestration is people sort of think about it as the model that you're working with, but that's really just a small piece of the overall picture, and orchestration helps with a lot of that other stuff.
00:03:37.840 --> 00:03:39.120
So, yeah, very cool.
00:03:39.360 --> 00:03:42.639
And tell me a little bit more about your current work.
00:03:42.800 --> 00:03:46.479
So you're a senior ML engineer, you're focused on these LLM systems.
00:03:46.639 --> 00:03:48.879
What does that look like in your day-to-day?
00:03:49.919 --> 00:03:55.360
So, in day-to-day, uh, let's say, like let me divide it into uh few categories.
00:03:55.599 --> 00:04:12.080
So, one one of uh so some some of my time goes in serving, getting to uh getting a model to run fast, uh identically every time, and managing the runtime session, tuning, batching, memory behavior, keeping the tail latency flat.
00:04:12.240 --> 00:04:14.159
That takes a lot of time and thinking.
00:04:14.400 --> 00:04:17.439
Then, very important next part is failure design.
00:04:17.600 --> 00:04:20.800
So, let's say everything is not ideal in this world.
00:04:20.879 --> 00:04:22.879
So, what happened when the model times out?
00:04:22.959 --> 00:04:24.879
Uh what when the tool call fails?
00:04:24.959 --> 00:04:27.759
Uh, what happens when the output parsing is wrong?
00:04:28.000 --> 00:04:34.079
So, building that failure design so we can rely on that's the second part where I spend a lot of time.
00:04:34.240 --> 00:04:39.040
And the third, with the new way the AI is changing, that's agentic tooling.
00:04:39.199 --> 00:04:47.600
Uh, let's say we build a land graph pipeline with multiple MCP tools, uh let observability setup, benchmarking, a lot of this stuff.
00:04:47.920 --> 00:04:56.160
But uh we need to make sure the tools are getting uh called properly in the right order, and we are getting the expected result.
00:04:56.720 --> 00:05:02.000
And then we work with Airflow, Kubernetes, Grafana, all threaded all together.
00:05:02.319 --> 00:05:10.959
Model maybe one-fifth of my work, but a lot of uh a lot of my time goes in uh surveying failure design and agent tooling.
00:05:12.079 --> 00:05:13.199
Okay, got it.
00:05:13.279 --> 00:05:14.480
That makes sense.
00:05:14.800 --> 00:05:24.800
And then you spoke recently at the Airflow Summit about, I think, some of this work on orchestrating local LLMs for sensitive data pipelines.
00:05:24.879 --> 00:05:31.120
So let's talk a little bit more about that aspect of your work, especially around the sensitive data piece.
00:05:31.439 --> 00:05:37.920
Uh, tell me before we talk about the implementation, kind of what is the core problem that you're trying to solve here?
00:05:39.120 --> 00:05:39.439
Sure.
00:05:39.519 --> 00:05:47.439
Uh so let me take an example of it can be any sensitive field, be it government, healthcare, finance.
00:05:47.839 --> 00:05:51.759
Uh let me take an example of a healthcare field.
00:05:52.000 --> 00:06:02.079
So uh let's say we have uh enormous clinical text data, just our summary, nursing node, physician narrative, and there are a lot of other free test text.
00:06:02.399 --> 00:06:05.519
And these these are unstructured records.
00:06:05.680 --> 00:06:11.439
We might have a structured record which gives us the ICD code, but they don't give us the real story behind it.
00:06:11.680 --> 00:06:14.240
The real story comes from the unstructured data.
00:06:14.480 --> 00:06:20.800
Let's say unstructured node says the patient lives alone and they don't have any transportation to follow up to the clinic.
00:06:21.199 --> 00:06:26.639
That's where the real that's where the real knowledge is.
00:06:27.040 --> 00:06:40.000
And we need text turned into a structured uh structured features for a model, uh, be it financial model, healthcare model, uh, there can be notes from there can be notes out of financial data as well.
00:06:40.160 --> 00:06:45.199
So that's the classic extraction problem, and LLMs are generally good at it.
00:06:45.439 --> 00:06:58.720
But the blocker, when it comes to sensitive data, they have PHIs, protected health information, they have uh they have credit card information, or the person who is making that transaction, they have information about that.
00:06:58.959 --> 00:07:03.120
So the first is you cannot make that data leave out of the system.
00:07:03.360 --> 00:07:08.959
So calling a hosted API is off the table at all, not on the cost, not on the quality.
00:07:09.279 --> 00:07:11.759
So here the problem becomes this.
00:07:11.839 --> 00:07:26.879
Uh the problem becomes run an open weight model like Lama 3 on your own GPU over million of nodes, let's say every night on a on a schedule, with enough checking that the downstream clinical uh model is allowed to depend on it output.
00:07:27.040 --> 00:07:28.480
It's a very sensitive field.
00:07:28.639 --> 00:07:31.199
So people should be able to depend on that.
00:07:31.439 --> 00:07:39.439
And the last uh the last clauses where all it works, getting a Lama 3 to summarize one of the nodes nicely is just a Tuesday afternoon.
00:07:39.600 --> 00:07:46.079
Getting it down to 2 million every night, reproducibility with a paper tail is the real system.
00:07:46.160 --> 00:07:52.959
That's exactly what the whole talk is about, and how we can do it in a good engineering way.
00:07:53.920 --> 00:07:55.759
Yeah, that makes sense with that data.
00:07:55.839 --> 00:08:04.879
That with uh anything super sensitive, you can't send it to external systems, and the traceability and kind of data quality aspects become really, really important.
00:08:05.759 --> 00:08:10.079
So then tell me about where Airflow fits into this architecture.
00:08:11.199 --> 00:08:11.680
Sure.
00:08:12.000 --> 00:08:18.319
So Airflow sits on top of it, like it never touches the data, but it orchestrates it.
00:08:18.399 --> 00:08:21.680
So it's one of the most important parts.
00:08:22.079 --> 00:08:40.480
Let's say the heavy lifting is done by GPU ports on Kubernetes that run the model, object storage holds the uh raw nodes, Postgres holds the result, but Airflow is the one which decides what's to what to run, in what order, and what happens if any of this uh service breaks.
00:08:40.799 --> 00:08:44.159
So if I have to divide, let we can divide it into four jobs.
00:08:44.320 --> 00:08:45.679
One is the sensors.
00:08:45.919 --> 00:08:52.960
So we say every site, let's say every site drops files on its own schedule, and some nights there might not be any files.
00:08:53.120 --> 00:08:58.799
So there is a sensor per site waiting on its data, and each site has its own task.
00:08:58.960 --> 00:09:03.600
So if one site is four hours later, only that branch waits.
00:09:03.679 --> 00:09:09.679
So that's what AFLO comes into picture and it decides which branch to wait, which branch to run.
00:09:09.919 --> 00:09:14.879
And when it when I talk about branches, second thought can come about dependencies.
00:09:15.120 --> 00:09:21.519
Validation directly cannot start until uh validation cannot start until inference for that batch finishes.
00:09:21.679 --> 00:09:24.960
So load cannot start until validation passes.
00:09:25.360 --> 00:09:35.840
That ordering itself is the entire safety design and it literally lives inside the DAG instead of somebody's head, and Airflow is the one that's orchestrating it.
00:09:36.399 --> 00:09:39.039
Next is the retries and backfills.
00:09:39.200 --> 00:09:42.080
What if uh the site uh delivers one day late?
00:09:42.399 --> 00:09:46.799
Or uh let's say there was a there was a failure.
00:09:46.960 --> 00:09:50.080
So we need to make sure we don't retry everything.
00:09:50.320 --> 00:09:56.639
Airflow takes care of retrying that specific node or backfilling that specific uh specific data.
00:09:56.799 --> 00:09:59.440
So you don't need to rerun everything again.
00:09:59.840 --> 00:10:02.240
And the last is the run record.
00:10:02.320 --> 00:10:10.639
When DAG runs with with configs, uh, we need to know which config is it, which model version it's producing, and that's the linear start.
00:10:10.720 --> 00:10:15.759
And you get it free because uh Airflow is tracking every run for it.
00:10:15.919 --> 00:10:26.320
So this is where uh Airflow is an integral part of any orchestration that we do, be it uh LLMs or traditional uh ML systems.
00:10:27.519 --> 00:10:28.960
Yeah, that's interesting.
00:10:29.120 --> 00:10:37.840
I think you're describing a lot of the very bread and butter airflow orchestration functionality, and I love how you've talked about how it's so important for this use case.
00:10:37.919 --> 00:10:38.720
That's great.
00:10:39.039 --> 00:10:43.919
I'm wondering if you can give me any more detail on how you're structuring those DAGs.
00:10:44.000 --> 00:10:49.440
So you're managing quite a lot with them from the LLM inference and the validation and those ingestion stages.
00:10:49.600 --> 00:10:51.600
What do those DAGs look like?
00:10:53.120 --> 00:10:57.279
So, yeah, the uh DAGs can have multiple nodes, a lot of connections.
00:10:57.440 --> 00:11:05.279
There can be multiple stages, but if I have to divide it into some core buckets, I can divide it into four stages.
00:11:05.519 --> 00:11:08.960
Stage one, we can say we can ingest and normalize it.
00:11:09.039 --> 00:11:12.720
So one task per group site, uh, we can run all in parallel.
00:11:12.879 --> 00:11:24.240
Each one lands raw nodes in a big object storage, let's say S3, and writes those manifest rows so you know what arrived, when it arrived, and how it arrived.
00:11:24.480 --> 00:11:28.639
Once we have all that data, second can be the inference.
00:11:29.039 --> 00:11:40.879
So all these nodes or the data, it can get grouped in badges and run on GPU, GPU pods, and that can be done through Kubernetes pod of uh operators.
00:11:41.039 --> 00:11:43.840
So the GPU works in isolation from the scheduler.
00:11:43.919 --> 00:11:47.840
That's very important to keep things isolated at their own level.
00:11:48.000 --> 00:11:55.440
And this stage sits in its own air uh airport, sorry, airflow pool uh with no hard concurrency cap.
00:11:55.759 --> 00:12:02.639
So here we need to make sure uh we don't have a lot of tasks and less number of GPUs.
00:12:02.960 --> 00:12:06.080
So that's uh where we need to configure it properly.
00:12:06.240 --> 00:12:08.159
Uh stage three can be validation.
00:12:08.240 --> 00:12:12.720
So uh let's say we have the data, our inference has uh already ran properly.
00:12:12.879 --> 00:12:23.279
Now we need to make sure the output that is being generated goes through a parentic schema, then cross-checks against the structure record, and we can make sure the output is correct.
00:12:23.519 --> 00:12:26.879
And the final one we can talk about that's load.
00:12:27.039 --> 00:12:31.840
So uh we load those all the output into a feature store.
00:12:32.000 --> 00:12:44.960
So a final model that's serving a readmission or fraud detection, whatever, that can read from the data feature store and give a good, well-structured output or a scoring.
00:12:45.200 --> 00:12:52.000
So these those are the four main stages where we are using Airflow and how we are using it.
00:12:53.279 --> 00:12:54.960
Okay, interesting.
00:12:55.200 --> 00:13:01.279
And then I'd also love to dig a little bit deeper into how you're handling failures.
00:13:01.360 --> 00:13:08.399
So you touched on this a little bit earlier uh with using Airflow to manage retries and things like that.
00:13:08.559 --> 00:13:15.919
But tell me in a little bit more detail if something goes wrong, or you mentioned you have a lot of data quality checks built in.
00:13:16.000 --> 00:13:20.879
Uh, when there's also this LLM in the loop that you're managing, how does all of that work?
00:13:22.080 --> 00:13:22.480
Yes.
00:13:22.639 --> 00:13:28.799
So before directly uh diving into the failures, we can divide it into two types of failure.
00:13:29.039 --> 00:13:31.840
One can be the easier one, that's the loud failure.
00:13:32.000 --> 00:13:36.320
The pod gets OOM killed, the GPU is busy, the model timeout.
00:13:36.720 --> 00:13:39.200
That is an ordinary airflow failure.
00:13:39.360 --> 00:13:44.000
We can retry with exponential fallbacks, we can do task isolation as I mentioned.
00:13:44.240 --> 00:13:46.320
Task isolation is very important.
00:13:46.480 --> 00:13:53.200
We can rerun the task and make sure that the things are not overwritten and we are replacing it with the correct output.
00:13:53.360 --> 00:13:54.720
So those are loud failures.
00:13:54.799 --> 00:13:59.759
We can go and check the system and see everything is working fine or if there is a failure.
00:14:00.399 --> 00:14:03.759
The category two, that's the scarier one.
00:14:04.080 --> 00:14:10.639
Because here, let's assume model runs, it returns a clean output, and nothing is wrong.
00:14:10.960 --> 00:14:15.120
For a normal person, everything is green, the graph is green, everything is looking good.
00:14:15.360 --> 00:14:17.279
That's a good picture.
00:14:18.000 --> 00:14:22.480
But here, uh there are a lot of scenarios that can happen.
00:14:22.720 --> 00:14:30.799
Uh, let's say uh the output is not correctly in not in a correct process or not in a correct format.
00:14:31.039 --> 00:14:33.200
So here we have constraint output.
00:14:33.279 --> 00:14:36.000
Uh the model fills in the uh piedenic schema.
00:14:36.080 --> 00:14:38.879
We cannot have a free text, so it does not pass.
00:14:39.120 --> 00:14:47.519
The record the record is injected, but if it's not in a piedentic schema, we do not let it uh go to a next stage.
00:14:47.600 --> 00:14:50.320
So that is how we check this silent failure.
00:14:50.399 --> 00:14:55.120
We have piedantic for this, then we can check against the structure record.
00:14:55.360 --> 00:15:08.559
So let's say if Note is extracting something about a heart failure and the core diagnosis doesn't have anything about it, so we can write some logic to check this kind of issues.
00:15:08.879 --> 00:15:10.879
Then we can do a distribution check.
00:15:11.039 --> 00:15:24.639
Let's say uh for any fraud detection or for any kind of let's say for fraud detection or any kind of anomaly, uh every day it's just 2%, but suddenly we saw it shot up to 8%.
00:15:25.279 --> 00:15:30.320
Everything is on a rosy picture, everything is green, but here you see there is some anomaly.
00:15:30.480 --> 00:15:34.000
So we need to go in manually and check what's going wrong.
00:15:34.159 --> 00:15:36.399
So that is something we need to debug.
00:15:36.720 --> 00:15:41.039
And initially, when you are setting a system, it's in a dev stage and not prod.
00:15:41.600 --> 00:15:48.720
We can have a human in loop who uh checks every X sample and makes sure what we are outputting is correct.
00:15:48.960 --> 00:15:58.159
So the silent failures are more challenging, and these are some of the ways we can handle them, but still we might miss at some stage.
00:15:58.240 --> 00:16:01.279
But currently, this is the most challenging part in the industry.
00:16:02.399 --> 00:16:03.600
Yeah, that makes sense.
00:16:03.759 --> 00:16:10.799
And how do you go about trying to solve those the silent failures, as you've said, with Airflow?
00:16:10.879 --> 00:16:13.120
Like where do you think Airflow fits in there?
00:16:13.279 --> 00:16:17.679
Because I think everybody probably knows how you can use Airflow to manage the loud ones.
00:16:17.919 --> 00:16:19.440
As you say, that's really easy.
00:16:19.600 --> 00:16:24.879
Obviously, harder to build in some logic to manage the softer failures.
00:16:25.679 --> 00:16:30.960
So we we can at each step we know what are the outputs, what are the inputs.
00:16:31.120 --> 00:16:50.080
So we can have some Chrome jobs or Python script as one of the nodes, which uh which after a specific uh uh output, it checks uh after a specific step, it checks if uh the output is correct and it will go through the as I say, it will go through the pyentic validation.
00:16:50.159 --> 00:16:52.320
So that can be one of the nodes in the airflow.
00:16:52.559 --> 00:17:00.960
So putting those check notes in the whole orchestration, that can be how that that's one of the ways to check these silent flows.
00:17:01.360 --> 00:17:01.840
Got it.
00:17:01.919 --> 00:17:06.160
So you're building in data quality checks of a sort into the pipelines.
00:17:06.400 --> 00:17:07.680
Yeah, that makes sense.
00:17:09.119 --> 00:17:17.359
This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:17:17.759 --> 00:17:21.680
From batch pipelines to training models and wiring up agents.
00:17:21.920 --> 00:17:27.920
Astro handles the infrastructure upgrades and scaling so you can focus on building.
00:17:28.240 --> 00:17:30.480
Check it out at astronomer.io.
00:17:32.960 --> 00:17:34.000
Okay, great.
00:17:34.240 --> 00:17:45.680
And then I guess on a slightly different but related topic, I understand that part of your approach uh to managing these pipelines includes fine-tuning.
00:17:45.839 --> 00:17:53.599
And so I'm curious if you could tell me more about what that looks like and where that fits into your Airflow pipelines.
00:17:54.720 --> 00:17:55.119
Sure.
00:17:55.440 --> 00:18:06.480
So uh as you mentioned, it's a little bit uh away from the core DAC topics, but uh uh let's think it's not something that we need to run every night.
00:18:06.799 --> 00:18:11.359
Fine-tuning is not something we we need every day or every night or every week.
00:18:11.680 --> 00:18:14.160
It is it has his own Airflow DAC.
00:18:14.400 --> 00:18:28.960
It doesn't sit in the core flow and it fires maybe once a month or maybe once in a quarter when we have enough new data piled up so we can fine-tune the already fine-tuned model we have.
00:18:29.119 --> 00:18:38.960
So it pulls the label set, runs a lot of fine-tuning with Hugging Face, PEFT on top of Lama 3, then scores it against the frozen holdout.
00:18:39.200 --> 00:18:45.839
And if it only if it beats what we have, it writes a new adapter version to the model registry.
00:18:46.079 --> 00:18:54.160
So that is something, that is something has to be a totally different Airflow DAC apart from the gore one which we are working.
00:18:55.359 --> 00:19:10.720
And so Nightly Drag uh, and one more thing is the Nightly DAC trains nothing, it just loads the base Lama 3 weight, loads the current adapter, file uh loads the current adapter file next to them and runs runs it basically uh to the old files.
00:19:10.960 --> 00:19:14.960
And the adapter is maybe let's say a few hundred megabytes on the disk.
00:19:15.200 --> 00:19:22.079
So all of the fine tuning cost you uh you run is just about reading one extra file.
00:19:22.240 --> 00:19:23.119
That's it.
00:19:23.440 --> 00:19:34.559
And one more thing where Airflow can fit perfectly here is when we write to these adapters and we keep updating the record, Airflow helps us to maintain different versions with it.
00:19:34.720 --> 00:19:42.960
So six months later down the line, you can look back, row, uh look back at a row in a feature, and we can know exactly which adapter produced it.
00:19:43.119 --> 00:19:51.519
And if we want to go to the specific version, we can use Airflow, run it across along with that adapter and get the same result.
00:19:51.759 --> 00:20:00.240
So, in this sensitive type of field, if there is an audit or something, we'll have a very good trace, and that is because of Airflow, how we are managing it.
00:20:00.640 --> 00:20:02.400
Okay, that's really cool.
00:20:02.559 --> 00:20:09.039
Uh, I have heard of a few other use cases of people using Airflow to manage fine-tuning like this, but not a ton.
00:20:09.200 --> 00:20:12.079
So this is great to hear your perspective on.
00:20:12.400 --> 00:20:20.559
And then tell me more about so when you're fine-tuning the models, are you looking for better performance?
00:20:20.720 --> 00:20:29.680
Are you looking for better, or I should say, performance in terms of accuracy or performance in terms of speed or efficiency, or is it all of the above?
00:20:30.880 --> 00:20:34.160
It's mostly on term uh terms of accuracy.
00:20:34.480 --> 00:20:53.759
It's uh because we have Lama 3, it's a very good model on uh normal English, but when it comes to a specific domain, let's say financial domain or a medical domain, it lacks a little bit because there is a context to because it's a context uh that it's lacking.
00:20:54.000 --> 00:21:00.400
So when we talk about fine-tuning, we take we freeze the core weights and just train on the last layer.
00:21:00.559 --> 00:21:02.079
So, let me give you an example.
00:21:02.319 --> 00:21:09.680
Let's say uh doctors talk in a language, which say PT denies CP, and that means patients say they have no chest pain.
00:21:09.839 --> 00:21:14.160
So there is a very small word that says does not have chest pain.
00:21:14.319 --> 00:21:17.680
And there is some acronyms, PT, CP, all those stuff.
00:21:17.839 --> 00:21:22.640
So those get lost when it's being trained on normal English.
00:21:22.880 --> 00:21:30.480
And when we fine-tune on uh medical data, we can pick all of those and we can have a much more accurate output.
00:21:31.440 --> 00:21:32.000
Got it.
00:21:32.160 --> 00:21:33.359
That makes sense.
00:21:33.680 --> 00:21:46.319
And then I guess zooming back a little bit and looking at this high level, what were the hardest lessons you learned when trying to productionize this and getting it to run at scale?
00:21:46.880 --> 00:21:49.119
So, first is reproducibility.
00:21:49.279 --> 00:21:58.079
So, what happens when we are uh when we are working at a at a lightning speed with all the AI assisted coding and everything?
00:21:58.240 --> 00:21:59.759
We forget how to train.
00:22:00.319 --> 00:22:08.480
Back and how to uh if something fails, or uh how to trace back how we have iterated and how did we come in so far.
00:22:08.960 --> 00:22:13.440
So let's say uh we uh we need to pin those stuff.
00:22:13.599 --> 00:22:19.200
We need to pin, uh, we need to pin the temperature because there can be a lot of variance.
00:22:19.440 --> 00:22:26.000
The feature starts to drift if we don't pin the model version, we don't pin the features, we don't pin the temperature.
00:22:26.160 --> 00:22:28.079
So that is one important thing.
00:22:28.319 --> 00:22:38.559
Make sure every time you are running a model or any orchestration, pin everything versions, uh versions, seed, features, everything.
00:22:38.880 --> 00:22:41.359
Next is about the GPUs.
00:22:41.680 --> 00:22:48.799
So uh a lot of time uh what we did, we scheduled a lot more tasks in parallel than we had GPUs.
00:22:49.200 --> 00:22:56.000
So even if the tasks are in parallel, we don't have enough resources for them to run in parallel.
00:22:56.160 --> 00:23:03.039
So make sure that the amount of infra you have, you parallelize your orchestration based on that.
00:23:03.200 --> 00:23:05.519
You don't over-parallelize or over-complicate it.
00:23:05.680 --> 00:23:07.039
That's the second thing.
00:23:07.359 --> 00:23:12.480
And the third one that matters the most is the traceability.
00:23:12.799 --> 00:23:18.160
How we can look six months back and say how we reach to that particular output.
00:23:18.400 --> 00:23:29.359
So uh here we need to make sure uh that uh we have a proper feature storage, we have a proper isolation, and every service has a very good trace logs.
00:23:29.599 --> 00:23:45.279
And if anything fails, we can isolate it and we can we can isolate it and make sure that uh uh we can isolate it and make sure that uh some specific thing failed here, and that's the reason these results are broken.
00:23:45.440 --> 00:23:48.559
So these are the three things that I think is most important.
00:23:48.960 --> 00:23:57.759
Yeah, I think those are great things for everybody, maybe even using LLMs or not to think about when trying to productionize stuff with Airflow.
00:23:57.920 --> 00:24:00.079
So, yeah, that's great insight there.
00:24:00.559 --> 00:24:03.920
Okay, well, Chang, thank you for going through all of that.
00:24:04.079 --> 00:24:10.000
I again think this is a super interesting use case that a lot of our listeners will benefit from hearing about.
00:24:10.160 --> 00:24:20.559
Uh, in the interest of time, I will wrap us up and I always like to end with a standard final question, and that is what would you most like to see from development of the Airflow project in the near future?
00:24:22.720 --> 00:24:28.160
Yes, so if I have to look into it right now, uh Airflow has been really useful.
00:24:28.240 --> 00:24:40.880
And with Airflow 3 coming and it's already out, it has been really helpful for developer, but few things if I had to uh pinpoint, then we can have the GPU awareness uh in Airflow itself.
00:24:41.039 --> 00:24:51.440
So right now, if our task needs a GPU, you can express it sideways through pools, cues, and naming convention, and every team reinvents it slightly differently.
00:24:51.680 --> 00:24:59.440
So I would like a task to declare that if it needs a GPU for 20 minutes and have the shed and let the scheduler honor it.
00:24:59.680 --> 00:25:03.599
It is the piece that I have hand-rolled on every project I have done.
00:25:03.759 --> 00:25:12.480
So instead of each developer or each team hand holding uh hand-rolling it, we can have a first-class awareness GPU inbuilt in Airflow system.
00:25:13.039 --> 00:25:21.519
And second is treat model as assets basically, because a lot of time, a lot of teams now are modeling and they are using Airflow for that.
00:25:21.759 --> 00:25:28.000
So I know Airflow already understands that a data set is produced by one task and consumed by another.
00:25:28.160 --> 00:25:33.119
Uh and a base model and a LoRa adapter are exactly that shape.
00:25:33.279 --> 00:25:40.960
So if assets covered a model version, uh, then which model version produced this table becomes a question Airflow can answer.
00:25:41.119 --> 00:25:44.400
And a new developer doesn't have to come with their own versioning.
00:25:44.559 --> 00:25:50.240
So I think these two are the main thing treating model as assets and having more first-class GPU awareness.
00:25:50.720 --> 00:25:51.839
Yeah, got it.
00:25:52.000 --> 00:25:54.160
I think those are both great requests.
00:25:54.480 --> 00:25:57.599
Okay, well, again, thank you so much for going through all this.
00:25:57.680 --> 00:25:58.880
I think this has been great.
00:25:59.039 --> 00:26:02.559
To close us out, what is the best way for folks to get in touch with you?
00:26:02.960 --> 00:26:06.160
Uh I think LinkedIn is the best way that they can get in touch with me.
00:26:06.400 --> 00:26:10.799
And uh I can also have my email ID in it, so LinkedIn and email ID.
00:26:10.960 --> 00:26:11.119
Okay.
00:26:11.920 --> 00:26:15.839
And later I'll be I'll be having a talk at Airflow Summit.
00:26:15.920 --> 00:26:17.839
So more than open to seeing people there.
00:26:18.160 --> 00:26:18.799
Yeah, great.
00:26:18.960 --> 00:26:25.920
Well, I think this episode will probably be released after Airflow Summit since we're in very short time for Airflow Summit this year.
00:26:26.000 --> 00:26:28.720
Uh, but hopefully some folks will have seen you there.
00:26:29.039 --> 00:26:31.839
And yeah, we'll throw links to those other things in the show notes.
00:26:32.000 --> 00:26:34.319
So uh Chang, thank you again so much.
00:26:34.480 --> 00:26:35.680
This has been great.
00:26:36.079 --> 00:26:37.759
Yeah, nice, nice meeting you.
00:26:38.160 --> 00:26:40.880
Thank you for listening to the Data Flowcast.
00:26:41.039 --> 00:26:44.559
Check the show notes for links to everything mentioned today.
00:26:44.720 --> 00:26:49.279
If you're enjoying the podcast, subscribe so you never miss an episode.
00:26:49.519 --> 00:26:51.759
And leave us a five-star review.
00:00:26.480 --> 00:00:28.320
$150 value.
00:00:28.399 --> 00:00:35.600
Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
00:00:35.759 --> 00:00:38.000
Now let's get into the show.
00:00:38.399 --> 00:00:39.520
Okay, hi everyone.
00:00:39.679 --> 00:00:41.359
Welcome to the Data Flowcast.
00:00:41.439 --> 00:00:47.359
I'm your host, Kenton Dannis, and today I'm joined by Chiyonk Jane, who is a senior ML engineer at Visa.
00:00:47.759 --> 00:00:49.359
Chiyonk, how are you doing?
00:00:49.840 --> 00:00:50.640
I'm doing good.
00:00:50.719 --> 00:00:51.280
How are you?
00:00:51.600 --> 00:00:52.159
I'm good.
00:00:52.320 --> 00:00:53.600
It's great to have you on.
00:00:53.679 --> 00:00:55.600
I'm excited to dig into it.
00:00:55.759 --> 00:01:02.159
Uh for today's podcast, we're going to discuss orchestrating local LLMs for sensitive data pipelines with Airflow.
00:01:02.399 --> 00:01:05.680
Of course, this is a very popular topic right now.
00:01:06.239 --> 00:01:12.959
Lots of talk within the Airflow community about using Airflow for AI orchestration of various kinds.
00:01:13.040 --> 00:01:16.159
So I'm really excited to get your perspective on that.
00:01:16.480 --> 00:01:21.439
Before we dive into all of the details, maybe let's start with a little bit of background.
00:01:21.599 --> 00:01:28.480
Tell me about yourself and you know how you got into ML engineering and what you do with Airflow and data pipelines.
00:01:29.280 --> 00:01:29.680
Sure.
00:01:29.920 --> 00:01:36.079
So uh right now I am a senior ML engineer working on inference and LLM system in payments.
00:01:36.480 --> 00:01:40.079
A lot of my time goes in uh serving with Onyx runtime.
00:01:40.239 --> 00:01:43.280
Some some of this there is rust in the serving path.
00:01:43.439 --> 00:01:48.640
But lately, a lot of my time and interest has been going in agentic tooling with Landgraph.
00:01:48.959 --> 00:01:55.840
But I exactly didn't come here with a mindset of building pipelines or something around that.
00:01:56.079 --> 00:02:00.239
I came in through modeling, not just uh uh not just infrastructure.
00:02:00.400 --> 00:02:06.239
Um my first job out of school was uh uh NLP, uh NLP engineer in pharma.
00:02:06.560 --> 00:02:13.520
Uh I built a tool with Spacey, which had which had medical terminology dictionary and compared drug side effects.
00:02:13.680 --> 00:02:22.560
Then I went into risk modeling, then I did uh masters in computer science and spent a few years in clinical machine learning.
00:02:22.719 --> 00:02:30.800
Then I led a five time a five uh five people team building re-admission risk model for electronic health system.
00:02:31.599 --> 00:02:51.680
And that's exactly where my uh my mind shifted, and that's exactly the job that turned me into a pipeline person because we pulled data from uh different hospitals, some were running on Epic, some were running on someone running on Cerner, and there were different formats, different delivery time.
00:02:52.159 --> 00:03:08.560
And I realized I was able to write the core model logic in a few months or um five or six weeks, but I spent a lot more time than expected uh with uh airflow dash, data quality, and backsfill.
00:03:08.879 --> 00:03:19.680
And that's where it clicked to me that uh model is just one task, and there let's say there are 40 tasks, model is just one task, and there are 39 others that I need to look into.
00:03:20.159 --> 00:03:22.240
So that's that's how it came here.
00:03:22.800 --> 00:03:37.599
Yeah, that's a great summary, I think, of one of the challenges with AI orchestration is people sort of think about it as the model that you're working with, but that's really just a small piece of the overall picture, and orchestration helps with a lot of that other stuff.
00:03:37.840 --> 00:03:39.120
So, yeah, very cool.
00:03:39.360 --> 00:03:42.639
And tell me a little bit more about your current work.
00:03:42.800 --> 00:03:46.479
So you're a senior ML engineer, you're focused on these LLM systems.
00:03:46.639 --> 00:03:48.879
What does that look like in your day-to-day?
00:03:49.919 --> 00:03:55.360
So, in day-to-day, uh, let's say, like let me divide it into uh few categories.
00:03:55.599 --> 00:04:12.080
So, one one of uh so some some of my time goes in serving, getting to uh getting a model to run fast, uh identically every time, and managing the runtime session, tuning, batching, memory behavior, keeping the tail latency flat.
00:04:12.240 --> 00:04:14.159
That takes a lot of time and thinking.
00:04:14.400 --> 00:04:17.439
Then, very important next part is failure design.
00:04:17.600 --> 00:04:20.800
So, let's say everything is not ideal in this world.
00:04:20.879 --> 00:04:22.879
So, what happened when the model times out?
00:04:22.959 --> 00:04:24.879
Uh what when the tool call fails?
00:04:24.959 --> 00:04:27.759
Uh, what happens when the output parsing is wrong?
00:04:28.000 --> 00:04:34.079
So, building that failure design so we can rely on that's the second part where I spend a lot of time.
00:04:34.240 --> 00:04:39.040
And the third, with the new way the AI is changing, that's agentic tooling.
00:04:39.199 --> 00:04:47.600
Uh, let's say we build a land graph pipeline with multiple MCP tools, uh let observability setup, benchmarking, a lot of this stuff.
00:04:47.920 --> 00:04:56.160
But uh we need to make sure the tools are getting uh called properly in the right order, and we are getting the expected result.
00:04:56.720 --> 00:05:02.000
And then we work with Airflow, Kubernetes, Grafana, all threaded all together.
00:05:02.319 --> 00:05:10.959
Model maybe one-fifth of my work, but a lot of uh a lot of my time goes in uh surveying failure design and agent tooling.
00:05:12.079 --> 00:05:13.199
Okay, got it.
00:05:13.279 --> 00:05:14.480
That makes sense.
00:05:14.800 --> 00:05:24.800
And then you spoke recently at the Airflow Summit about, I think, some of this work on orchestrating local LLMs for sensitive data pipelines.
00:05:24.879 --> 00:05:31.120
So let's talk a little bit more about that aspect of your work, especially around the sensitive data piece.
00:05:31.439 --> 00:05:37.920
Uh, tell me before we talk about the implementation, kind of what is the core problem that you're trying to solve here?
00:05:39.120 --> 00:05:39.439
Sure.
00:05:39.519 --> 00:05:47.439
Uh so let me take an example of it can be any sensitive field, be it government, healthcare, finance.
00:05:47.839 --> 00:05:51.759
Uh let me take an example of a healthcare field.
00:05:52.000 --> 00:06:02.079
So uh let's say we have uh enormous clinical text data, just our summary, nursing node, physician narrative, and there are a lot of other free test text.
00:06:02.399 --> 00:06:05.519
And these these are unstructured records.
00:06:05.680 --> 00:06:11.439
We might have a structured record which gives us the ICD code, but they don't give us the real story behind it.
00:06:11.680 --> 00:06:14.240
The real story comes from the unstructured data.
00:06:14.480 --> 00:06:20.800
Let's say unstructured node says the patient lives alone and they don't have any transportation to follow up to the clinic.
00:06:21.199 --> 00:06:26.639
That's where the real that's where the real knowledge is.
00:06:27.040 --> 00:06:40.000
And we need text turned into a structured uh structured features for a model, uh, be it financial model, healthcare model, uh, there can be notes from there can be notes out of financial data as well.
00:06:40.160 --> 00:06:45.199
So that's the classic extraction problem, and LLMs are generally good at it.
00:06:45.439 --> 00:06:58.720
But the blocker, when it comes to sensitive data, they have PHIs, protected health information, they have uh they have credit card information, or the person who is making that transaction, they have information about that.
00:06:58.959 --> 00:07:03.120
So the first is you cannot make that data leave out of the system.
00:07:03.360 --> 00:07:08.959
So calling a hosted API is off the table at all, not on the cost, not on the quality.
00:07:09.279 --> 00:07:11.759
So here the problem becomes this.
00:07:11.839 --> 00:07:26.879
Uh the problem becomes run an open weight model like Lama 3 on your own GPU over million of nodes, let's say every night on a on a schedule, with enough checking that the downstream clinical uh model is allowed to depend on it output.
00:07:27.040 --> 00:07:28.480
It's a very sensitive field.
00:07:28.639 --> 00:07:31.199
So people should be able to depend on that.
00:07:31.439 --> 00:07:39.439
And the last uh the last clauses where all it works, getting a Lama 3 to summarize one of the nodes nicely is just a Tuesday afternoon.
00:07:39.600 --> 00:07:46.079
Getting it down to 2 million every night, reproducibility with a paper tail is the real system.
00:07:46.160 --> 00:07:52.959
That's exactly what the whole talk is about, and how we can do it in a good engineering way.
00:07:53.920 --> 00:07:55.759
Yeah, that makes sense with that data.
00:07:55.839 --> 00:08:04.879
That with uh anything super sensitive, you can't send it to external systems, and the traceability and kind of data quality aspects become really, really important.
00:08:05.759 --> 00:08:10.079
So then tell me about where Airflow fits into this architecture.
00:08:11.199 --> 00:08:11.680
Sure.
00:08:12.000 --> 00:08:18.319
So Airflow sits on top of it, like it never touches the data, but it orchestrates it.
00:08:18.399 --> 00:08:21.680
So it's one of the most important parts.
00:08:22.079 --> 00:08:40.480
Let's say the heavy lifting is done by GPU ports on Kubernetes that run the model, object storage holds the uh raw nodes, Postgres holds the result, but Airflow is the one which decides what's to what to run, in what order, and what happens if any of this uh service breaks.
00:08:40.799 --> 00:08:44.159
So if I have to divide, let we can divide it into four jobs.
00:08:44.320 --> 00:08:45.679
One is the sensors.
00:08:45.919 --> 00:08:52.960
So we say every site, let's say every site drops files on its own schedule, and some nights there might not be any files.
00:08:53.120 --> 00:08:58.799
So there is a sensor per site waiting on its data, and each site has its own task.
00:08:58.960 --> 00:09:03.600
So if one site is four hours later, only that branch waits.
00:09:03.679 --> 00:09:09.679
So that's what AFLO comes into picture and it decides which branch to wait, which branch to run.
00:09:09.919 --> 00:09:14.879
And when it when I talk about branches, second thought can come about dependencies.
00:09:15.120 --> 00:09:21.519
Validation directly cannot start until uh validation cannot start until inference for that batch finishes.
00:09:21.679 --> 00:09:24.960
So load cannot start until validation passes.
00:09:25.360 --> 00:09:35.840
That ordering itself is the entire safety design and it literally lives inside the DAG instead of somebody's head, and Airflow is the one that's orchestrating it.
00:09:36.399 --> 00:09:39.039
Next is the retries and backfills.
00:09:39.200 --> 00:09:42.080
What if uh the site uh delivers one day late?
00:09:42.399 --> 00:09:46.799
Or uh let's say there was a there was a failure.
00:09:46.960 --> 00:09:50.080
So we need to make sure we don't retry everything.
00:09:50.320 --> 00:09:56.639
Airflow takes care of retrying that specific node or backfilling that specific uh specific data.
00:09:56.799 --> 00:09:59.440
So you don't need to rerun everything again.
00:09:59.840 --> 00:10:02.240
And the last is the run record.
00:10:02.320 --> 00:10:10.639
When DAG runs with with configs, uh, we need to know which config is it, which model version it's producing, and that's the linear start.
00:10:10.720 --> 00:10:15.759
And you get it free because uh Airflow is tracking every run for it.
00:10:15.919 --> 00:10:26.320
So this is where uh Airflow is an integral part of any orchestration that we do, be it uh LLMs or traditional uh ML systems.
00:10:27.519 --> 00:10:28.960
Yeah, that's interesting.
00:10:29.120 --> 00:10:37.840
I think you're describing a lot of the very bread and butter airflow orchestration functionality, and I love how you've talked about how it's so important for this use case.
00:10:37.919 --> 00:10:38.720
That's great.
00:10:39.039 --> 00:10:43.919
I'm wondering if you can give me any more detail on how you're structuring those DAGs.
00:10:44.000 --> 00:10:49.440
So you're managing quite a lot with them from the LLM inference and the validation and those ingestion stages.
00:10:49.600 --> 00:10:51.600
What do those DAGs look like?
00:10:53.120 --> 00:10:57.279
So, yeah, the uh DAGs can have multiple nodes, a lot of connections.
00:10:57.440 --> 00:11:05.279
There can be multiple stages, but if I have to divide it into some core buckets, I can divide it into four stages.
00:11:05.519 --> 00:11:08.960
Stage one, we can say we can ingest and normalize it.
00:11:09.039 --> 00:11:12.720
So one task per group site, uh, we can run all in parallel.
00:11:12.879 --> 00:11:24.240
Each one lands raw nodes in a big object storage, let's say S3, and writes those manifest rows so you know what arrived, when it arrived, and how it arrived.
00:11:24.480 --> 00:11:28.639
Once we have all that data, second can be the inference.
00:11:29.039 --> 00:11:40.879
So all these nodes or the data, it can get grouped in badges and run on GPU, GPU pods, and that can be done through Kubernetes pod of uh operators.
00:11:41.039 --> 00:11:43.840
So the GPU works in isolation from the scheduler.
00:11:43.919 --> 00:11:47.840
That's very important to keep things isolated at their own level.
00:11:48.000 --> 00:11:55.440
And this stage sits in its own air uh airport, sorry, airflow pool uh with no hard concurrency cap.
00:11:55.759 --> 00:12:02.639
So here we need to make sure uh we don't have a lot of tasks and less number of GPUs.
00:12:02.960 --> 00:12:06.080
So that's uh where we need to configure it properly.
00:12:06.240 --> 00:12:08.159
Uh stage three can be validation.
00:12:08.240 --> 00:12:12.720
So uh let's say we have the data, our inference has uh already ran properly.
00:12:12.879 --> 00:12:23.279
Now we need to make sure the output that is being generated goes through a parentic schema, then cross-checks against the structure record, and we can make sure the output is correct.
00:12:23.519 --> 00:12:26.879
And the final one we can talk about that's load.
00:12:27.039 --> 00:12:31.840
So uh we load those all the output into a feature store.
00:12:32.000 --> 00:12:44.960
So a final model that's serving a readmission or fraud detection, whatever, that can read from the data feature store and give a good, well-structured output or a scoring.
00:12:45.200 --> 00:12:52.000
So these those are the four main stages where we are using Airflow and how we are using it.
00:12:53.279 --> 00:12:54.960
Okay, interesting.
00:12:55.200 --> 00:13:01.279
And then I'd also love to dig a little bit deeper into how you're handling failures.
00:13:01.360 --> 00:13:08.399
So you touched on this a little bit earlier uh with using Airflow to manage retries and things like that.
00:13:08.559 --> 00:13:15.919
But tell me in a little bit more detail if something goes wrong, or you mentioned you have a lot of data quality checks built in.
00:13:16.000 --> 00:13:20.879
Uh, when there's also this LLM in the loop that you're managing, how does all of that work?
00:13:22.080 --> 00:13:22.480
Yes.
00:13:22.639 --> 00:13:28.799
So before directly uh diving into the failures, we can divide it into two types of failure.
00:13:29.039 --> 00:13:31.840
One can be the easier one, that's the loud failure.
00:13:32.000 --> 00:13:36.320
The pod gets OOM killed, the GPU is busy, the model timeout.
00:13:36.720 --> 00:13:39.200
That is an ordinary airflow failure.
00:13:39.360 --> 00:13:44.000
We can retry with exponential fallbacks, we can do task isolation as I mentioned.
00:13:44.240 --> 00:13:46.320
Task isolation is very important.
00:13:46.480 --> 00:13:53.200
We can rerun the task and make sure that the things are not overwritten and we are replacing it with the correct output.
00:13:53.360 --> 00:13:54.720
So those are loud failures.
00:13:54.799 --> 00:13:59.759
We can go and check the system and see everything is working fine or if there is a failure.
00:14:00.399 --> 00:14:03.759
The category two, that's the scarier one.
00:14:04.080 --> 00:14:10.639
Because here, let's assume model runs, it returns a clean output, and nothing is wrong.
00:14:10.960 --> 00:14:15.120
For a normal person, everything is green, the graph is green, everything is looking good.
00:14:15.360 --> 00:14:17.279
That's a good picture.
00:14:18.000 --> 00:14:22.480
But here, uh there are a lot of scenarios that can happen.
00:14:22.720 --> 00:14:30.799
Uh, let's say uh the output is not correctly in not in a correct process or not in a correct format.
00:14:31.039 --> 00:14:33.200
So here we have constraint output.
00:14:33.279 --> 00:14:36.000
Uh the model fills in the uh piedenic schema.
00:14:36.080 --> 00:14:38.879
We cannot have a free text, so it does not pass.
00:14:39.120 --> 00:14:47.519
The record the record is injected, but if it's not in a piedentic schema, we do not let it uh go to a next stage.
00:14:47.600 --> 00:14:50.320
So that is how we check this silent failure.
00:14:50.399 --> 00:14:55.120
We have piedantic for this, then we can check against the structure record.
00:14:55.360 --> 00:15:08.559
So let's say if Note is extracting something about a heart failure and the core diagnosis doesn't have anything about it, so we can write some logic to check this kind of issues.
00:15:08.879 --> 00:15:10.879
Then we can do a distribution check.
00:15:11.039 --> 00:15:24.639
Let's say uh for any fraud detection or for any kind of let's say for fraud detection or any kind of anomaly, uh every day it's just 2%, but suddenly we saw it shot up to 8%.
00:15:25.279 --> 00:15:30.320
Everything is on a rosy picture, everything is green, but here you see there is some anomaly.
00:15:30.480 --> 00:15:34.000
So we need to go in manually and check what's going wrong.
00:15:34.159 --> 00:15:36.399
So that is something we need to debug.
00:15:36.720 --> 00:15:41.039
And initially, when you are setting a system, it's in a dev stage and not prod.
00:15:41.600 --> 00:15:48.720
We can have a human in loop who uh checks every X sample and makes sure what we are outputting is correct.
00:15:48.960 --> 00:15:58.159
So the silent failures are more challenging, and these are some of the ways we can handle them, but still we might miss at some stage.
00:15:58.240 --> 00:16:01.279
But currently, this is the most challenging part in the industry.
00:16:02.399 --> 00:16:03.600
Yeah, that makes sense.
00:16:03.759 --> 00:16:10.799
And how do you go about trying to solve those the silent failures, as you've said, with Airflow?
00:16:10.879 --> 00:16:13.120
Like where do you think Airflow fits in there?
00:16:13.279 --> 00:16:17.679
Because I think everybody probably knows how you can use Airflow to manage the loud ones.
00:16:17.919 --> 00:16:19.440
As you say, that's really easy.
00:16:19.600 --> 00:16:24.879
Obviously, harder to build in some logic to manage the softer failures.
00:16:25.679 --> 00:16:30.960
So we we can at each step we know what are the outputs, what are the inputs.
00:16:31.120 --> 00:16:50.080
So we can have some Chrome jobs or Python script as one of the nodes, which uh which after a specific uh uh output, it checks uh after a specific step, it checks if uh the output is correct and it will go through the as I say, it will go through the pyentic validation.
00:16:50.159 --> 00:16:52.320
So that can be one of the nodes in the airflow.
00:16:52.559 --> 00:17:00.960
So putting those check notes in the whole orchestration, that can be how that that's one of the ways to check these silent flows.
00:17:01.360 --> 00:17:01.840
Got it.
00:17:01.919 --> 00:17:06.160
So you're building in data quality checks of a sort into the pipelines.
00:17:06.400 --> 00:17:07.680
Yeah, that makes sense.
00:17:09.119 --> 00:17:17.359
This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:17:17.759 --> 00:17:21.680
From batch pipelines to training models and wiring up agents.
00:17:21.920 --> 00:17:27.920
Astro handles the infrastructure upgrades and scaling so you can focus on building.
00:17:28.240 --> 00:17:30.480
Check it out at astronomer.io.
00:17:32.960 --> 00:17:34.000
Okay, great.
00:17:34.240 --> 00:17:45.680
And then I guess on a slightly different but related topic, I understand that part of your approach uh to managing these pipelines includes fine-tuning.
00:17:45.839 --> 00:17:53.599
And so I'm curious if you could tell me more about what that looks like and where that fits into your Airflow pipelines.
00:17:54.720 --> 00:17:55.119
Sure.
00:17:55.440 --> 00:18:06.480
So uh as you mentioned, it's a little bit uh away from the core DAC topics, but uh uh let's think it's not something that we need to run every night.
00:18:06.799 --> 00:18:11.359
Fine-tuning is not something we we need every day or every night or every week.
00:18:11.680 --> 00:18:14.160
It is it has his own Airflow DAC.
00:18:14.400 --> 00:18:28.960
It doesn't sit in the core flow and it fires maybe once a month or maybe once in a quarter when we have enough new data piled up so we can fine-tune the already fine-tuned model we have.
00:18:29.119 --> 00:18:38.960
So it pulls the label set, runs a lot of fine-tuning with Hugging Face, PEFT on top of Lama 3, then scores it against the frozen holdout.
00:18:39.200 --> 00:18:45.839
And if it only if it beats what we have, it writes a new adapter version to the model registry.
00:18:46.079 --> 00:18:54.160
So that is something, that is something has to be a totally different Airflow DAC apart from the gore one which we are working.
00:18:55.359 --> 00:19:10.720
And so Nightly Drag uh, and one more thing is the Nightly DAC trains nothing, it just loads the base Lama 3 weight, loads the current adapter, file uh loads the current adapter file next to them and runs runs it basically uh to the old files.
00:19:10.960 --> 00:19:14.960
And the adapter is maybe let's say a few hundred megabytes on the disk.
00:19:15.200 --> 00:19:22.079
So all of the fine tuning cost you uh you run is just about reading one extra file.
00:19:22.240 --> 00:19:23.119
That's it.
00:19:23.440 --> 00:19:34.559
And one more thing where Airflow can fit perfectly here is when we write to these adapters and we keep updating the record, Airflow helps us to maintain different versions with it.
00:19:34.720 --> 00:19:42.960
So six months later down the line, you can look back, row, uh look back at a row in a feature, and we can know exactly which adapter produced it.
00:19:43.119 --> 00:19:51.519
And if we want to go to the specific version, we can use Airflow, run it across along with that adapter and get the same result.
00:19:51.759 --> 00:20:00.240
So, in this sensitive type of field, if there is an audit or something, we'll have a very good trace, and that is because of Airflow, how we are managing it.
00:20:00.640 --> 00:20:02.400
Okay, that's really cool.
00:20:02.559 --> 00:20:09.039
Uh, I have heard of a few other use cases of people using Airflow to manage fine-tuning like this, but not a ton.
00:20:09.200 --> 00:20:12.079
So this is great to hear your perspective on.
00:20:12.400 --> 00:20:20.559
And then tell me more about so when you're fine-tuning the models, are you looking for better performance?
00:20:20.720 --> 00:20:29.680
Are you looking for better, or I should say, performance in terms of accuracy or performance in terms of speed or efficiency, or is it all of the above?
00:20:30.880 --> 00:20:34.160
It's mostly on term uh terms of accuracy.
00:20:34.480 --> 00:20:53.759
It's uh because we have Lama 3, it's a very good model on uh normal English, but when it comes to a specific domain, let's say financial domain or a medical domain, it lacks a little bit because there is a context to because it's a context uh that it's lacking.
00:20:54.000 --> 00:21:00.400
So when we talk about fine-tuning, we take we freeze the core weights and just train on the last layer.
00:21:00.559 --> 00:21:02.079
So, let me give you an example.
00:21:02.319 --> 00:21:09.680
Let's say uh doctors talk in a language, which say PT denies CP, and that means patients say they have no chest pain.
00:21:09.839 --> 00:21:14.160
So there is a very small word that says does not have chest pain.
00:21:14.319 --> 00:21:17.680
And there is some acronyms, PT, CP, all those stuff.
00:21:17.839 --> 00:21:22.640
So those get lost when it's being trained on normal English.
00:21:22.880 --> 00:21:30.480
And when we fine-tune on uh medical data, we can pick all of those and we can have a much more accurate output.
00:21:31.440 --> 00:21:32.000
Got it.
00:21:32.160 --> 00:21:33.359
That makes sense.
00:21:33.680 --> 00:21:46.319
And then I guess zooming back a little bit and looking at this high level, what were the hardest lessons you learned when trying to productionize this and getting it to run at scale?
00:21:46.880 --> 00:21:49.119
So, first is reproducibility.
00:21:49.279 --> 00:21:58.079
So, what happens when we are uh when we are working at a at a lightning speed with all the AI assisted coding and everything?
00:21:58.240 --> 00:21:59.759
We forget how to train.
00:22:00.319 --> 00:22:08.480
Back and how to uh if something fails, or uh how to trace back how we have iterated and how did we come in so far.
00:22:08.960 --> 00:22:13.440
So let's say uh we uh we need to pin those stuff.
00:22:13.599 --> 00:22:19.200
We need to pin, uh, we need to pin the temperature because there can be a lot of variance.
00:22:19.440 --> 00:22:26.000
The feature starts to drift if we don't pin the model version, we don't pin the features, we don't pin the temperature.
00:22:26.160 --> 00:22:28.079
So that is one important thing.
00:22:28.319 --> 00:22:38.559
Make sure every time you are running a model or any orchestration, pin everything versions, uh versions, seed, features, everything.
00:22:38.880 --> 00:22:41.359
Next is about the GPUs.
00:22:41.680 --> 00:22:48.799
So uh a lot of time uh what we did, we scheduled a lot more tasks in parallel than we had GPUs.
00:22:49.200 --> 00:22:56.000
So even if the tasks are in parallel, we don't have enough resources for them to run in parallel.
00:22:56.160 --> 00:23:03.039
So make sure that the amount of infra you have, you parallelize your orchestration based on that.
00:23:03.200 --> 00:23:05.519
You don't over-parallelize or over-complicate it.
00:23:05.680 --> 00:23:07.039
That's the second thing.
00:23:07.359 --> 00:23:12.480
And the third one that matters the most is the traceability.
00:23:12.799 --> 00:23:18.160
How we can look six months back and say how we reach to that particular output.
00:23:18.400 --> 00:23:29.359
So uh here we need to make sure uh that uh we have a proper feature storage, we have a proper isolation, and every service has a very good trace logs.
00:23:29.599 --> 00:23:45.279
And if anything fails, we can isolate it and we can we can isolate it and make sure that uh uh we can isolate it and make sure that uh some specific thing failed here, and that's the reason these results are broken.
00:23:45.440 --> 00:23:48.559
So these are the three things that I think is most important.
00:23:48.960 --> 00:23:57.759
Yeah, I think those are great things for everybody, maybe even using LLMs or not to think about when trying to productionize stuff with Airflow.
00:23:57.920 --> 00:24:00.079
So, yeah, that's great insight there.
00:24:00.559 --> 00:24:03.920
Okay, well, Chang, thank you for going through all of that.
00:24:04.079 --> 00:24:10.000
I again think this is a super interesting use case that a lot of our listeners will benefit from hearing about.
00:24:10.160 --> 00:24:20.559
Uh, in the interest of time, I will wrap us up and I always like to end with a standard final question, and that is what would you most like to see from development of the Airflow project in the near future?
00:24:22.720 --> 00:24:28.160
Yes, so if I have to look into it right now, uh Airflow has been really useful.
00:24:28.240 --> 00:24:40.880
And with Airflow 3 coming and it's already out, it has been really helpful for developer, but few things if I had to uh pinpoint, then we can have the GPU awareness uh in Airflow itself.
00:24:41.039 --> 00:24:51.440
So right now, if our task needs a GPU, you can express it sideways through pools, cues, and naming convention, and every team reinvents it slightly differently.
00:24:51.680 --> 00:24:59.440
So I would like a task to declare that if it needs a GPU for 20 minutes and have the shed and let the scheduler honor it.
00:24:59.680 --> 00:25:03.599
It is the piece that I have hand-rolled on every project I have done.
00:25:03.759 --> 00:25:12.480
So instead of each developer or each team hand holding uh hand-rolling it, we can have a first-class awareness GPU inbuilt in Airflow system.
00:25:13.039 --> 00:25:21.519
And second is treat model as assets basically, because a lot of time, a lot of teams now are modeling and they are using Airflow for that.
00:25:21.759 --> 00:25:28.000
So I know Airflow already understands that a data set is produced by one task and consumed by another.
00:25:28.160 --> 00:25:33.119
Uh and a base model and a LoRa adapter are exactly that shape.
00:25:33.279 --> 00:25:40.960
So if assets covered a model version, uh, then which model version produced this table becomes a question Airflow can answer.
00:25:41.119 --> 00:25:44.400
And a new developer doesn't have to come with their own versioning.
00:25:44.559 --> 00:25:50.240
So I think these two are the main thing treating model as assets and having more first-class GPU awareness.
00:25:50.720 --> 00:25:51.839
Yeah, got it.
00:25:52.000 --> 00:25:54.160
I think those are both great requests.
00:25:54.480 --> 00:25:57.599
Okay, well, again, thank you so much for going through all this.
00:25:57.680 --> 00:25:58.880
I think this has been great.
00:25:59.039 --> 00:26:02.559
To close us out, what is the best way for folks to get in touch with you?
00:26:02.960 --> 00:26:06.160
Uh I think LinkedIn is the best way that they can get in touch with me.
00:26:06.400 --> 00:26:10.799
And uh I can also have my email ID in it, so LinkedIn and email ID.
00:26:10.960 --> 00:26:11.119
Okay.
00:26:11.920 --> 00:26:15.839
And later I'll be I'll be having a talk at Airflow Summit.
00:26:15.920 --> 00:26:17.839
So more than open to seeing people there.
00:26:18.160 --> 00:26:18.799
Yeah, great.
00:26:18.960 --> 00:26:25.920
Well, I think this episode will probably be released after Airflow Summit since we're in very short time for Airflow Summit this year.
00:26:26.000 --> 00:26:28.720
Uh, but hopefully some folks will have seen you there.
00:26:29.039 --> 00:26:31.839
And yeah, we'll throw links to those other things in the show notes.
00:26:32.000 --> 00:26:34.319
So uh Chang, thank you again so much.
00:26:34.480 --> 00:26:35.680
This has been great.
00:26:36.079 --> 00:26:37.759
Yeah, nice, nice meeting you.
00:26:38.160 --> 00:26:40.880
Thank you for listening to the Data Flowcast.
00:26:41.039 --> 00:26:44.559
Check the show notes for links to everything mentioned today.
00:26:44.720 --> 00:26:49.279
If you're enjoying the podcast, subscribe so you never miss an episode.
00:26:49.519 --> 00:26:51.759
And leave us a five-star review.
