WEBVTT
00:00:33.520 --> 00:00:35.520
Or click the link in our show notes.
00:00:35.759 --> 00:00:37.679
Now let's get into the show.
00:00:38.560 --> 00:00:39.679
Okay, hi everyone.
00:00:39.840 --> 00:00:41.520
Welcome to the Data Flowcast.
00:00:41.600 --> 00:00:50.960
I'm your host, Kenton Danis, and today I'm joined by Anish Geary, who is a senior software engineer and active contributor to Apache Airflow focused on security.
00:00:51.280 --> 00:00:52.240
Anish, how are you?
00:00:52.640 --> 00:00:53.439
Yeah, I'm doing fine.
00:00:53.520 --> 00:00:54.000
How are you?
00:00:54.079 --> 00:00:55.280
Thanks for having me in today.
00:00:55.679 --> 00:00:56.079
I'm good.
00:00:56.159 --> 00:00:57.439
It's really great to have you on.
00:00:57.520 --> 00:00:59.200
I'm excited to dive into this.
00:00:59.280 --> 00:01:03.600
I always love having people who are contributing to the project on the podcast.
00:01:03.920 --> 00:01:13.920
And for today, for our listeners, we're going to discuss your security contributions to Airflow, including things like connection testing, token revocation, and token scoping.
00:01:14.319 --> 00:01:19.920
I think these are going to end up being things that probably every Airflow user encounters to some degree.
00:01:20.079 --> 00:01:23.040
So it should be very helpful for all of our listeners.
00:01:23.280 --> 00:01:26.239
But before we get into all of that, I want to start with some background.
00:01:26.400 --> 00:01:27.439
So let's talk about you.
00:01:27.599 --> 00:01:29.200
How long have you been using Airflow?
00:01:29.840 --> 00:01:38.879
Yeah, before starting, I want to put a little disclaimer out that all the any views that I put out here is just mine and it does not relate to any organization.
00:01:39.599 --> 00:01:47.359
And yeah, uh, well, yeah, it's funny thing that I'm not a data engineer and I've not uh run any pipelines in production yet.
00:01:47.519 --> 00:01:51.760
I got into Airflow by reading the source code and docs itself.
00:01:52.079 --> 00:01:55.359
And yeah, I see many people coming from outside in.
00:01:55.439 --> 00:01:58.239
My journey was inside out, I believe.
00:01:59.120 --> 00:02:00.319
Yeah, that's cool.
00:02:00.640 --> 00:02:06.159
I think you're right that probably most people are Airflow users themselves and then start contributing.
00:02:06.239 --> 00:02:23.599
Although we do certainly have people at Astronomer who are on the Airflow PMC or who are committers who spend most of their day job working on Airflow, who that is their background as well as they have worked in software and worked in open source software, but they are not using Airflow themselves.
00:02:23.759 --> 00:02:25.919
So you're definitely not alone there.
00:02:26.800 --> 00:02:33.280
And I understand that you, you know, started contributing to Airflow fairly recently, um, like at the end of last year.
00:02:33.439 --> 00:02:38.000
I guess just kind of high-level what got you into open source contribution?
00:02:38.080 --> 00:02:40.800
Like you mentioned, you are not an Airflow user yourself.
00:02:40.879 --> 00:02:43.840
What made you interested in working with the project?
00:02:44.240 --> 00:02:53.199
Yeah, well, well, my whole career is, you know, um dependent on users of uh the open source uh all the time.
00:02:53.360 --> 00:03:00.479
And uh I think I know many people out there are spending their time, their precious time, building this for free.
00:03:00.560 --> 00:03:02.479
And that always fascinated me.
00:03:02.719 --> 00:03:06.400
And at one point of time I wanted to be in the giving side of the spectrum.
00:03:06.479 --> 00:03:10.400
So I started small and got got hooked into it.
00:03:10.960 --> 00:03:12.080
Yeah, that's amazing.
00:03:12.240 --> 00:03:16.800
One of the best things I think about the Airflow project is just the number of people contributing.
00:03:16.960 --> 00:03:23.039
I saw, I think they just passed 4,000 contributors uh to the project, which is so cool.
00:03:23.199 --> 00:03:28.080
And so obviously, a very small fraction of those people are doing it as part of their day job.
00:03:28.159 --> 00:03:34.719
And as you said, a lot of them are doing it just because they want to, they want to make the project better, whether they're using it themselves or not.
00:03:34.879 --> 00:03:36.800
So that's really cool.
00:03:37.120 --> 00:03:42.960
And then we're gonna talk a lot about your uh contributions focused on security.
00:03:43.039 --> 00:03:47.280
And I'm curious what drew you to that sort of work specifically.
00:03:47.759 --> 00:03:50.879
Well, yeah, well, while uh I think it likewise.
00:03:50.960 --> 00:03:55.759
So while we think about software development, we think about uh what can this do.
00:03:55.919 --> 00:04:03.199
But I think when it comes to security, uh the question slips and uh it becomes what can in uh any anyone make this do?
00:04:03.280 --> 00:04:12.800
So this later part of the question always, you know, uh excited me about software development itself, and it drew always drew to me the security side of the engineering.
00:04:13.919 --> 00:04:15.599
Yeah, very interesting.
00:04:15.840 --> 00:04:20.720
Well, let's go ahead and dig into some of the work that you've done on the project recently.
00:04:20.879 --> 00:04:25.360
And I'd love to talk uh first about connection testing.
00:04:25.519 --> 00:04:31.680
So I know this is a feature that was disabled by default in Airflow 2.7 after a CVE was reported on it.
00:04:31.759 --> 00:04:41.680
And I guess for some background, I think probably most Airflow users know this, but this is the feature within the Airflow connections UI where you can input all of the information and then just say test connection.
00:04:41.839 --> 00:04:54.879
So it in theory makes the connection to the external system, but you don't have to run a DAG that uses that connection in a task to see, oh, something about the way that I formatted my credentials or whatever doesn't work.
00:04:55.040 --> 00:04:56.639
So pretty useful feature.
00:04:56.800 --> 00:05:03.360
I have always said myself and my own experience as a data engineer working with Airflow, I think connections are one of the hardest parts.
00:05:03.600 --> 00:05:10.160
It can just be really tricky to get all of the information in them right based on all of these different external systems.
00:05:10.639 --> 00:05:15.439
But again, this feature uh still exists but is disabled by default.
00:05:15.600 --> 00:05:24.800
Um so tell me a little bit, yeah, about your work on this and how what change you contributed and kind of what that does to the way connection testing works.
00:05:25.199 --> 00:05:26.240
Yeah, yeah, for sure.
00:05:26.399 --> 00:05:32.240
So uh while Airflow executes this connection testing, it uh loads the whole driver.
00:05:32.319 --> 00:05:43.040
And I think in some cases it loads it can load the entire class and also you know can do an outbound request with the user-supplied values that uh that comes from that form.
00:05:43.279 --> 00:05:49.040
And that form basically acts as uh you know uh run code in the server uh form.
00:05:49.759 --> 00:05:56.639
And uh and this is specifically why it was disabled in back in uh there in 2.7, I believe.
00:05:56.800 --> 00:06:08.879
And you know, it this connection runs in that time it ran in the web server, which was the most crucial process of the Airflow itself that had all the DB connection and the uh secrets, and that is why it was twist off.
00:06:09.199 --> 00:06:16.720
Now uh with my um contribution, uh what it made is it it just uh made it asynchronous currently.
00:06:16.800 --> 00:06:31.279
What it does is in current mechanism, not the new design, what it does is it makes a connection request uh to the API server, it executes that right there, and UI uh waits for the API server to come back.
00:06:31.360 --> 00:06:44.480
Uh, but with the new design, uh the API server does nothing, uh just records the connection testing uh that it gets and uh hands back UI uh token and the ID so that it can pull uh for the results later on.
00:06:44.639 --> 00:06:49.040
And now the execution part of it happens in uh the worker.
00:06:49.279 --> 00:06:58.319
Uh and here uh you know uh it executes uh with the same mechanism that uh the current tasks execution uh machinery uses.
00:06:58.480 --> 00:07:01.920
So yeah, this is what the actual chains look like.
00:07:02.879 --> 00:07:03.920
Yeah, that makes sense.
00:07:04.000 --> 00:07:16.399
So changing kind of the architecture of how this feature works sounds like also based on the new Airflow 3 architecture now that you do have this separate API server and that workers no longer directly access the metadata database.
00:07:16.480 --> 00:07:17.439
So am I getting that right?
00:07:17.519 --> 00:07:21.519
That because this runs in a worker, now you're not accessing those secrets directly.
00:07:21.759 --> 00:07:25.680
Yeah, yeah, with this new design, it actually became a job, you know.
00:07:25.759 --> 00:07:28.959
And with job, none of the credential passes along with it.
00:07:29.040 --> 00:07:39.600
Uh, and when it reaches workers, it uh does uh API requests to uh the API server spill itself to get the details uh of the connection testing to run it.
00:07:39.680 --> 00:07:44.000
And API server, the pro the worker doesn't have the privilege that API server has.
00:07:44.319 --> 00:07:44.560
Okay.
00:07:44.720 --> 00:07:45.519
Yeah, that's great.
00:07:45.600 --> 00:07:54.639
I would imagine that makes a big de difference for a lot of organizations on whether or not they can use this feature at all or whether their security teams are okay with somebody enabling it.
00:07:55.040 --> 00:07:56.399
Yeah, very cool.
00:07:56.720 --> 00:08:01.279
And I also am curious about the process for making this change.
00:08:01.439 --> 00:08:22.399
So I think uh changes like this that have kind of an architectural component, especially with something that has previously had um, you know, a CVE tied to it or has ongoing potential security issues, something like connections, you know, those often go through a discussion on the dev list uh for airflow and you know an architecture review.
00:08:22.560 --> 00:08:24.079
So is that something you did for this?
00:08:24.160 --> 00:08:25.839
And what did that process look like?
00:08:26.319 --> 00:08:28.560
Yeah, this story is actually interesting.
00:08:28.720 --> 00:08:32.159
So this all started with the PR from a different commuter.
00:08:32.240 --> 00:08:37.600
Uh and uh, you know, it was a bug fix in the current uh connection testing feature.
00:08:37.840 --> 00:08:49.120
There, I saw a comment from Jarek, uh, which whose comment was, you know, uh this the ideal fix is to move this into the worker self and not patch it here.
00:08:49.279 --> 00:08:50.960
So he wrote the all history there.
00:08:51.120 --> 00:08:54.480
I read that comment and I was like, I had to build this uh stuff.
00:08:54.720 --> 00:09:18.320
And yeah, I I uh built up my uh uh design and uh I wired up it and I um took it to the mailing list, and then all the PMC uh members and the commuters came came in, and you know, my design got torn apart and it was even rejected at first, and then it uh got revived along the way on uh you know different iteration.
00:09:18.480 --> 00:09:26.639
I mean, it takes time, but you know, the design that we have uh at last is is, I believe, good, and the thing that is checked in, I'm I'm really proud of.
00:09:27.440 --> 00:09:28.480
Yeah, that's great.
00:09:28.720 --> 00:09:38.639
Definitely, I think a strength of open source projects is the kind of consensus-driven approach, but as you describe, it can also make it quite challenging.
00:09:38.799 --> 00:09:51.519
I have heard stories from lots of people who have come with an idea uh that was then sort of torn apart by a lot of the Airflow PMC or committers and then eventually land on something that is much stronger.
00:09:51.679 --> 00:10:00.399
And so that's that's really cool that you were able to um come to that consensus and um deliver something that, like I said, is really meaningful to a lot of users.
00:10:00.799 --> 00:10:01.679
Yeah, for sure.
00:10:02.000 --> 00:10:03.279
Yeah, that's great.
00:10:03.600 --> 00:10:10.080
Well, I also want to talk about some of the other security-related contributions that you have worked on recently.
00:10:10.240 --> 00:10:14.879
And one of them I understand was this token revocation in Airflow 3.
00:10:15.200 --> 00:10:26.639
I think compared to what we just talked about, this is something that happens much more on the kind of behind the scenes on the back end, so that users probably aren't even aware of if everything is going right.
00:10:26.799 --> 00:10:35.200
Uh, so explain to me a little bit kind of what is token revocation, how does that work, and what was the problem with how it was handled before?
00:10:35.679 --> 00:10:42.159
Yeah, so this basically uh is uh drills down to the nature of the JWT itself.
00:10:42.240 --> 00:10:48.799
You know, it's stateless and server can validate uh JWT without maintaining any state at all.
00:10:48.960 --> 00:10:54.240
And it can uh always validate it if uh the signature is correct, the token is fine.
00:10:54.399 --> 00:11:09.919
So in Airflow, when user did the logout, um uh we just uh in Airflow 3 specifically, um, you know, we just removed that uh token from the browser itself, uh, but the token was still valid because it was it not it was not expired.
00:11:10.080 --> 00:11:17.679
Uh well it has certain default time uh where it would expire, but it was it's not it was not expired for that particular time.
00:11:17.919 --> 00:11:28.639
And you know, anyone uh that could get hold of that token uh you know could act as you and do operation in behalf of you, which which which was not correct uh and ideal.
00:11:28.799 --> 00:11:33.679
Uh so we uh built a revocation system to eliminate this.
00:11:34.159 --> 00:11:51.200
And this uh this revocation system uh in this revocation system we built a tiny table which is just basically a dead list, and uh where we uh you know uh do the capture all of uh the tokens that are expired.
00:11:51.360 --> 00:11:58.320
And uh whenever user clicks log out now, uh we add a record there with its JTI and the expiry token.
00:11:58.559 --> 00:12:06.000
And whenever subsequent request comes with the token, we check that uh the token against that table, and if it exists here, we just reject it.
00:12:06.080 --> 00:12:09.840
So now with this mechanism in place, the logout is actual logout.
00:12:10.399 --> 00:12:18.159
And I want to be a bit clear, this uh revocation is just for the UI tokens, not on the execution side of the tokens that we have.
00:12:18.559 --> 00:12:27.039
And uh yeah, this is how uh we uh we are able to solve uh the actual revocation problem in Airflow 3.
00:12:27.440 --> 00:12:29.200
Okay, got it, makes sense.
00:12:29.440 --> 00:12:34.799
Uh yeah, so I guess, like I said before, uh a change that is completely on the back end.
00:12:34.879 --> 00:12:42.399
So hopefully most users are not aware of this at all when they're working with Airflow, but definitely a big security implication.
00:12:43.120 --> 00:12:44.080
Yeah, very cool.
00:12:45.759 --> 00:12:54.080
This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:12:54.480 --> 00:12:58.320
From beta pipelines to training models and wiring up agents.
00:13:04.879 --> 00:13:07.120
Check it out at astronomer.io.
00:13:08.720 --> 00:13:17.039
And then I understand you have also worked recently on uh uh a project you split Airflow's tokens into two types.
00:13:17.200 --> 00:13:21.039
Was this related to token revocation or is this something totally different?
00:13:21.600 --> 00:13:23.840
Well, yeah, this was totally, totally different.
00:13:24.559 --> 00:13:25.039
Yeah.
00:13:25.279 --> 00:13:30.399
Um it it came out uh it was a very strange issue, actually.
00:13:30.720 --> 00:13:38.639
And uh uh one use it was uh in this case, uh the tax was in the queue for a very long time.
00:13:39.039 --> 00:13:49.679
Uh and uh whenever worker picked the tax again up, uh the tax was uh getting 403 on the first call to the uh execution API.
00:13:49.919 --> 00:14:04.960
And the strange thing was, you know, the Airflow 3 already had uh a refresh uh token mechanism built in, and it uh you know refreshes the token on every subsequent request that comes from uh the uh you know workers.
00:14:05.200 --> 00:14:13.600
Uh but in this case the token was living there and none of the API calls was made to the server, so token refresh never happened.
00:14:14.080 --> 00:14:17.679
And uh as it never happened, it eventually died.
00:14:17.919 --> 00:14:31.600
And for this very case, the user was you know running a GPU pipeline and his uh his tasks was failing uh after 15 minutes, and he was he was uh kind of confused why that was happening because refresh also happens.
00:14:31.679 --> 00:14:51.600
Uh but and also he tried multiple things to it, uh like increasing the uh task queue timeout, which is basically a timeout uh in uh uh until which uh tasks can stay in the queue itself, uh, but that didn't work because it was not tied to the DWT uh expiration at all.
00:14:52.000 --> 00:14:57.919
So actually, with to solve these kind of you know uh invariation, we have to split that token into two.
00:14:59.039 --> 00:14:59.360
Okay.
00:14:59.919 --> 00:15:00.399
Got it.
00:15:00.480 --> 00:15:01.360
That makes sense.
00:15:01.679 --> 00:15:06.639
And then so tell me how does how does splitting those?
00:15:06.799 --> 00:15:11.120
You gave the one example, um maybe more generally.
00:15:11.200 --> 00:15:14.879
So how how does splitting those impact like the end user?
00:15:15.039 --> 00:15:16.879
What would they see working differently?
00:15:17.600 --> 00:15:28.480
So uh well I would say previous uh previous token uh you know uh was one token and having two entirely different jobs.
00:15:28.639 --> 00:15:33.360
That is one was staying in the queue alive and one was authentication itself.
00:15:33.919 --> 00:15:39.039
And uh for uh staying in the queue alive, it needs longer time.
00:15:39.120 --> 00:15:44.080
Uh and uh for you know for authentication it needs a shorter time, actually.
00:15:44.399 --> 00:16:00.720
Uh so basically, and uh with the longer time token as it uh as a workload travels through you know this executor transport, it is uh uh in a you know on a bit of an untrusted environment like uh broker logs and uh the KDS spec itself.
00:16:00.799 --> 00:16:06.080
So it has to be tightly scoped so that the leak token is you know uh kind of unusable.
00:16:06.159 --> 00:16:16.000
And in this case as well, uh the workload token is kind of unusable uh totally because it can only do one thing that is run uh the tax and that's it.
00:16:16.080 --> 00:16:24.480
Uh then when it reaches worker, the execution uh uh you know API already have swapped that token uh while the first request happened.
00:16:24.720 --> 00:16:37.759
So yeah, and the actual uh solving part happened here when we already increased the length of the workload token to be identical to the tax queue timeout timeout because it is now the same.
00:16:37.840 --> 00:16:44.240
The same configuration is picked up for both workload token lifetime and the queue type uh queue tax timeout.
00:16:44.720 --> 00:16:48.879
Uh, so that you know the two clock doesn't fight with each other, it is the same now.
00:16:48.960 --> 00:16:56.799
And uh now, however time uh the uh you know tax is in that queue, it won't uh face the issue that user was facing.
00:16:56.879 --> 00:17:03.279
And also, we already have uh uh uh scope so narrow that security is already baked into it.
00:17:04.079 --> 00:17:05.119
Okay, got it.
00:17:05.279 --> 00:17:06.240
Yeah, that's great.
00:17:06.400 --> 00:17:08.880
What what an impactful change.
00:17:09.119 --> 00:17:15.599
And all all three of these things that we've just talked about, you've done you know, this year basically, or since kind of the end of last year.
00:17:16.480 --> 00:17:17.920
Yeah, that's awesome.
00:17:18.160 --> 00:17:35.519
Um I'm curious, do you have any uh security-related projects that you're currently working on uh with Airflow, or are these uh three uh updates that we just mentioned, are they completely done?
00:17:35.759 --> 00:17:41.920
Are they concluded projects, or is there are there still optimizations that you're hoping to make or future work planned?
00:17:42.480 --> 00:17:48.240
Yeah, so basically two token mechanism is already checked in and it's already working in production.
00:17:48.319 --> 00:17:51.839
And also uh the revocation system is already in production.
00:17:52.000 --> 00:18:03.759
But with the test connection, um the groundwork and the base architecture is checked in, but we uh and only local executor is now uh you know migrating to the new uh format, new job that we have.
00:18:03.920 --> 00:18:14.079
We still have to migrate all of the executors and uh then uh just swap up the um last piece of UI uh and yeah, then then we are done on that part as well.
00:18:14.400 --> 00:18:15.359
Yeah, that's great.
00:18:15.599 --> 00:18:23.680
And for the two that are completed, I should have asked this earlier, but for people to take advantage of those, what version of Airflow do they need to be on?
00:18:24.000 --> 00:18:28.400
Uh they have to be on 3.2 uh and a higher.
00:18:28.960 --> 00:18:29.759
Okay, great.
00:18:30.000 --> 00:18:30.799
Makes sense.
00:18:31.039 --> 00:18:32.160
Yeah, that's awesome.
00:18:32.319 --> 00:18:44.480
Well, and then I always love to talk to people who have started contributing to the project relatively recently about any advice that you would give to someone who is thinking about contributing to Airflow.
00:18:44.559 --> 00:18:55.759
Uh, we talk to folks in the community all the time who want to contribute but feel a little intimidated by both the process, I think can seem quite complex when you're new to it.
00:18:55.839 --> 00:19:02.240
And also I think people have some just sort of factor of, you know, do I want to go through the review process?
00:19:02.319 --> 00:19:04.400
Are people going to accept my changes?
00:19:04.559 --> 00:19:06.640
You know, uh all of that stuff.
00:19:06.799 --> 00:19:12.319
So yeah, what what would you say to anybody who's potentially on the fence about contributing?
00:19:12.720 --> 00:19:13.440
Yeah, for sure.
00:19:13.599 --> 00:19:17.039
I mean, yeah, uh, I would say just start small.
00:19:17.200 --> 00:19:23.119
Uh, there are lots of good first uh issue labeled issues in uh, you know, GitHub.
00:19:23.200 --> 00:19:32.880
And I would say don't be afraid to pull out a PR because you know, in the review process, there is where you get mentored by the best engineers that we have in the community of Airflow.
00:19:33.359 --> 00:19:38.400
And yeah, I would say just don't be afraid, pull out a PR and go get uh started.
00:19:38.559 --> 00:19:39.839
That's it, I think.
00:19:40.160 --> 00:19:41.119
Yeah, that's great.
00:19:41.279 --> 00:19:42.319
I would echo that.
00:19:42.400 --> 00:19:45.680
Uh certainly the advice I always give people is start small.
00:19:45.759 --> 00:19:48.160
Uh that's what I have done for my own contribution.
00:19:48.319 --> 00:19:49.279
Same with my team.
00:19:49.359 --> 00:19:55.519
And you know, everybody on the Airflow PMC or who are reviewing PRs, they actually prefer smaller changes.
00:19:55.680 --> 00:19:57.359
It's a lot easier to review.
00:19:57.519 --> 00:20:06.000
A lot of the stuff that you've been describing is pretty big in terms of functional changes to Airflow, but a contribution certainly does not need to be that large.
00:20:06.400 --> 00:20:23.039
And yeah, I think I've also heard from you know the folks at Astronomer who are PMC members now and some of the most heavy contributors to the project over its history that they also started by just submitting a PR and they got tons of feedback on it and had to go back and rework.
00:20:23.119 --> 00:20:25.759
And, you know, they think it makes them stronger engineers.
00:20:25.920 --> 00:20:29.759
So I would say to people also don't be afraid of the feedback.
00:20:29.920 --> 00:20:33.839
Like nobody expects you to submit something that's perfect the first time.
00:20:34.160 --> 00:20:35.440
Yeah, I totally agree.
00:20:35.839 --> 00:20:36.960
Yeah, that's great.
00:20:37.200 --> 00:20:46.559
Uh well, Anish, I related to contributing, I guess this can be either related to something that you are working on or maybe not, and it's completely different.
00:20:46.720 --> 00:20:54.880
But I always end uh by asking a standard final question, and that is what would you most like to see from development of the Airplug project in the near future?
00:20:55.359 --> 00:21:10.160
Yeah, you know, uh we just checked in this uh uh principle of two token mechanism where we scope things uh of different parts of the airflow really tight so that if any uh authorization or credential linked in, it would be totally useless to attacker.
00:21:10.400 --> 00:21:16.640
So I think I would like to see more of these kinds of scoping uh spread upon all of the creational part of the airflow.
00:21:16.720 --> 00:21:18.720
And yeah, I'll be looking forward for it.
00:21:19.119 --> 00:21:19.920
Okay, great.
00:21:20.000 --> 00:21:21.359
That's a great answer.
00:21:21.680 --> 00:21:22.319
Awesome.
00:21:22.559 --> 00:21:24.240
Well, this has been so great.
00:21:24.400 --> 00:21:27.119
Thank you so much for coming on and talking about your contributions.
00:21:27.279 --> 00:21:31.279
Again, I'll say I think these are really impactful for everybody using Airflow.
00:21:31.359 --> 00:21:35.920
So the community, of course, really appreciates all of the time that you spent on them.
00:21:36.079 --> 00:21:39.440
Uh, just to close this out, what is the best way for folks to get in touch with you?
00:21:39.839 --> 00:21:40.880
Yeah, for sure.
00:21:41.039 --> 00:21:47.359
Anyone can connect me in the GitHub or Airflow Slack uh or LinkedIn if they want uh to connect there.
00:21:47.440 --> 00:21:51.599
I'm always open to talk, uh, have a coffee any anytime, yeah, for sure.
00:21:52.000 --> 00:21:52.640
Okay, great.
00:21:52.799 --> 00:21:54.720
We'll put links to those things in the show notes.
00:21:54.799 --> 00:21:56.640
Uh Anish, thank you again so much.
00:21:56.720 --> 00:21:57.920
It's been great speaking with you.
00:21:58.480 --> 00:22:02.880
Yeah, it was great um being here and thank you very much for uh having me here.
00:22:03.359 --> 00:22:06.240
Thank you for listening to the Data Flowcast.
00:22:06.480 --> 00:22:09.920
Check the show notes for links to everything mentioned today.
00:22:10.079 --> 00:22:14.319
If you're enjoying the podcast, subscribe so you never miss an episode.
00:22:14.960 --> 00:22:17.039
And leave us a five-star review.
00:00:33.520 --> 00:00:35.520
Or click the link in our show notes.
00:00:35.759 --> 00:00:37.679
Now let's get into the show.
00:00:38.560 --> 00:00:39.679
Okay, hi everyone.
00:00:39.840 --> 00:00:41.520
Welcome to the Data Flowcast.
00:00:41.600 --> 00:00:50.960
I'm your host, Kenton Danis, and today I'm joined by Anish Geary, who is a senior software engineer and active contributor to Apache Airflow focused on security.
00:00:51.280 --> 00:00:52.240
Anish, how are you?
00:00:52.640 --> 00:00:53.439
Yeah, I'm doing fine.
00:00:53.520 --> 00:00:54.000
How are you?
00:00:54.079 --> 00:00:55.280
Thanks for having me in today.
00:00:55.679 --> 00:00:56.079
I'm good.
00:00:56.159 --> 00:00:57.439
It's really great to have you on.
00:00:57.520 --> 00:00:59.200
I'm excited to dive into this.
00:00:59.280 --> 00:01:03.600
I always love having people who are contributing to the project on the podcast.
00:01:03.920 --> 00:01:13.920
And for today, for our listeners, we're going to discuss your security contributions to Airflow, including things like connection testing, token revocation, and token scoping.
00:01:14.319 --> 00:01:19.920
I think these are going to end up being things that probably every Airflow user encounters to some degree.
00:01:20.079 --> 00:01:23.040
So it should be very helpful for all of our listeners.
00:01:23.280 --> 00:01:26.239
But before we get into all of that, I want to start with some background.
00:01:26.400 --> 00:01:27.439
So let's talk about you.
00:01:27.599 --> 00:01:29.200
How long have you been using Airflow?
00:01:29.840 --> 00:01:38.879
Yeah, before starting, I want to put a little disclaimer out that all the any views that I put out here is just mine and it does not relate to any organization.
00:01:39.599 --> 00:01:47.359
And yeah, uh, well, yeah, it's funny thing that I'm not a data engineer and I've not uh run any pipelines in production yet.
00:01:47.519 --> 00:01:51.760
I got into Airflow by reading the source code and docs itself.
00:01:52.079 --> 00:01:55.359
And yeah, I see many people coming from outside in.
00:01:55.439 --> 00:01:58.239
My journey was inside out, I believe.
00:01:59.120 --> 00:02:00.319
Yeah, that's cool.
00:02:00.640 --> 00:02:06.159
I think you're right that probably most people are Airflow users themselves and then start contributing.
00:02:06.239 --> 00:02:23.599
Although we do certainly have people at Astronomer who are on the Airflow PMC or who are committers who spend most of their day job working on Airflow, who that is their background as well as they have worked in software and worked in open source software, but they are not using Airflow themselves.
00:02:23.759 --> 00:02:25.919
So you're definitely not alone there.
00:02:26.800 --> 00:02:33.280
And I understand that you, you know, started contributing to Airflow fairly recently, um, like at the end of last year.
00:02:33.439 --> 00:02:38.000
I guess just kind of high-level what got you into open source contribution?
00:02:38.080 --> 00:02:40.800
Like you mentioned, you are not an Airflow user yourself.
00:02:40.879 --> 00:02:43.840
What made you interested in working with the project?
00:02:44.240 --> 00:02:53.199
Yeah, well, well, my whole career is, you know, um dependent on users of uh the open source uh all the time.
00:02:53.360 --> 00:03:00.479
And uh I think I know many people out there are spending their time, their precious time, building this for free.
00:03:00.560 --> 00:03:02.479
And that always fascinated me.
00:03:02.719 --> 00:03:06.400
And at one point of time I wanted to be in the giving side of the spectrum.
00:03:06.479 --> 00:03:10.400
So I started small and got got hooked into it.
00:03:10.960 --> 00:03:12.080
Yeah, that's amazing.
00:03:12.240 --> 00:03:16.800
One of the best things I think about the Airflow project is just the number of people contributing.
00:03:16.960 --> 00:03:23.039
I saw, I think they just passed 4,000 contributors uh to the project, which is so cool.
00:03:23.199 --> 00:03:28.080
And so obviously, a very small fraction of those people are doing it as part of their day job.
00:03:28.159 --> 00:03:34.719
And as you said, a lot of them are doing it just because they want to, they want to make the project better, whether they're using it themselves or not.
00:03:34.879 --> 00:03:36.800
So that's really cool.
00:03:37.120 --> 00:03:42.960
And then we're gonna talk a lot about your uh contributions focused on security.
00:03:43.039 --> 00:03:47.280
And I'm curious what drew you to that sort of work specifically.
00:03:47.759 --> 00:03:50.879
Well, yeah, well, while uh I think it likewise.
00:03:50.960 --> 00:03:55.759
So while we think about software development, we think about uh what can this do.
00:03:55.919 --> 00:04:03.199
But I think when it comes to security, uh the question slips and uh it becomes what can in uh any anyone make this do?
00:04:03.280 --> 00:04:12.800
So this later part of the question always, you know, uh excited me about software development itself, and it drew always drew to me the security side of the engineering.
00:04:13.919 --> 00:04:15.599
Yeah, very interesting.
00:04:15.840 --> 00:04:20.720
Well, let's go ahead and dig into some of the work that you've done on the project recently.
00:04:20.879 --> 00:04:25.360
And I'd love to talk uh first about connection testing.
00:04:25.519 --> 00:04:31.680
So I know this is a feature that was disabled by default in Airflow 2.7 after a CVE was reported on it.
00:04:31.759 --> 00:04:41.680
And I guess for some background, I think probably most Airflow users know this, but this is the feature within the Airflow connections UI where you can input all of the information and then just say test connection.
00:04:41.839 --> 00:04:54.879
So it in theory makes the connection to the external system, but you don't have to run a DAG that uses that connection in a task to see, oh, something about the way that I formatted my credentials or whatever doesn't work.
00:04:55.040 --> 00:04:56.639
So pretty useful feature.
00:04:56.800 --> 00:05:03.360
I have always said myself and my own experience as a data engineer working with Airflow, I think connections are one of the hardest parts.
00:05:03.600 --> 00:05:10.160
It can just be really tricky to get all of the information in them right based on all of these different external systems.
00:05:10.639 --> 00:05:15.439
But again, this feature uh still exists but is disabled by default.
00:05:15.600 --> 00:05:24.800
Um so tell me a little bit, yeah, about your work on this and how what change you contributed and kind of what that does to the way connection testing works.
00:05:25.199 --> 00:05:26.240
Yeah, yeah, for sure.
00:05:26.399 --> 00:05:32.240
So uh while Airflow executes this connection testing, it uh loads the whole driver.
00:05:32.319 --> 00:05:43.040
And I think in some cases it loads it can load the entire class and also you know can do an outbound request with the user-supplied values that uh that comes from that form.
00:05:43.279 --> 00:05:49.040
And that form basically acts as uh you know uh run code in the server uh form.
00:05:49.759 --> 00:05:56.639
And uh and this is specifically why it was disabled in back in uh there in 2.7, I believe.
00:05:56.800 --> 00:06:08.879
And you know, it this connection runs in that time it ran in the web server, which was the most crucial process of the Airflow itself that had all the DB connection and the uh secrets, and that is why it was twist off.
00:06:09.199 --> 00:06:16.720
Now uh with my um contribution, uh what it made is it it just uh made it asynchronous currently.
00:06:16.800 --> 00:06:31.279
What it does is in current mechanism, not the new design, what it does is it makes a connection request uh to the API server, it executes that right there, and UI uh waits for the API server to come back.
00:06:31.360 --> 00:06:44.480
Uh, but with the new design, uh the API server does nothing, uh just records the connection testing uh that it gets and uh hands back UI uh token and the ID so that it can pull uh for the results later on.
00:06:44.639 --> 00:06:49.040
And now the execution part of it happens in uh the worker.
00:06:49.279 --> 00:06:58.319
Uh and here uh you know uh it executes uh with the same mechanism that uh the current tasks execution uh machinery uses.
00:06:58.480 --> 00:07:01.920
So yeah, this is what the actual chains look like.
00:07:02.879 --> 00:07:03.920
Yeah, that makes sense.
00:07:04.000 --> 00:07:16.399
So changing kind of the architecture of how this feature works sounds like also based on the new Airflow 3 architecture now that you do have this separate API server and that workers no longer directly access the metadata database.
00:07:16.480 --> 00:07:17.439
So am I getting that right?
00:07:17.519 --> 00:07:21.519
That because this runs in a worker, now you're not accessing those secrets directly.
00:07:21.759 --> 00:07:25.680
Yeah, yeah, with this new design, it actually became a job, you know.
00:07:25.759 --> 00:07:28.959
And with job, none of the credential passes along with it.
00:07:29.040 --> 00:07:39.600
Uh, and when it reaches workers, it uh does uh API requests to uh the API server spill itself to get the details uh of the connection testing to run it.
00:07:39.680 --> 00:07:44.000
And API server, the pro the worker doesn't have the privilege that API server has.
00:07:44.319 --> 00:07:44.560
Okay.
00:07:44.720 --> 00:07:45.519
Yeah, that's great.
00:07:45.600 --> 00:07:54.639
I would imagine that makes a big de difference for a lot of organizations on whether or not they can use this feature at all or whether their security teams are okay with somebody enabling it.
00:07:55.040 --> 00:07:56.399
Yeah, very cool.
00:07:56.720 --> 00:08:01.279
And I also am curious about the process for making this change.
00:08:01.439 --> 00:08:22.399
So I think uh changes like this that have kind of an architectural component, especially with something that has previously had um, you know, a CVE tied to it or has ongoing potential security issues, something like connections, you know, those often go through a discussion on the dev list uh for airflow and you know an architecture review.
00:08:22.560 --> 00:08:24.079
So is that something you did for this?
00:08:24.160 --> 00:08:25.839
And what did that process look like?
00:08:26.319 --> 00:08:28.560
Yeah, this story is actually interesting.
00:08:28.720 --> 00:08:32.159
So this all started with the PR from a different commuter.
00:08:32.240 --> 00:08:37.600
Uh and uh, you know, it was a bug fix in the current uh connection testing feature.
00:08:37.840 --> 00:08:49.120
There, I saw a comment from Jarek, uh, which whose comment was, you know, uh this the ideal fix is to move this into the worker self and not patch it here.
00:08:49.279 --> 00:08:50.960
So he wrote the all history there.
00:08:51.120 --> 00:08:54.480
I read that comment and I was like, I had to build this uh stuff.
00:08:54.720 --> 00:09:18.320
And yeah, I I uh built up my uh uh design and uh I wired up it and I um took it to the mailing list, and then all the PMC uh members and the commuters came came in, and you know, my design got torn apart and it was even rejected at first, and then it uh got revived along the way on uh you know different iteration.
00:09:18.480 --> 00:09:26.639
I mean, it takes time, but you know, the design that we have uh at last is is, I believe, good, and the thing that is checked in, I'm I'm really proud of.
00:09:27.440 --> 00:09:28.480
Yeah, that's great.
00:09:28.720 --> 00:09:38.639
Definitely, I think a strength of open source projects is the kind of consensus-driven approach, but as you describe, it can also make it quite challenging.
00:09:38.799 --> 00:09:51.519
I have heard stories from lots of people who have come with an idea uh that was then sort of torn apart by a lot of the Airflow PMC or committers and then eventually land on something that is much stronger.
00:09:51.679 --> 00:10:00.399
And so that's that's really cool that you were able to um come to that consensus and um deliver something that, like I said, is really meaningful to a lot of users.
00:10:00.799 --> 00:10:01.679
Yeah, for sure.
00:10:02.000 --> 00:10:03.279
Yeah, that's great.
00:10:03.600 --> 00:10:10.080
Well, I also want to talk about some of the other security-related contributions that you have worked on recently.
00:10:10.240 --> 00:10:14.879
And one of them I understand was this token revocation in Airflow 3.
00:10:15.200 --> 00:10:26.639
I think compared to what we just talked about, this is something that happens much more on the kind of behind the scenes on the back end, so that users probably aren't even aware of if everything is going right.
00:10:26.799 --> 00:10:35.200
Uh, so explain to me a little bit kind of what is token revocation, how does that work, and what was the problem with how it was handled before?
00:10:35.679 --> 00:10:42.159
Yeah, so this basically uh is uh drills down to the nature of the JWT itself.
00:10:42.240 --> 00:10:48.799
You know, it's stateless and server can validate uh JWT without maintaining any state at all.
00:10:48.960 --> 00:10:54.240
And it can uh always validate it if uh the signature is correct, the token is fine.
00:10:54.399 --> 00:11:09.919
So in Airflow, when user did the logout, um uh we just uh in Airflow 3 specifically, um, you know, we just removed that uh token from the browser itself, uh, but the token was still valid because it was it not it was not expired.
00:11:10.080 --> 00:11:17.679
Uh well it has certain default time uh where it would expire, but it was it's not it was not expired for that particular time.
00:11:17.919 --> 00:11:28.639
And you know, anyone uh that could get hold of that token uh you know could act as you and do operation in behalf of you, which which which was not correct uh and ideal.
00:11:28.799 --> 00:11:33.679
Uh so we uh built a revocation system to eliminate this.
00:11:34.159 --> 00:11:51.200
And this uh this revocation system uh in this revocation system we built a tiny table which is just basically a dead list, and uh where we uh you know uh do the capture all of uh the tokens that are expired.
00:11:51.360 --> 00:11:58.320
And uh whenever user clicks log out now, uh we add a record there with its JTI and the expiry token.
00:11:58.559 --> 00:12:06.000
And whenever subsequent request comes with the token, we check that uh the token against that table, and if it exists here, we just reject it.
00:12:06.080 --> 00:12:09.840
So now with this mechanism in place, the logout is actual logout.
00:12:10.399 --> 00:12:18.159
And I want to be a bit clear, this uh revocation is just for the UI tokens, not on the execution side of the tokens that we have.
00:12:18.559 --> 00:12:27.039
And uh yeah, this is how uh we uh we are able to solve uh the actual revocation problem in Airflow 3.
00:12:27.440 --> 00:12:29.200
Okay, got it, makes sense.
00:12:29.440 --> 00:12:34.799
Uh yeah, so I guess, like I said before, uh a change that is completely on the back end.
00:12:34.879 --> 00:12:42.399
So hopefully most users are not aware of this at all when they're working with Airflow, but definitely a big security implication.
00:12:43.120 --> 00:12:44.080
Yeah, very cool.
00:12:45.759 --> 00:12:54.080
This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:12:54.480 --> 00:12:58.320
From beta pipelines to training models and wiring up agents.
00:13:04.879 --> 00:13:07.120
Check it out at astronomer.io.
00:13:08.720 --> 00:13:17.039
And then I understand you have also worked recently on uh uh a project you split Airflow's tokens into two types.
00:13:17.200 --> 00:13:21.039
Was this related to token revocation or is this something totally different?
00:13:21.600 --> 00:13:23.840
Well, yeah, this was totally, totally different.
00:13:24.559 --> 00:13:25.039
Yeah.
00:13:25.279 --> 00:13:30.399
Um it it came out uh it was a very strange issue, actually.
00:13:30.720 --> 00:13:38.639
And uh uh one use it was uh in this case, uh the tax was in the queue for a very long time.
00:13:39.039 --> 00:13:49.679
Uh and uh whenever worker picked the tax again up, uh the tax was uh getting 403 on the first call to the uh execution API.
00:13:49.919 --> 00:14:04.960
And the strange thing was, you know, the Airflow 3 already had uh a refresh uh token mechanism built in, and it uh you know refreshes the token on every subsequent request that comes from uh the uh you know workers.
00:14:05.200 --> 00:14:13.600
Uh but in this case the token was living there and none of the API calls was made to the server, so token refresh never happened.
00:14:14.080 --> 00:14:17.679
And uh as it never happened, it eventually died.
00:14:17.919 --> 00:14:31.600
And for this very case, the user was you know running a GPU pipeline and his uh his tasks was failing uh after 15 minutes, and he was he was uh kind of confused why that was happening because refresh also happens.
00:14:31.679 --> 00:14:51.600
Uh but and also he tried multiple things to it, uh like increasing the uh task queue timeout, which is basically a timeout uh in uh uh until which uh tasks can stay in the queue itself, uh, but that didn't work because it was not tied to the DWT uh expiration at all.
00:14:52.000 --> 00:14:57.919
So actually, with to solve these kind of you know uh invariation, we have to split that token into two.
00:14:59.039 --> 00:14:59.360
Okay.
00:14:59.919 --> 00:15:00.399
Got it.
00:15:00.480 --> 00:15:01.360
That makes sense.
00:15:01.679 --> 00:15:06.639
And then so tell me how does how does splitting those?
00:15:06.799 --> 00:15:11.120
You gave the one example, um maybe more generally.
00:15:11.200 --> 00:15:14.879
So how how does splitting those impact like the end user?
00:15:15.039 --> 00:15:16.879
What would they see working differently?
00:15:17.600 --> 00:15:28.480
So uh well I would say previous uh previous token uh you know uh was one token and having two entirely different jobs.
00:15:28.639 --> 00:15:33.360
That is one was staying in the queue alive and one was authentication itself.
00:15:33.919 --> 00:15:39.039
And uh for uh staying in the queue alive, it needs longer time.
00:15:39.120 --> 00:15:44.080
Uh and uh for you know for authentication it needs a shorter time, actually.
00:15:44.399 --> 00:16:00.720
Uh so basically, and uh with the longer time token as it uh as a workload travels through you know this executor transport, it is uh uh in a you know on a bit of an untrusted environment like uh broker logs and uh the KDS spec itself.
00:16:00.799 --> 00:16:06.080
So it has to be tightly scoped so that the leak token is you know uh kind of unusable.
00:16:06.159 --> 00:16:16.000
And in this case as well, uh the workload token is kind of unusable uh totally because it can only do one thing that is run uh the tax and that's it.
00:16:16.080 --> 00:16:24.480
Uh then when it reaches worker, the execution uh uh you know API already have swapped that token uh while the first request happened.
00:16:24.720 --> 00:16:37.759
So yeah, and the actual uh solving part happened here when we already increased the length of the workload token to be identical to the tax queue timeout timeout because it is now the same.
00:16:37.840 --> 00:16:44.240
The same configuration is picked up for both workload token lifetime and the queue type uh queue tax timeout.
00:16:44.720 --> 00:16:48.879
Uh, so that you know the two clock doesn't fight with each other, it is the same now.
00:16:48.960 --> 00:16:56.799
And uh now, however time uh the uh you know tax is in that queue, it won't uh face the issue that user was facing.
00:16:56.879 --> 00:17:03.279
And also, we already have uh uh uh scope so narrow that security is already baked into it.
00:17:04.079 --> 00:17:05.119
Okay, got it.
00:17:05.279 --> 00:17:06.240
Yeah, that's great.
00:17:06.400 --> 00:17:08.880
What what an impactful change.
00:17:09.119 --> 00:17:15.599
And all all three of these things that we've just talked about, you've done you know, this year basically, or since kind of the end of last year.
00:17:16.480 --> 00:17:17.920
Yeah, that's awesome.
00:17:18.160 --> 00:17:35.519
Um I'm curious, do you have any uh security-related projects that you're currently working on uh with Airflow, or are these uh three uh updates that we just mentioned, are they completely done?
00:17:35.759 --> 00:17:41.920
Are they concluded projects, or is there are there still optimizations that you're hoping to make or future work planned?
00:17:42.480 --> 00:17:48.240
Yeah, so basically two token mechanism is already checked in and it's already working in production.
00:17:48.319 --> 00:17:51.839
And also uh the revocation system is already in production.
00:17:52.000 --> 00:18:03.759
But with the test connection, um the groundwork and the base architecture is checked in, but we uh and only local executor is now uh you know migrating to the new uh format, new job that we have.
00:18:03.920 --> 00:18:14.079
We still have to migrate all of the executors and uh then uh just swap up the um last piece of UI uh and yeah, then then we are done on that part as well.
00:18:14.400 --> 00:18:15.359
Yeah, that's great.
00:18:15.599 --> 00:18:23.680
And for the two that are completed, I should have asked this earlier, but for people to take advantage of those, what version of Airflow do they need to be on?
00:18:24.000 --> 00:18:28.400
Uh they have to be on 3.2 uh and a higher.
00:18:28.960 --> 00:18:29.759
Okay, great.
00:18:30.000 --> 00:18:30.799
Makes sense.
00:18:31.039 --> 00:18:32.160
Yeah, that's awesome.
00:18:32.319 --> 00:18:44.480
Well, and then I always love to talk to people who have started contributing to the project relatively recently about any advice that you would give to someone who is thinking about contributing to Airflow.
00:18:44.559 --> 00:18:55.759
Uh, we talk to folks in the community all the time who want to contribute but feel a little intimidated by both the process, I think can seem quite complex when you're new to it.
00:18:55.839 --> 00:19:02.240
And also I think people have some just sort of factor of, you know, do I want to go through the review process?
00:19:02.319 --> 00:19:04.400
Are people going to accept my changes?
00:19:04.559 --> 00:19:06.640
You know, uh all of that stuff.
00:19:06.799 --> 00:19:12.319
So yeah, what what would you say to anybody who's potentially on the fence about contributing?
00:19:12.720 --> 00:19:13.440
Yeah, for sure.
00:19:13.599 --> 00:19:17.039
I mean, yeah, uh, I would say just start small.
00:19:17.200 --> 00:19:23.119
Uh, there are lots of good first uh issue labeled issues in uh, you know, GitHub.
00:19:23.200 --> 00:19:32.880
And I would say don't be afraid to pull out a PR because you know, in the review process, there is where you get mentored by the best engineers that we have in the community of Airflow.
00:19:33.359 --> 00:19:38.400
And yeah, I would say just don't be afraid, pull out a PR and go get uh started.
00:19:38.559 --> 00:19:39.839
That's it, I think.
00:19:40.160 --> 00:19:41.119
Yeah, that's great.
00:19:41.279 --> 00:19:42.319
I would echo that.
00:19:42.400 --> 00:19:45.680
Uh certainly the advice I always give people is start small.
00:19:45.759 --> 00:19:48.160
Uh that's what I have done for my own contribution.
00:19:48.319 --> 00:19:49.279
Same with my team.
00:19:49.359 --> 00:19:55.519
And you know, everybody on the Airflow PMC or who are reviewing PRs, they actually prefer smaller changes.
00:19:55.680 --> 00:19:57.359
It's a lot easier to review.
00:19:57.519 --> 00:20:06.000
A lot of the stuff that you've been describing is pretty big in terms of functional changes to Airflow, but a contribution certainly does not need to be that large.
00:20:06.400 --> 00:20:23.039
And yeah, I think I've also heard from you know the folks at Astronomer who are PMC members now and some of the most heavy contributors to the project over its history that they also started by just submitting a PR and they got tons of feedback on it and had to go back and rework.
00:20:23.119 --> 00:20:25.759
And, you know, they think it makes them stronger engineers.
00:20:25.920 --> 00:20:29.759
So I would say to people also don't be afraid of the feedback.
00:20:29.920 --> 00:20:33.839
Like nobody expects you to submit something that's perfect the first time.
00:20:34.160 --> 00:20:35.440
Yeah, I totally agree.
00:20:35.839 --> 00:20:36.960
Yeah, that's great.
00:20:37.200 --> 00:20:46.559
Uh well, Anish, I related to contributing, I guess this can be either related to something that you are working on or maybe not, and it's completely different.
00:20:46.720 --> 00:20:54.880
But I always end uh by asking a standard final question, and that is what would you most like to see from development of the Airplug project in the near future?
00:20:55.359 --> 00:21:10.160
Yeah, you know, uh we just checked in this uh uh principle of two token mechanism where we scope things uh of different parts of the airflow really tight so that if any uh authorization or credential linked in, it would be totally useless to attacker.
00:21:10.400 --> 00:21:16.640
So I think I would like to see more of these kinds of scoping uh spread upon all of the creational part of the airflow.
00:21:16.720 --> 00:21:18.720
And yeah, I'll be looking forward for it.
00:21:19.119 --> 00:21:19.920
Okay, great.
00:21:20.000 --> 00:21:21.359
That's a great answer.
00:21:21.680 --> 00:21:22.319
Awesome.
00:21:22.559 --> 00:21:24.240
Well, this has been so great.
00:21:24.400 --> 00:21:27.119
Thank you so much for coming on and talking about your contributions.
00:21:27.279 --> 00:21:31.279
Again, I'll say I think these are really impactful for everybody using Airflow.
00:21:31.359 --> 00:21:35.920
So the community, of course, really appreciates all of the time that you spent on them.
00:21:36.079 --> 00:21:39.440
Uh, just to close this out, what is the best way for folks to get in touch with you?
00:21:39.839 --> 00:21:40.880
Yeah, for sure.
00:21:41.039 --> 00:21:47.359
Anyone can connect me in the GitHub or Airflow Slack uh or LinkedIn if they want uh to connect there.
00:21:47.440 --> 00:21:51.599
I'm always open to talk, uh, have a coffee any anytime, yeah, for sure.
00:21:52.000 --> 00:21:52.640
Okay, great.
00:21:52.799 --> 00:21:54.720
We'll put links to those things in the show notes.
00:21:54.799 --> 00:21:56.640
Uh Anish, thank you again so much.
00:21:56.720 --> 00:21:57.920
It's been great speaking with you.
00:21:58.480 --> 00:22:02.880
Yeah, it was great um being here and thank you very much for uh having me here.
00:22:03.359 --> 00:22:06.240
Thank you for listening to the Data Flowcast.
00:22:06.480 --> 00:22:09.920
Check the show notes for links to everything mentioned today.
00:22:10.079 --> 00:22:14.319
If you're enjoying the podcast, subscribe so you never miss an episode.
00:22:14.960 --> 00:22:17.039
And leave us a five-star review.
