WEBVTT
00:00:00.320 --> 00:00:07.519
You're listening to the Data Flowcast, a podcast about high airflow and the world of data and AI around it.
00:00:07.759 --> 00:00:14.640
We are excited to invite you to orchestrate everything for a free online conference on September 30th.
00:00:15.119 --> 00:00:28.000
Engineering leaders from Liverpool and Wicker will care how they're hoping for Dutch AI workflows on Air Along a credit code for the new Astronomer Certified AI Orchestration Fundamental Data.
00:00:28.559 --> 00:00:34.159
Attendees get a discount code for a free certification for a$150 value.
00:00:34.320 --> 00:00:39.439
Go to astronomer.link slash data minus flowcast minus OE.
00:00:39.759 --> 00:00:41.119
Link in the show notes.
00:00:41.359 --> 00:00:43.039
Now let's get into the show.
00:00:43.759 --> 00:00:44.799
Okay, hi everyone.
00:00:44.960 --> 00:00:46.640
Welcome to the Data Flowcast.
00:00:46.719 --> 00:00:53.200
I'm your host, Kent and Danis, and today I'm joined by Antonello Benedetto, who is a staff data engineer at WISE.
00:00:53.520 --> 00:00:54.960
Antonello, how are you doing?
00:00:55.359 --> 00:00:56.159
I'm doing very well.
00:00:56.240 --> 00:00:58.479
Thanks for the invite, Kent, and thanks for having me.
00:00:58.560 --> 00:00:59.920
I'm very excited about this.
00:01:00.479 --> 00:01:02.079
Yeah, I'm excited as well.
00:01:02.320 --> 00:01:03.679
Thank you so much for joining.
00:01:03.840 --> 00:01:05.040
We'll go ahead and dive in.
00:01:05.200 --> 00:01:14.640
So for today's podcast, we're going to talk about how WISE uses Airflow and some of their insight around things like managing data reliability and LLM-enabled workflows.
00:01:14.799 --> 00:01:17.840
I think this is going to be really useful to our listeners.
00:01:18.079 --> 00:01:22.000
And before we get into some of those details, I want to start with a little bit of background.
00:01:22.079 --> 00:01:25.760
So tell me about WISE, what the company does, and what you do there.
00:01:26.159 --> 00:01:27.519
Yeah, yeah, absolutely.
00:01:27.599 --> 00:01:44.079
So WISE can be seen as a technology company, and uh uh we are helping our customers, you know, moving money across borders, but also uh holding money in uh 40 different currencies via WISE account and uh WISE business.
00:01:44.400 --> 00:02:02.480
And um, you know, like we uh are also uh selling and you know allowing um you know um big companies and uh uh banks to use our technologies uh via WISE platform and offer the same services we use to our customers to their existing customer base.
00:02:02.719 --> 00:02:27.520
And uh just to give you some numbers, uh Kentin, in uh the fiscal year 2026 we uh supported 19 million customers to move uh 240 billion dollars across border and uh we saved um around three billion in fees, so it's uh quite an effort there, and uh uh it we keep, you know, uh it keeps uh growing and we keep you know saving money to our customers.
00:02:27.680 --> 00:02:28.800
That that's great.
00:02:29.120 --> 00:02:31.039
That's great, that's so impactful.
00:02:31.120 --> 00:02:35.520
What a huge amount of customers and money that you're helping with.
00:02:35.759 --> 00:02:37.360
Yeah, very cool to hear about.
00:02:37.520 --> 00:02:40.960
And so tell me more about then what you do at the company and what is your role?
00:02:41.280 --> 00:02:41.919
Yeah, absolutely.
00:02:42.000 --> 00:02:48.960
So I'm currently serving as a staff um analytics engineer in the central data and analytics engineering team.
00:02:49.120 --> 00:02:56.800
And uh really what I do is coordinating technical um you know projects across multiple engineering teams.
00:02:57.039 --> 00:03:13.360
Um particularly I collaborate very closely with platform engineering uh and uh with security engineering engineering, security ops, and the purpose is uh serving analytics uh you know at uh you know according to our best standards.
00:03:13.520 --> 00:03:34.400
Um and so like is mainly a role as an individual contributor, but before that I used to let the uh data engineering effort within finance and treasury, uh so I was working very closely with Airflow and uh you know um analytical pipelines orchestrated via Airflow in finance and treasury to start with.
00:03:34.719 --> 00:03:35.439
Yeah, great.
00:03:35.599 --> 00:03:43.840
And maybe if you can tell me in a little bit more detail about the kinds of problems that your central analytics engineering team is working on.
00:03:43.919 --> 00:03:47.919
And you mentioned using Airflow, and so where does Airflow fit into those problems?
00:03:48.240 --> 00:03:48.960
Yeah, absolutely.
00:03:49.039 --> 00:03:56.800
So you can see you can look at uh the central data and analytics engineering team as a sort of uh hybrid team.
00:03:56.879 --> 00:04:10.800
Uh so I will define ourselves as uh a part platform team because we own actually the infrastructure, the tooling, and the services that actually power our you know analytics and reporting offering.
00:04:10.960 --> 00:04:26.800
Uh, for example, we we own directly DBT infrastructure, uh BI infrastructure within uh within WISE, and uh you know, more recently, uh Analytics MCPs as a service uh to empower people to use AI locally.
00:04:27.120 --> 00:04:40.079
Um and um and on the other side, uh we work as you know analytics engineers, data engineers, uh managing ETL pipelines uh you know on uh uh on behalf of tribes.
00:04:40.319 --> 00:04:53.360
Uh, in particular, we manage uh you know um uh a pretty small amount of uh critical data sets that uh um you know are orchestrated with Airflow natively.
00:04:53.600 --> 00:05:02.800
And uh the reason why they are critical is because they are highly referenced uh you know downstream by uh thousands, I would say, of DBT models.
00:05:02.879 --> 00:05:09.040
Uh so and you know Airflow is um is actually uh on top of this orchestration.
00:05:09.279 --> 00:05:28.879
So, like it's uh as I said, an hybrid team, and every um you know, every analytics engineer in our team is um you know embedded in a tribe, so they are contributing to the work and technical initiatives within a tribe, say finance servicing, treasury marketing, as well as you know, contributing to central projects.
00:05:28.959 --> 00:05:52.240
So, in this way, uh people working in our team, you know, really have an opportunity to work across a very wide range of problems and uh get experience uh, you know, uh on the pure data engineering uh role, as well as you know, like also uh having uh exposure to platform work that is is quite uh empowering.
00:05:52.959 --> 00:05:54.480
Yeah, that's very cool.
00:05:54.720 --> 00:06:00.959
And so I'd love to get a little bit more insight from you on kind of the principles that your team works with.
00:06:01.120 --> 00:06:03.759
You mentioned that you're managing these critical data sets.
00:06:03.920 --> 00:06:20.480
And in my experience, finance companies or anybody dealing with financial data, you're obviously very highly regulated, very sensitive information, and a very high bar for the data quality, because if it's wrong, that's gonna have a huge negative impact on your customers.
00:06:20.800 --> 00:06:32.720
So when you're working with Airflow and these pipelines that help you manage this data, you know, what principles guide how your team thinks about making sure that those workflows are maintainable and that they're producing reliable data?
00:06:33.040 --> 00:06:34.800
Yeah, it's uh an excellent question, actually.
00:06:34.879 --> 00:06:43.920
And uh, you know, your point uh about like us being uh a financial uh like a fintech company and uh being quite regulated is uh spot on.
00:06:44.000 --> 00:06:49.120
Um I would say uh there are three principles that come to mind while managing pipelines.
00:06:49.279 --> 00:06:55.519
Uh uh first one is uh a net separation in between uh computation and orchestration logic.
00:06:55.680 --> 00:07:04.879
So I've seen in the past um uh you know people using uh airflow uh in a way that is is mixing the two uh components.
00:07:05.040 --> 00:07:24.160
Whereas I believe a well-structured um you know, like Airflow instance is one where uh let's say the DAG skeleton and the DAG sort of uh components uh are kept separate from like the Python um logic that gets executed via Airflow.
00:07:24.319 --> 00:07:27.279
Airflow at the end of the day is like an orchestration tool.
00:07:27.439 --> 00:07:51.920
Um, and this allows us to actually maintain the code much more easily, make it more readable because it separates the two um the two pieces of logic completely, and also it allows us to touch the least amount of uh let's say uh files as possible while fixing a bug, for example, because either the bug is uh identified in the orchestration portion or in the computation portion.
00:07:52.000 --> 00:08:00.800
So this is like um an extremely important principle for us to keep modularity and uh keep our uh you know uh pipelines reliable.
00:08:01.040 --> 00:08:07.040
The second um principle is that we are really computational aware uh as a as a team.
00:08:07.279 --> 00:08:19.600
So we know again that um uh the the strength of airflow relies in the orchestration, despite really you could do uh everything with with Airflow with a little bit of creativity.
00:08:19.759 --> 00:08:29.279
Uh, we know that at our scale uh we cannot really rely on airflow and we cannot really bombard Airflow with uh uh uh memory-intense um tasks.
00:08:29.439 --> 00:08:57.600
So memory-intensive tasks are usually uh redistributed to external uh workers, um, you know, being that on EWS EMR or SageMaker, uh keeping Airflow tasks really light, and because Airflow has such a great integration with all these external tools, it's so easy to actually develop the orchestration portion and deliver the job or the task somewhere else to be executed.
00:08:57.759 --> 00:09:11.519
Um, so like uh the the last principle is uh making uh standardizing a lot uh how you know uh Airflow is uh deployed centrally but but keeping the flexibility for uh stakeholders.
00:09:11.600 --> 00:09:34.159
And what I mean is um you know uh migrating to a new Airflow version um uh initially required quite a lot of effort because uh we were all using different you know deployments of uh the Docker image, whereas uh more recently we moved uh you know to Python to using Python virtual environment operator.
00:09:34.399 --> 00:09:47.360
And this means that basically we can deploy a standardized version uh across all our Airflow instances by keeping the you know flexibility for the analysts or data scientists uh downstream.
00:09:47.440 --> 00:09:57.519
So they can create their own environment, they can uh choose what Python package they they need, what type of environment they want to create without impacting uh on anyone else.
00:09:57.679 --> 00:10:00.639
So they are pretty isolated as a jobs uh jobs.
00:10:00.799 --> 00:10:08.000
So in this way, we make platform engineering happy and uh you know also stakeholders happy at the same time.
00:10:08.080 --> 00:10:11.919
So I would say these three principles are uh really guiding us.
00:10:12.399 --> 00:10:12.879
Got it.
00:10:13.039 --> 00:10:22.000
That's great that you've been able to implement that sort of standardization while also still giving teams a lot of flexibility in how they're using Airflow.
00:10:22.159 --> 00:10:33.279
I think sometimes those things can be a little bit mutually exclusive at some companies, and it makes it really hard for data engineers to you know try and find the best way of doing things.
00:10:33.440 --> 00:10:58.559
So yeah, yeah, I can't absolutely can because uh uh if you wish our role uh is um let's say translating it translating you know like technical concepts uh coming from platform to something that uh you know like less technical stakeholders can digest, as well as uh on the other side uh translating uh you know business requirements into automation.
00:10:58.720 --> 00:11:14.879
So uh I will define our team sometimes as a translator team, an intermediate team that is acting as a sort of uh let's say that is uh fostering communication in between uh pure platform engineering and analytics function itself.
00:11:15.840 --> 00:11:17.360
Yeah, very cool.
00:11:17.679 --> 00:11:25.039
And I want to talk a little bit more about the reliability aspect of the data management that you're doing.
00:11:25.200 --> 00:11:36.799
I know a lot of engineering teams use some form of tiering to try and figure out uh I guess what what sort of level the data is at and how accurate and how clean it is.
00:11:37.039 --> 00:11:44.559
So, how do you think about prioritizing the reliability for these you know critical or high impact data sets that you've mentioned?
00:11:44.960 --> 00:11:46.080
Yeah, uh absolutely.
00:11:46.320 --> 00:11:55.519
So um, first of all, like in um at wise, I would say particularly in our team, uh, we can distinguish uh three flavors of uh pipelines.
00:11:55.600 --> 00:12:05.039
We have like the uh pure Python uh uh DAG in Airflow, then we have like the dbt uh sort of DAG and the what I call hybrid DAGs.
00:12:05.200 --> 00:12:13.200
So uh those DAGs that um run both dbt pure tasks as well as um as well as uh Python tasks.
00:12:13.360 --> 00:12:41.039
This gives us quite a lot of flexibility and uh over time you know like we uh we built we know we ended up managing these pipelines that are highly uh reliable and highly referenced, and uh we had to introduce a tiering system centrally, and the uh the the tiering system um rely is is needed to distinguish those highly reliable, highly controlled pipelines from like new pipelines, right?
00:12:41.200 --> 00:12:44.639
So those pipelines pipelines that are under development.
00:12:44.879 --> 00:13:07.200
Um and um, you know, like generally what what that means is that um uh I tier uh pipelines are also highly controlled, and they are also you know um they are also extremely well documented and also observable observability on those is uh much stricter and more stringent.
00:13:07.360 --> 00:13:23.840
Um, but generally um uh let's say all uh all pipelines can be moved up uh in tier uh by allowing people to let's say understand what is missing to move from uh a lower tier to an higher tier, if that makes sense.
00:13:24.480 --> 00:13:25.279
Yeah, it does.
00:13:25.360 --> 00:13:36.639
And so to make sure I'm understanding correctly, so it's maybe let the tiering is less about the data that is being produced or managed by the pipeline, but about the pipeline code itself.
00:13:36.720 --> 00:13:41.679
And so how kind of ready is it to produce something that is trusted in production?
00:13:42.080 --> 00:13:42.559
Absolutely.
00:13:42.720 --> 00:13:56.559
So it's uh it's it's about um um it's about uh the the infrastructure behind the pipelines, you know, like the the level of of controls being um uh you know implemented on top of the pipeline itself.
00:13:56.639 --> 00:14:00.879
Um and also do we have metadata about uh the owner?
00:14:01.039 --> 00:14:08.720
Do we uh let's say do we um do we have SLAs about um a specific pipeline?
00:14:08.879 --> 00:14:12.159
Did we agree them with uh with a specific team?
00:14:12.240 --> 00:14:16.799
Uh has it been running for a long time or is it just a new effort, right?
00:14:17.039 --> 00:14:30.720
And uh you can have two pipelines um in two different teams like that we that we own centrally uh that uh you know meet the same very high standards and tier, uh but they are serving two different purposes, right?
00:14:30.879 --> 00:14:40.720
So um some of them have uh certain SLAs because uh we reached uh specific agreements with stakeholders, and others might have like uh different SLAs.
00:14:40.879 --> 00:14:46.480
So we uh the the real question that we answer is uh uh what's the use case?
00:14:46.639 --> 00:14:49.519
What does the customer require?
00:14:49.679 --> 00:14:57.600
Um, you know, in principle, and uh how quickly do we need to uh let's say fix the pipeline in case it you know it breaks.
00:14:57.679 --> 00:14:59.200
So this is a real question.
00:14:59.279 --> 00:15:06.080
Um and this imposes different SLAs, uh, but the tiering system uh is uh standardized.
00:15:06.159 --> 00:15:24.879
And as I was saying, um our guidelines on uh um how to uh let's say promote pipelines uh to higher system uh tiering uh are really transparent and we encourage analysts downstream to to also uh look at these improvements uh constantly.
00:15:25.360 --> 00:15:27.039
Yeah, that's a cool methodology.
00:15:27.120 --> 00:15:43.919
And I'm curious, do you have a sense of how much time typically it takes for if say somebody is doing like a totally new pipeline for a new use case for them to move up into kind of the highest tier of reliability?
00:15:44.080 --> 00:15:48.000
Or are there some pipelines that never get to that highest tier because they don't need to be?
00:15:48.159 --> 00:15:49.360
Or what does that look like?
00:15:49.759 --> 00:15:55.679
It's uh it's pretty much uh uh let's say that is uh like a sort of it comes both ways.
00:15:55.840 --> 00:16:04.480
There are some pipelines that uh will never you know reach that uh top tier uh level because uh it's not needed, right?
00:16:04.559 --> 00:16:07.919
So it's not needed, uh the effort is not needed needed downstream.
00:16:08.080 --> 00:16:18.639
The um let's say the the perhaps the the customer, the stakeholders is not requiring requiring that level of uh complexity, that level of uh efficient efficiency and controls.
00:16:18.879 --> 00:16:27.039
There are other pipelines that naturally are pushed uh upper and upper because there's uh probably a requirement coming from from below.
00:16:27.120 --> 00:16:33.679
And usually requirements coming from below uh tend to speed up the process of uh pushing the tier up.
00:16:33.759 --> 00:16:42.720
Uh, but we also centrally we tend to encourage analysts to uh you know move and promote uh pipelines to higher tiers when possible.
00:16:43.039 --> 00:16:44.639
Yeah, very interesting.
00:16:44.879 --> 00:16:46.320
Yeah, I love learning about that.
00:16:46.399 --> 00:16:55.600
That is a pattern that I don't think I've heard about before, but it totally makes sense for a way of standardizing internally, especially around that sensitive data.
00:16:55.759 --> 00:16:57.200
So, yeah, that's great to hear.
00:16:57.519 --> 00:17:21.119
And then and then as we uh Kent, and also to add to that, as we build a semantic layer, of course, uh having pipelines that are already uh a top top-tier pipelines gives us um sort of uh um foundational sort of information on what should be queried, like where um good quality data comes from, where we should retrieve information uh to start with, right?
00:17:21.200 --> 00:17:25.519
So it gives us a good foundational layer on top of which we can keep building, right?
00:17:25.680 --> 00:17:33.839
Or uh on top of which we can keep uh building downstream DBT models as well as uh you know LLM applications, for example, right?
00:17:35.359 --> 00:17:43.599
This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:17:44.000 --> 00:17:47.920
From beta pipelines to training models and wiring up agents.
00:17:48.160 --> 00:17:54.160
Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
00:17:54.400 --> 00:17:56.640
Check it out at astronomer.io.
00:17:57.839 --> 00:18:09.359
Changing topics a little bit, I also want to talk about the LLM-enabled workflows that you've been working on since this is, of course, a very hot topic right now and something I think a lot of teams are grappling with.
00:18:09.519 --> 00:18:12.799
This is a big you know emerging area for orchestration.
00:18:13.119 --> 00:18:22.000
So at a high level, again, thinking about the principles that you're using again within the context of working with some very sensitive data.
00:18:22.480 --> 00:18:31.200
What do you think that teams should keep in mind when they're experimenting with you know adding LLMs into their workflows or orchestrating agents or something like that?
00:18:31.599 --> 00:18:38.640
Yeah, that's a great question because um, you know, at first we have been experimenting with uh LLM-enabled workflows indeed.
00:18:39.119 --> 00:18:55.119
And um, of course, uh the uh you know, like uh on the surface, uh an LLM um let's say uh workflow orchestrated by airflow might seem very might look very similar to a standard pipeline, but actually it's not, right?
00:18:55.279 --> 00:19:04.799
Because um uh what what changes is the the you know um the sort of uh uh let's say control being applied on top of the pipeline itself.
00:19:04.960 --> 00:19:53.599
So if a standard pipeline is using standard controls right on uh on code or uh tests or unit tests or uh integration tests, uh when we um let's say talk about uh LLM-enabled uh workflows, we look more at uh validation layers, so building validation layers for specific use cases that are basically uh you know sense-checking what uh the LLM has generated as an output, as well as uh having also um observability platform as well as an evaluation platform that instead of looking at a specific uh use case like the validation layer does, looks uh looks at the performance of the model across a number of uh executions, right?
00:19:53.680 --> 00:19:55.920
So it's a more general layer.
00:19:56.160 --> 00:20:07.680
Um and um, for example, what um uh in an example of how to use um LLMs um via Airflow could be like the generation of technical commentary, right?
00:20:07.759 --> 00:20:19.039
So there's the this technical commentary that needs to be uh uh generated by analysts on a recurrent basis, uh, for example, monthly to perform variance analysis.
00:20:19.200 --> 00:20:28.720
And uh it's sometimes very tedious and time consuming for analysts adding to repeat the same process, you know, um uh let's say month on month, for example.
00:20:28.960 --> 00:20:48.079
So uh what what LLM um uh programmatic LLM workflow uh allows us to do is to actually collect data and then expose this data to um uh a large language model, and the large language model will then uh uh following some instructions generate commentary, for example, right?
00:20:48.319 --> 00:20:56.240
Um but uh but again um uh the the problem uh with uh LLM output is that it's not idempotent, right?
00:20:56.319 --> 00:21:09.279
Uh like like with standard Python pipelines and uh is pluripotent uh on the other side and can hallucinate, even though we see hallucination less and less recently with more advanced models.
00:21:09.359 --> 00:21:19.359
But still, uh being it pluripotent means that we need to be much more uh strict in terms of uh applying validation layers and observability and value and evaluation.
00:21:20.079 --> 00:21:20.960
Yeah, totally.
00:21:21.119 --> 00:21:28.400
I think that non-itempotent or non-deterministic aspect of working with these pipelines is maybe one of the hardest pieces.
00:21:28.559 --> 00:21:33.119
As I've seen, you know, lots of companies try to implement these workflows with Airflow.
00:21:33.200 --> 00:21:40.559
It always makes me think of the number of times that I've told people that a best practice is designing that your Airflow DAGs so that they are itempotent.
00:21:40.640 --> 00:21:42.640
And as you said, in this case, you really can't.
00:21:42.880 --> 00:21:44.160
And that's a big challenge.
00:21:44.319 --> 00:21:58.319
I'm curious if you think that you know, when taking a lot of care to build in some guardrails around that, as you have, do you think that solves the issue or at least mostly solves the issue?
00:21:58.480 --> 00:21:59.759
Like, can you get around that?
00:22:00.240 --> 00:24:01.279
non-deterministic aspect of it with enough sort of proper pipeline design and monitoring and observing yeah yeah actually I think we we we can uh in practice uh by applying this uh multi-layer validation as well as um let's say because at the end of the day what we are doing is passing some uh data that is our source of truth to let's say an LLM right so uh we need to verify whether the LLM is coming back with uh something sensible and something that can be uh you know um that can be verified by actually querying the source of truth data and that can be implemented programmatically so actually um all um uh despite the commentary wording can be uh slightly different from execution to execution uh the the directionality of like the variance analysis for example and the data like the actual figures and percentages being used to build a commentary need to match perfectly and that can be validated and um yeah so uh more recently um we also started thinking about having agents validating agents so having like uh a second agent validating the output of uh uh in an initial LLM call um and also implementing a way to uh let's say um you know how uh agents are also used for uh CICD checks to validate whether you know um uh your your your PR the code that you are that you included in in the last commit is valid or uh includes bugs right having also a CI CD check where uh an agent is checking the suggestion recommended by the initial agent is something that we are uh you know uh really pushing to implement internally so my my answer in the in not so short is uh yes with uh with you care everything can be actually replicated programmatically and uh tested.
00:24:02.160 --> 00:24:04.160
I think that's a great perspective.
00:24:04.480 --> 00:26:03.920
And as maybe a follow-up to that discussion on the reliability of these pipelines other than the item potency aspect is there anything that you have seen related to orchestrating AI-enabled workflows that differs from kind of more traditional airflow use cases like batch ETL uh in terms of the repeatability and monitoring aspect um yeah I mean the um I would say mainly um the way um uh the uh evaluation and uh validation is performed uh as we said uh but in terms of um let's let's say building the uh pipeline itself um uh the I would say the surface of um uh let's say how uh how um how advanced your agentic application can be can differ quite a lot from use case uh to use case for example there there might be um uh an AI enabled workflow that is just running a single call to an LLM and in that case a very well uh written prompt with a good validation layer on top can do the job and um let's say uh get um uh outputs close to hidden potency whereas there are some other cases that can still be orchestrated via airflow where uh fully fledged um uh agents are used with uh with a long graph and then the agent is um um let's say left on uh sort of in on it on its own uh and can use and leverage different tools and in that case um let's say there's much more room for error so I would say uh let's start small when trying uh for the first time like to to to work with this AI enabled workflow and then let's build from there uh more complex agents.
00:26:04.640 --> 00:26:06.559
Yeah I think that makes a lot of sense.
00:26:06.720 --> 00:26:22.160
Well thank you for going through that this has been really insightful and I'd love to hear about your experience in these areas I think that's a good place to transition to my standard standard final question for all of my guests is what would you most like to see from the Airflow project in the near future?
00:26:22.480 --> 00:27:41.200
Yeah absolutely um I was actually uh really keen to see um and to uh let's say all the advancements recently after the 2025 summit and uh I'm really keen to uh you know to see uh Airflow becoming more and more a platform also for uh uh you know uh agentic workflows and having some sort of native code to uh let's say implement uh uh validation and also to to remove to remove boilerplate code so to standardize a little bit more the way uh agents um uh are uh orchestrated and also like I would say I would say a bottleneck for us internally has always been how we test uh airflow uh you know locally uh and you know um with a lot of iteration we managed to get to let's say a very strong and solid uh local environment but this is not always easy if you're not an engineer so if you know if you have no um if you have not like an advanced knowledge about uh the components uh um behind airflow so making local uh development easier and closer to to production is always something that uh I think uh there's uh quite a lot of work that can be done uh there um and uh I would I would like to see how it goes in the future.
00:27:41.680 --> 00:27:45.279
Definitely I think those are both great answers and very timely.
00:27:45.519 --> 00:27:55.279
Well Antonella this has been a really great conversation really appreciate you sharing uh all of your experience just to close this out what is the best way for folks to get in touch with you?
00:27:55.599 --> 00:28:51.920
Yeah you can reach me on LinkedIn I'm very active there and very keen to respond to all your questions either about this uh uh podcast or any other uh let's say uh topic related to data and analytics engineering um and also um uh make sure if you if you like what you know what we discussed uh feel free to share our uh uh to to to check our career page uh um on the you know wise website and there are a lot of roles open in engineering and analytics engineering and again Kenton thanks for uh having me and thanks for the invite yeah thank you so much for coming on we will throw links to those things in the show notes so people can access them uh yeah thank you again this has been great thank you for listening to the data flowcast check the show notes for links to everything mentioned today if you're enjoying the podcast subscribe so you never miss an episode and leave us a five star review
00:00:00.320 --> 00:00:07.519
You're listening to the Data Flowcast, a podcast about high airflow and the world of data and AI around it.
00:00:07.759 --> 00:00:14.640
We are excited to invite you to orchestrate everything for a free online conference on September 30th.
00:00:15.119 --> 00:00:28.000
Engineering leaders from Liverpool and Wicker will care how they're hoping for Dutch AI workflows on Air Along a credit code for the new Astronomer Certified AI Orchestration Fundamental Data.
00:00:28.559 --> 00:00:34.159
Attendees get a discount code for a free certification for a$150 value.
00:00:34.320 --> 00:00:39.439
Go to astronomer.link slash data minus flowcast minus OE.
00:00:39.759 --> 00:00:41.119
Link in the show notes.
00:00:41.359 --> 00:00:43.039
Now let's get into the show.
00:00:43.759 --> 00:00:44.799
Okay, hi everyone.
00:00:44.960 --> 00:00:46.640
Welcome to the Data Flowcast.
00:00:46.719 --> 00:00:53.200
I'm your host, Kent and Danis, and today I'm joined by Antonello Benedetto, who is a staff data engineer at WISE.
00:00:53.520 --> 00:00:54.960
Antonello, how are you doing?
00:00:55.359 --> 00:00:56.159
I'm doing very well.
00:00:56.240 --> 00:00:58.479
Thanks for the invite, Kent, and thanks for having me.
00:00:58.560 --> 00:00:59.920
I'm very excited about this.
00:01:00.479 --> 00:01:02.079
Yeah, I'm excited as well.
00:01:02.320 --> 00:01:03.679
Thank you so much for joining.
00:01:03.840 --> 00:01:05.040
We'll go ahead and dive in.
00:01:05.200 --> 00:01:14.640
So for today's podcast, we're going to talk about how WISE uses Airflow and some of their insight around things like managing data reliability and LLM-enabled workflows.
00:01:14.799 --> 00:01:17.840
I think this is going to be really useful to our listeners.
00:01:18.079 --> 00:01:22.000
And before we get into some of those details, I want to start with a little bit of background.
00:01:22.079 --> 00:01:25.760
So tell me about WISE, what the company does, and what you do there.
00:01:26.159 --> 00:01:27.519
Yeah, yeah, absolutely.
00:01:27.599 --> 00:01:44.079
So WISE can be seen as a technology company, and uh uh we are helping our customers, you know, moving money across borders, but also uh holding money in uh 40 different currencies via WISE account and uh WISE business.
00:01:44.400 --> 00:02:02.480
And um, you know, like we uh are also uh selling and you know allowing um you know um big companies and uh uh banks to use our technologies uh via WISE platform and offer the same services we use to our customers to their existing customer base.
00:02:02.719 --> 00:02:27.520
And uh just to give you some numbers, uh Kentin, in uh the fiscal year 2026 we uh supported 19 million customers to move uh 240 billion dollars across border and uh we saved um around three billion in fees, so it's uh quite an effort there, and uh uh it we keep, you know, uh it keeps uh growing and we keep you know saving money to our customers.
00:02:27.680 --> 00:02:28.800
That that's great.
00:02:29.120 --> 00:02:31.039
That's great, that's so impactful.
00:02:31.120 --> 00:02:35.520
What a huge amount of customers and money that you're helping with.
00:02:35.759 --> 00:02:37.360
Yeah, very cool to hear about.
00:02:37.520 --> 00:02:40.960
And so tell me more about then what you do at the company and what is your role?
00:02:41.280 --> 00:02:41.919
Yeah, absolutely.
00:02:42.000 --> 00:02:48.960
So I'm currently serving as a staff um analytics engineer in the central data and analytics engineering team.
00:02:49.120 --> 00:02:56.800
And uh really what I do is coordinating technical um you know projects across multiple engineering teams.
00:02:57.039 --> 00:03:13.360
Um particularly I collaborate very closely with platform engineering uh and uh with security engineering engineering, security ops, and the purpose is uh serving analytics uh you know at uh you know according to our best standards.
00:03:13.520 --> 00:03:34.400
Um and so like is mainly a role as an individual contributor, but before that I used to let the uh data engineering effort within finance and treasury, uh so I was working very closely with Airflow and uh you know um analytical pipelines orchestrated via Airflow in finance and treasury to start with.
00:03:34.719 --> 00:03:35.439
Yeah, great.
00:03:35.599 --> 00:03:43.840
And maybe if you can tell me in a little bit more detail about the kinds of problems that your central analytics engineering team is working on.
00:03:43.919 --> 00:03:47.919
And you mentioned using Airflow, and so where does Airflow fit into those problems?
00:03:48.240 --> 00:03:48.960
Yeah, absolutely.
00:03:49.039 --> 00:03:56.800
So you can see you can look at uh the central data and analytics engineering team as a sort of uh hybrid team.
00:03:56.879 --> 00:04:10.800
Uh so I will define ourselves as uh a part platform team because we own actually the infrastructure, the tooling, and the services that actually power our you know analytics and reporting offering.
00:04:10.960 --> 00:04:26.800
Uh, for example, we we own directly DBT infrastructure, uh BI infrastructure within uh within WISE, and uh you know, more recently, uh Analytics MCPs as a service uh to empower people to use AI locally.
00:04:27.120 --> 00:04:40.079
Um and um and on the other side, uh we work as you know analytics engineers, data engineers, uh managing ETL pipelines uh you know on uh uh on behalf of tribes.
00:04:40.319 --> 00:04:53.360
Uh, in particular, we manage uh you know um uh a pretty small amount of uh critical data sets that uh um you know are orchestrated with Airflow natively.
00:04:53.600 --> 00:05:02.800
And uh the reason why they are critical is because they are highly referenced uh you know downstream by uh thousands, I would say, of DBT models.
00:05:02.879 --> 00:05:09.040
Uh so and you know Airflow is um is actually uh on top of this orchestration.
00:05:09.279 --> 00:05:28.879
So, like it's uh as I said, an hybrid team, and every um you know, every analytics engineer in our team is um you know embedded in a tribe, so they are contributing to the work and technical initiatives within a tribe, say finance servicing, treasury marketing, as well as you know, contributing to central projects.
00:05:28.959 --> 00:05:52.240
So, in this way, uh people working in our team, you know, really have an opportunity to work across a very wide range of problems and uh get experience uh, you know, uh on the pure data engineering uh role, as well as you know, like also uh having uh exposure to platform work that is is quite uh empowering.
00:05:52.959 --> 00:05:54.480
Yeah, that's very cool.
00:05:54.720 --> 00:06:00.959
And so I'd love to get a little bit more insight from you on kind of the principles that your team works with.
00:06:01.120 --> 00:06:03.759
You mentioned that you're managing these critical data sets.
00:06:03.920 --> 00:06:20.480
And in my experience, finance companies or anybody dealing with financial data, you're obviously very highly regulated, very sensitive information, and a very high bar for the data quality, because if it's wrong, that's gonna have a huge negative impact on your customers.
00:06:20.800 --> 00:06:32.720
So when you're working with Airflow and these pipelines that help you manage this data, you know, what principles guide how your team thinks about making sure that those workflows are maintainable and that they're producing reliable data?
00:06:33.040 --> 00:06:34.800
Yeah, it's uh an excellent question, actually.
00:06:34.879 --> 00:06:43.920
And uh, you know, your point uh about like us being uh a financial uh like a fintech company and uh being quite regulated is uh spot on.
00:06:44.000 --> 00:06:49.120
Um I would say uh there are three principles that come to mind while managing pipelines.
00:06:49.279 --> 00:06:55.519
Uh uh first one is uh a net separation in between uh computation and orchestration logic.
00:06:55.680 --> 00:07:04.879
So I've seen in the past um uh you know people using uh airflow uh in a way that is is mixing the two uh components.
00:07:05.040 --> 00:07:24.160
Whereas I believe a well-structured um you know, like Airflow instance is one where uh let's say the DAG skeleton and the DAG sort of uh components uh are kept separate from like the Python um logic that gets executed via Airflow.
00:07:24.319 --> 00:07:27.279
Airflow at the end of the day is like an orchestration tool.
00:07:27.439 --> 00:07:51.920
Um, and this allows us to actually maintain the code much more easily, make it more readable because it separates the two um the two pieces of logic completely, and also it allows us to touch the least amount of uh let's say uh files as possible while fixing a bug, for example, because either the bug is uh identified in the orchestration portion or in the computation portion.
00:07:52.000 --> 00:08:00.800
So this is like um an extremely important principle for us to keep modularity and uh keep our uh you know uh pipelines reliable.
00:08:01.040 --> 00:08:07.040
The second um principle is that we are really computational aware uh as a as a team.
00:08:07.279 --> 00:08:19.600
So we know again that um uh the the strength of airflow relies in the orchestration, despite really you could do uh everything with with Airflow with a little bit of creativity.
00:08:19.759 --> 00:08:29.279
Uh, we know that at our scale uh we cannot really rely on airflow and we cannot really bombard Airflow with uh uh uh memory-intense um tasks.
00:08:29.439 --> 00:08:57.600
So memory-intensive tasks are usually uh redistributed to external uh workers, um, you know, being that on EWS EMR or SageMaker, uh keeping Airflow tasks really light, and because Airflow has such a great integration with all these external tools, it's so easy to actually develop the orchestration portion and deliver the job or the task somewhere else to be executed.
00:08:57.759 --> 00:09:11.519
Um, so like uh the the last principle is uh making uh standardizing a lot uh how you know uh Airflow is uh deployed centrally but but keeping the flexibility for uh stakeholders.
00:09:11.600 --> 00:09:34.159
And what I mean is um you know uh migrating to a new Airflow version um uh initially required quite a lot of effort because uh we were all using different you know deployments of uh the Docker image, whereas uh more recently we moved uh you know to Python to using Python virtual environment operator.
00:09:34.399 --> 00:09:47.360
And this means that basically we can deploy a standardized version uh across all our Airflow instances by keeping the you know flexibility for the analysts or data scientists uh downstream.
00:09:47.440 --> 00:09:57.519
So they can create their own environment, they can uh choose what Python package they they need, what type of environment they want to create without impacting uh on anyone else.
00:09:57.679 --> 00:10:00.639
So they are pretty isolated as a jobs uh jobs.
00:10:00.799 --> 00:10:08.000
So in this way, we make platform engineering happy and uh you know also stakeholders happy at the same time.
00:10:08.080 --> 00:10:11.919
So I would say these three principles are uh really guiding us.
00:10:12.399 --> 00:10:12.879
Got it.
00:10:13.039 --> 00:10:22.000
That's great that you've been able to implement that sort of standardization while also still giving teams a lot of flexibility in how they're using Airflow.
00:10:22.159 --> 00:10:33.279
I think sometimes those things can be a little bit mutually exclusive at some companies, and it makes it really hard for data engineers to you know try and find the best way of doing things.
00:10:33.440 --> 00:10:58.559
So yeah, yeah, I can't absolutely can because uh uh if you wish our role uh is um let's say translating it translating you know like technical concepts uh coming from platform to something that uh you know like less technical stakeholders can digest, as well as uh on the other side uh translating uh you know business requirements into automation.
00:10:58.720 --> 00:11:14.879
So uh I will define our team sometimes as a translator team, an intermediate team that is acting as a sort of uh let's say that is uh fostering communication in between uh pure platform engineering and analytics function itself.
00:11:15.840 --> 00:11:17.360
Yeah, very cool.
00:11:17.679 --> 00:11:25.039
And I want to talk a little bit more about the reliability aspect of the data management that you're doing.
00:11:25.200 --> 00:11:36.799
I know a lot of engineering teams use some form of tiering to try and figure out uh I guess what what sort of level the data is at and how accurate and how clean it is.
00:11:37.039 --> 00:11:44.559
So, how do you think about prioritizing the reliability for these you know critical or high impact data sets that you've mentioned?
00:11:44.960 --> 00:11:46.080
Yeah, uh absolutely.
00:11:46.320 --> 00:11:55.519
So um, first of all, like in um at wise, I would say particularly in our team, uh, we can distinguish uh three flavors of uh pipelines.
00:11:55.600 --> 00:12:05.039
We have like the uh pure Python uh uh DAG in Airflow, then we have like the dbt uh sort of DAG and the what I call hybrid DAGs.
00:12:05.200 --> 00:12:13.200
So uh those DAGs that um run both dbt pure tasks as well as um as well as uh Python tasks.
00:12:13.360 --> 00:12:41.039
This gives us quite a lot of flexibility and uh over time you know like we uh we built we know we ended up managing these pipelines that are highly uh reliable and highly referenced, and uh we had to introduce a tiering system centrally, and the uh the the tiering system um rely is is needed to distinguish those highly reliable, highly controlled pipelines from like new pipelines, right?
00:12:41.200 --> 00:12:44.639
So those pipelines pipelines that are under development.
00:12:44.879 --> 00:13:07.200
Um and um, you know, like generally what what that means is that um uh I tier uh pipelines are also highly controlled, and they are also you know um they are also extremely well documented and also observable observability on those is uh much stricter and more stringent.
00:13:07.360 --> 00:13:23.840
Um, but generally um uh let's say all uh all pipelines can be moved up uh in tier uh by allowing people to let's say understand what is missing to move from uh a lower tier to an higher tier, if that makes sense.
00:13:24.480 --> 00:13:25.279
Yeah, it does.
00:13:25.360 --> 00:13:36.639
And so to make sure I'm understanding correctly, so it's maybe let the tiering is less about the data that is being produced or managed by the pipeline, but about the pipeline code itself.
00:13:36.720 --> 00:13:41.679
And so how kind of ready is it to produce something that is trusted in production?
00:13:42.080 --> 00:13:42.559
Absolutely.
00:13:42.720 --> 00:13:56.559
So it's uh it's it's about um um it's about uh the the infrastructure behind the pipelines, you know, like the the level of of controls being um uh you know implemented on top of the pipeline itself.
00:13:56.639 --> 00:14:00.879
Um and also do we have metadata about uh the owner?
00:14:01.039 --> 00:14:08.720
Do we uh let's say do we um do we have SLAs about um a specific pipeline?
00:14:08.879 --> 00:14:12.159
Did we agree them with uh with a specific team?
00:14:12.240 --> 00:14:16.799
Uh has it been running for a long time or is it just a new effort, right?
00:14:17.039 --> 00:14:30.720
And uh you can have two pipelines um in two different teams like that we that we own centrally uh that uh you know meet the same very high standards and tier, uh but they are serving two different purposes, right?
00:14:30.879 --> 00:14:40.720
So um some of them have uh certain SLAs because uh we reached uh specific agreements with stakeholders, and others might have like uh different SLAs.
00:14:40.879 --> 00:14:46.480
So we uh the the real question that we answer is uh uh what's the use case?
00:14:46.639 --> 00:14:49.519
What does the customer require?
00:14:49.679 --> 00:14:57.600
Um, you know, in principle, and uh how quickly do we need to uh let's say fix the pipeline in case it you know it breaks.
00:14:57.679 --> 00:14:59.200
So this is a real question.
00:14:59.279 --> 00:15:06.080
Um and this imposes different SLAs, uh, but the tiering system uh is uh standardized.
00:15:06.159 --> 00:15:24.879
And as I was saying, um our guidelines on uh um how to uh let's say promote pipelines uh to higher system uh tiering uh are really transparent and we encourage analysts downstream to to also uh look at these improvements uh constantly.
00:15:25.360 --> 00:15:27.039
Yeah, that's a cool methodology.
00:15:27.120 --> 00:15:43.919
And I'm curious, do you have a sense of how much time typically it takes for if say somebody is doing like a totally new pipeline for a new use case for them to move up into kind of the highest tier of reliability?
00:15:44.080 --> 00:15:48.000
Or are there some pipelines that never get to that highest tier because they don't need to be?
00:15:48.159 --> 00:15:49.360
Or what does that look like?
00:15:49.759 --> 00:15:55.679
It's uh it's pretty much uh uh let's say that is uh like a sort of it comes both ways.
00:15:55.840 --> 00:16:04.480
There are some pipelines that uh will never you know reach that uh top tier uh level because uh it's not needed, right?
00:16:04.559 --> 00:16:07.919
So it's not needed, uh the effort is not needed needed downstream.
00:16:08.080 --> 00:16:18.639
The um let's say the the perhaps the the customer, the stakeholders is not requiring requiring that level of uh complexity, that level of uh efficient efficiency and controls.
00:16:18.879 --> 00:16:27.039
There are other pipelines that naturally are pushed uh upper and upper because there's uh probably a requirement coming from from below.
00:16:27.120 --> 00:16:33.679
And usually requirements coming from below uh tend to speed up the process of uh pushing the tier up.
00:16:33.759 --> 00:16:42.720
Uh, but we also centrally we tend to encourage analysts to uh you know move and promote uh pipelines to higher tiers when possible.
00:16:43.039 --> 00:16:44.639
Yeah, very interesting.
00:16:44.879 --> 00:16:46.320
Yeah, I love learning about that.
00:16:46.399 --> 00:16:55.600
That is a pattern that I don't think I've heard about before, but it totally makes sense for a way of standardizing internally, especially around that sensitive data.
00:16:55.759 --> 00:16:57.200
So, yeah, that's great to hear.
00:16:57.519 --> 00:17:21.119
And then and then as we uh Kent, and also to add to that, as we build a semantic layer, of course, uh having pipelines that are already uh a top top-tier pipelines gives us um sort of uh um foundational sort of information on what should be queried, like where um good quality data comes from, where we should retrieve information uh to start with, right?
00:17:21.200 --> 00:17:25.519
So it gives us a good foundational layer on top of which we can keep building, right?
00:17:25.680 --> 00:17:33.839
Or uh on top of which we can keep uh building downstream DBT models as well as uh you know LLM applications, for example, right?
00:17:35.359 --> 00:17:43.599
This episode is brought to you by Astronomer, the team behind Astro, a managed airflow platform built for data engineering teams.
00:17:44.000 --> 00:17:47.920
From beta pipelines to training models and wiring up agents.
00:17:48.160 --> 00:17:54.160
Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
00:17:54.400 --> 00:17:56.640
Check it out at astronomer.io.
00:17:57.839 --> 00:18:09.359
Changing topics a little bit, I also want to talk about the LLM-enabled workflows that you've been working on since this is, of course, a very hot topic right now and something I think a lot of teams are grappling with.
00:18:09.519 --> 00:18:12.799
This is a big you know emerging area for orchestration.
00:18:13.119 --> 00:18:22.000
So at a high level, again, thinking about the principles that you're using again within the context of working with some very sensitive data.
00:18:22.480 --> 00:18:31.200
What do you think that teams should keep in mind when they're experimenting with you know adding LLMs into their workflows or orchestrating agents or something like that?
00:18:31.599 --> 00:18:38.640
Yeah, that's a great question because um, you know, at first we have been experimenting with uh LLM-enabled workflows indeed.
00:18:39.119 --> 00:18:55.119
And um, of course, uh the uh you know, like uh on the surface, uh an LLM um let's say uh workflow orchestrated by airflow might seem very might look very similar to a standard pipeline, but actually it's not, right?
00:18:55.279 --> 00:19:04.799
Because um uh what what changes is the the you know um the sort of uh uh let's say control being applied on top of the pipeline itself.
00:19:04.960 --> 00:19:53.599
So if a standard pipeline is using standard controls right on uh on code or uh tests or unit tests or uh integration tests, uh when we um let's say talk about uh LLM-enabled uh workflows, we look more at uh validation layers, so building validation layers for specific use cases that are basically uh you know sense-checking what uh the LLM has generated as an output, as well as uh having also um observability platform as well as an evaluation platform that instead of looking at a specific uh use case like the validation layer does, looks uh looks at the performance of the model across a number of uh executions, right?
00:19:53.680 --> 00:19:55.920
So it's a more general layer.
00:19:56.160 --> 00:20:07.680
Um and um, for example, what um uh in an example of how to use um LLMs um via Airflow could be like the generation of technical commentary, right?
00:20:07.759 --> 00:20:19.039
So there's the this technical commentary that needs to be uh uh generated by analysts on a recurrent basis, uh, for example, monthly to perform variance analysis.
00:20:19.200 --> 00:20:28.720
And uh it's sometimes very tedious and time consuming for analysts adding to repeat the same process, you know, um uh let's say month on month, for example.
00:20:28.960 --> 00:20:48.079
So uh what what LLM um uh programmatic LLM workflow uh allows us to do is to actually collect data and then expose this data to um uh a large language model, and the large language model will then uh uh following some instructions generate commentary, for example, right?
00:20:48.319 --> 00:20:56.240
Um but uh but again um uh the the problem uh with uh LLM output is that it's not idempotent, right?
00:20:56.319 --> 00:21:09.279
Uh like like with standard Python pipelines and uh is pluripotent uh on the other side and can hallucinate, even though we see hallucination less and less recently with more advanced models.
00:21:09.359 --> 00:21:19.359
But still, uh being it pluripotent means that we need to be much more uh strict in terms of uh applying validation layers and observability and value and evaluation.
00:21:20.079 --> 00:21:20.960
Yeah, totally.
00:21:21.119 --> 00:21:28.400
I think that non-itempotent or non-deterministic aspect of working with these pipelines is maybe one of the hardest pieces.
00:21:28.559 --> 00:21:33.119
As I've seen, you know, lots of companies try to implement these workflows with Airflow.
00:21:33.200 --> 00:21:40.559
It always makes me think of the number of times that I've told people that a best practice is designing that your Airflow DAGs so that they are itempotent.
00:21:40.640 --> 00:21:42.640
And as you said, in this case, you really can't.
00:21:42.880 --> 00:21:44.160
And that's a big challenge.
00:21:44.319 --> 00:21:58.319
I'm curious if you think that you know, when taking a lot of care to build in some guardrails around that, as you have, do you think that solves the issue or at least mostly solves the issue?
00:21:58.480 --> 00:21:59.759
Like, can you get around that?
00:22:00.240 --> 00:24:01.279
non-deterministic aspect of it with enough sort of proper pipeline design and monitoring and observing yeah yeah actually I think we we we can uh in practice uh by applying this uh multi-layer validation as well as um let's say because at the end of the day what we are doing is passing some uh data that is our source of truth to let's say an LLM right so uh we need to verify whether the LLM is coming back with uh something sensible and something that can be uh you know um that can be verified by actually querying the source of truth data and that can be implemented programmatically so actually um all um uh despite the commentary wording can be uh slightly different from execution to execution uh the the directionality of like the variance analysis for example and the data like the actual figures and percentages being used to build a commentary need to match perfectly and that can be validated and um yeah so uh more recently um we also started thinking about having agents validating agents so having like uh a second agent validating the output of uh uh in an initial LLM call um and also implementing a way to uh let's say um you know how uh agents are also used for uh CICD checks to validate whether you know um uh your your your PR the code that you are that you included in in the last commit is valid or uh includes bugs right having also a CI CD check where uh an agent is checking the suggestion recommended by the initial agent is something that we are uh you know uh really pushing to implement internally so my my answer in the in not so short is uh yes with uh with you care everything can be actually replicated programmatically and uh tested.
00:24:02.160 --> 00:24:04.160
I think that's a great perspective.
00:24:04.480 --> 00:26:03.920
And as maybe a follow-up to that discussion on the reliability of these pipelines other than the item potency aspect is there anything that you have seen related to orchestrating AI-enabled workflows that differs from kind of more traditional airflow use cases like batch ETL uh in terms of the repeatability and monitoring aspect um yeah I mean the um I would say mainly um the way um uh the uh evaluation and uh validation is performed uh as we said uh but in terms of um let's let's say building the uh pipeline itself um uh the I would say the surface of um uh let's say how uh how um how advanced your agentic application can be can differ quite a lot from use case uh to use case for example there there might be um uh an AI enabled workflow that is just running a single call to an LLM and in that case a very well uh written prompt with a good validation layer on top can do the job and um let's say uh get um uh outputs close to hidden potency whereas there are some other cases that can still be orchestrated via airflow where uh fully fledged um uh agents are used with uh with a long graph and then the agent is um um let's say left on uh sort of in on it on its own uh and can use and leverage different tools and in that case um let's say there's much more room for error so I would say uh let's start small when trying uh for the first time like to to to work with this AI enabled workflow and then let's build from there uh more complex agents.
00:26:04.640 --> 00:26:06.559
Yeah I think that makes a lot of sense.
00:26:06.720 --> 00:26:22.160
Well thank you for going through that this has been really insightful and I'd love to hear about your experience in these areas I think that's a good place to transition to my standard standard final question for all of my guests is what would you most like to see from the Airflow project in the near future?
00:26:22.480 --> 00:27:41.200
Yeah absolutely um I was actually uh really keen to see um and to uh let's say all the advancements recently after the 2025 summit and uh I'm really keen to uh you know to see uh Airflow becoming more and more a platform also for uh uh you know uh agentic workflows and having some sort of native code to uh let's say implement uh uh validation and also to to remove to remove boilerplate code so to standardize a little bit more the way uh agents um uh are uh orchestrated and also like I would say I would say a bottleneck for us internally has always been how we test uh airflow uh you know locally uh and you know um with a lot of iteration we managed to get to let's say a very strong and solid uh local environment but this is not always easy if you're not an engineer so if you know if you have no um if you have not like an advanced knowledge about uh the components uh um behind airflow so making local uh development easier and closer to to production is always something that uh I think uh there's uh quite a lot of work that can be done uh there um and uh I would I would like to see how it goes in the future.
00:27:41.680 --> 00:27:45.279
Definitely I think those are both great answers and very timely.
00:27:45.519 --> 00:27:55.279
Well Antonella this has been a really great conversation really appreciate you sharing uh all of your experience just to close this out what is the best way for folks to get in touch with you?
00:27:55.599 --> 00:28:51.920
Yeah you can reach me on LinkedIn I'm very active there and very keen to respond to all your questions either about this uh uh podcast or any other uh let's say uh topic related to data and analytics engineering um and also um uh make sure if you if you like what you know what we discussed uh feel free to share our uh uh to to to check our career page uh um on the you know wise website and there are a lot of roles open in engineering and analytics engineering and again Kenton thanks for uh having me and thanks for the invite yeah thank you so much for coming on we will throw links to those things in the show notes so people can access them uh yeah thank you again this has been great thank you for listening to the data flowcast check the show notes for links to everything mentioned today if you're enjoying the podcast subscribe so you never miss an episode and leave us a five star review
