WEBVTT
00:00:00.320 --> 00:00:02.240
You are listening to the data of podcasts.
00:00:07.360 --> 00:00:09.599
We are excited to announce the law of data.
00:00:17.359 --> 00:00:19.839
We are offering Survey Respondents something special.
00:00:20.399 --> 00:00:25.920
A free code to take the new Astronomer Certified AI orchestrate fundamental text.
00:00:26.480 --> 00:00:28.239
$150 value.
00:00:28.399 --> 00:00:35.600
Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
00:00:35.759 --> 00:00:38.000
Now let's get into the show.
00:00:38.479 --> 00:00:38.719
Okay.
00:00:38.880 --> 00:00:39.679
Hi everyone.
00:00:39.920 --> 00:00:41.600
Welcome to the Data Flowcast.
00:00:41.679 --> 00:00:48.399
I'm your host, Kenton Danis, and today I'm joined by Anaruda Sengupta, who is a staff data engineer at Insignus.
00:00:48.719 --> 00:00:50.079
Anaruda, how are you?
00:00:50.560 --> 00:00:51.200
Yeah, I'm good.
00:00:51.359 --> 00:00:51.759
Doing good.
00:00:51.840 --> 00:00:52.079
Thanks.
00:00:52.159 --> 00:00:52.560
How are you?
00:00:52.880 --> 00:00:53.359
I'm good.
00:00:53.520 --> 00:00:54.799
It's great to have you on.
00:00:54.880 --> 00:00:56.719
I'm excited to chat today.
00:00:57.600 --> 00:01:01.200
For our listeners, bit of a preview for today's podcast.
00:01:01.280 --> 00:01:16.560
We're going to discuss how Insignus runs its data platform on Airflow and some advice from the team on controlling costs and implementing best practices, which I have to say, I think are two of maybe the most popular topics ever within Airflow and orchestration.
00:01:16.719 --> 00:01:20.319
So I think, yeah, our listeners will get a lot out of this.
00:01:20.640 --> 00:01:23.760
But as always, I want to start with a little bit of background.
00:01:23.920 --> 00:01:27.200
So tell me first about the company you work for.
00:01:27.280 --> 00:01:28.799
What does Insignus do?
00:01:29.599 --> 00:01:30.319
Yeah, sure.
00:01:30.400 --> 00:01:34.400
So Insignus is a FinTech based in the UK.
00:01:34.480 --> 00:01:37.040
We've got offices in London and Cambridge.
00:01:37.840 --> 00:01:43.280
We basically operate in the cash savings industry in the UK.
00:01:43.439 --> 00:01:59.439
So we manage cash savings across businesses and people and basically make it as easy as possible for people to manage and diversify their cash holdings to get it to yield the highest possible returns.
00:01:59.680 --> 00:02:14.159
And we do that by um, you know, we've built a client platform where clients basically instruct us to move their money into differing savings accounts that yield the highest interest rates.
00:02:14.319 --> 00:02:22.639
So basically, we don't want um for charities, for schools, for example, we don't want their cash reserves sitting idle.
00:02:22.879 --> 00:02:33.120
We also want it to do the hard work so that you know it it works hard alongside them in a it's it's not just a case of it sits there and does nothing.
00:02:33.759 --> 00:02:35.360
Okay, yeah, that's great.
00:02:35.599 --> 00:02:44.479
And I would imagine for a company like that, I mean FinSurf always has a lot of data to manage, I think, lots of sensitive data that you're dealing with.
00:02:45.520 --> 00:02:46.080
Yeah, for sure.
00:02:46.319 --> 00:02:52.560
Like, yeah, like um there's lots of transactional data, um, which you can imagine flying around.
00:02:52.719 --> 00:03:12.479
Um, so yeah, it's a it's a case of um implementing uh best practices in a way so that you know we don't get bankrupted, but also keeping that uh kind of PII front and center, um, and just being aware uh of what is PII and what is not.
00:03:13.199 --> 00:03:14.319
Yeah, absolutely.
00:03:14.479 --> 00:03:21.199
And so tell me a little bit more about then how your data team is structured and kind of what you're responsible for.
00:03:21.599 --> 00:03:29.919
Yeah, so we basically um our the the core um platform is built in C sharp.
00:03:30.080 --> 00:03:38.400
And so we've got three kind of operational teams that look after that, which is you know look divided into three distinct uh areas.
00:03:38.960 --> 00:03:58.000
The data team basically sits across the entire business um to kind of consume data from everywhere uh in the Insignus engine, as I like to call it, and try and deliver data-driven insights to the rest of the business and to the C-suite.
00:03:58.479 --> 00:04:09.199
Um and uh we not only sit across the operational team, so we collect internal data, but we also collect a lot of external data as well.
00:04:09.520 --> 00:04:22.720
So our um customer services team, for example, their calls such as they have with their with clients, get recorded in Ringover, so we export data from Ringover.
00:04:22.959 --> 00:04:50.000
We've got our um CRM set up in Salesforce, so we um import data from Salesforce, and you know, so we try and build a kind of single plane of glass so that anyone in the company can just come to us with uh uh an insight they either want to know or or they want to look at data, where that single place where you can look at all the data of what's happening in the company.
00:04:50.240 --> 00:04:54.800
I like to think of us as a basically a newspaper.
00:04:54.959 --> 00:05:02.160
You you you kind of get an idea of what's happening in the business, but and we'll we'll get I'm sure we'll get into that later.
00:05:02.480 --> 00:05:11.279
What is important for us is we don't have um core business kind of logic sitting within our data platform.
00:05:11.439 --> 00:05:15.839
We are just simply there to be the newspaper for the rest of the business.
00:05:16.240 --> 00:05:17.600
Okay, interesting.
00:05:17.920 --> 00:05:27.759
I love the analogy of the newspaper, and so you're just providing all of the information, then what the other teams do with it is sort of up to them based on their use cases.
00:05:28.000 --> 00:05:34.160
Yeah, it's almost like yeah, it like and sometimes I I think it's served us very well so far.
00:05:34.240 --> 00:05:38.959
And sometimes a bit people have been like, uh, you know, why why is this broken?
00:05:39.040 --> 00:05:42.079
It's like, don't know, like you have to ask the operational teams.
00:05:42.160 --> 00:05:44.800
I'm just uh I'm almost like the messenger here.
00:05:44.879 --> 00:05:46.480
I just report on what I see.
00:05:46.800 --> 00:05:48.879
Yeah, yeah, that makes sense.
00:05:49.040 --> 00:05:56.639
And I think you have told me previously, if I'm not mistaken, that you are the only data engineer at the company.
00:05:56.800 --> 00:06:00.800
So your function, at least within that team, is of one person.
00:06:01.680 --> 00:06:02.319
Yes, correct.
00:06:02.399 --> 00:06:06.079
We've uh we've got um um another person.
00:06:06.240 --> 00:06:12.639
He's he's more of a data analyst like his job title says data engineer, but we also need a data analyst within the team.
00:06:12.720 --> 00:06:18.639
So he looks after the data analysis side, and yeah, I looked after mostly the data engineering side.
00:06:18.959 --> 00:06:19.279
Okay.
00:06:19.920 --> 00:06:24.879
And that means you probably got a lot of responsibility that's just on you.
00:06:25.040 --> 00:06:32.560
I'm curious how does that shape you know the way that you work and yeah, like in in the beginning it was tough for sure.
00:06:32.800 --> 00:06:41.040
Um I came from JLR from Jaguar Land Rover, and that had a very big data engineering team.
00:06:41.360 --> 00:06:50.959
So in the beginning, really, I didn't if I had ideas that I wanted to discuss, I couldn't really discuss them with anybody else.
00:06:51.120 --> 00:07:02.399
Um, so for me personally, AI has been really helpful with that, just as a like someone to have a back and forth, like a like to bounce ideas off.
00:07:02.560 --> 00:07:17.360
I you know, I've I've definitely done it with my software engineering colleagues, um, and they have been helpful to an extent, but to like it's different when you're talking to like a data engineering colleague because they they might not have all the context.
00:07:17.519 --> 00:07:22.720
Yeah, a software engineer might not have all the context a data engineering does, a data engineer does.
00:07:23.040 --> 00:07:35.839
Um and but but what's also important for us is the way we shape our work is we still want the data engineering pipelines to be treated as first-class citizens.
00:07:36.000 --> 00:07:42.959
So it has the same rigorous set of testing, see set of CICDs, extensive documentation.
00:07:43.040 --> 00:07:49.199
You know, it's not I certainly don't allow it to be treated like an afterthought.
00:07:49.519 --> 00:07:58.399
Um, you know, it it it deserves the same kind of um rigorousness as the other software engineering teams as well.
00:07:58.720 --> 00:07:59.920
Yeah, absolutely.
00:08:00.160 --> 00:08:06.560
Well, I'm sure that constraint will be relevant to some of our upcoming discussion about airflow as well.
00:08:06.720 --> 00:08:16.160
I have also been myself not in my current role, but in pastorals in this position of being the only analyst at a company or something like that.
00:08:16.240 --> 00:08:19.439
And yeah, it's a hard, it's a hard spot to be in sometimes, definitely.
00:08:19.680 --> 00:08:20.879
Yeah, definitely.
00:08:21.120 --> 00:08:25.120
Um yeah, so that's great that you've managed to make that work.
00:08:25.279 --> 00:08:29.680
But again, I I do want to talk about how you know Airflow fits into this.
00:08:29.839 --> 00:08:35.200
So tell me about how your data platform is structured and what role Airflow plays.
00:08:35.759 --> 00:08:41.519
Yeah, so Airflow is basically at the very heart of our data, of our data infrastructure.
00:08:41.759 --> 00:08:45.360
Um, it's our main tool for orchestrating data pipelines.
00:08:45.919 --> 00:08:58.559
So we not only use it to orchestrate when we, for example, get uh third-party data from using APIs, but we also use it as an observation layer.
00:08:58.799 --> 00:09:17.919
So we've got um a couple of real-time pipelines that we have, which is not built with Airflow, but Airflow kind of sits on top of those to build an observation layer to ensure that you know those real-time pipelines are definitely working uh as intended.
00:09:18.159 --> 00:09:36.240
And that kind of observe like observability layers we found that it's very useful with what capability Airflow provides for us, uh, you know, as well as being you know our standard tool for kind of like you know, the standard ETL job.
00:09:36.480 --> 00:09:37.840
Um, yeah.
00:09:38.080 --> 00:09:53.279
And yeah, and we also use a lot of and again I'll get into that, but yeah, like for us, and I'm sure this is not a uh not a revolutionary thing, I'm gonna say, is Airflow for us is the the orchestrator for everything.
00:09:53.679 --> 00:10:00.720
It it tells other things to do things and then returns for a result back, but doesn't actually do the things itself.
00:10:01.200 --> 00:10:11.600
Um so it yeah, so it it like it it controls everything, it's like the brain, but it doesn't actually perform any of the heavy lifting as much as possible.
00:10:12.080 --> 00:10:22.000
Other tools perform, you know, for example, SQL pipelines, they perform the head like in BigQuery, it performs the heavy lifting, and Airflow just orchestrates that.
00:10:22.480 --> 00:10:23.440
Yeah, absolutely.
00:10:23.600 --> 00:10:29.679
I think that is a really common pattern, and I do want to get into that a little more before we do, just high level.
00:10:29.840 --> 00:10:33.759
Tell me a little bit about what kinds of data pipelines you're running with Airflow.
00:10:34.320 --> 00:11:05.279
Yeah, so we've got um we've got a few operational ones, we've got a few where we use uh where we try and get data from external systems, and we try and do that as um like even if they're comp like even if we are using APIs, we try and use methods that are kind of best practices, not just to data engineering, but to software engineering, things like using uh watermarks when retrieving API results.
00:11:05.519 --> 00:11:13.279
We try in terms of kind of cost saving, we try and stay two or three steps ahead.
00:11:13.519 --> 00:11:19.919
You know, you get data from an API, you you get all the data there, it's fine, it works, you know.
00:11:20.159 --> 00:11:31.759
But we've because we don't have that lock, that luxury of having a huge budget behind us, we kind of have to think two or three steps ahead, like right, it works, but now how can I make it as efficiently?
00:11:31.919 --> 00:11:34.879
How can I get the data as efficiently as possible?
00:11:35.039 --> 00:11:46.000
And on top of that as well, we've got some we've got some operational pipelines of our own, which the businesses rely on um to use every day.
00:11:46.159 --> 00:12:09.519
So we also make sure that in our the data pipelines that we do have, the relevant observe observability layers, the relevant testing, making sure that you know things like if it fails or if it misses an SLA, it's all kind of there as in best practice to catch it so that incidents can be resolved as painlessly as possible.
00:12:09.840 --> 00:12:10.159
Okay.
00:12:10.399 --> 00:12:11.679
Yeah, got it.
00:12:12.000 --> 00:12:22.799
And then you've alluded to this a little bit, but I know you've also spoken about building, you know, a low-cost, scalable platform for airflow.
00:12:22.879 --> 00:12:28.320
And we've talked already about how you're working within the constraint of you know a very small team.
00:12:28.559 --> 00:12:28.879
Yeah.
00:12:29.120 --> 00:12:35.279
Tell me about how you have gone about implementing this, maybe anything else about kind of the importance of doing it.
00:12:35.440 --> 00:12:47.360
And I guess, you know, what advice you would give to other small teams who are trying to use Airflow, but you know, need to be very conscious of cost and things like that, regardless of where or how they're running it.
00:12:47.840 --> 00:13:03.759
Yeah, I think I think if you're in a similar position, one of the things I'd say first is um observability building in observability layers is should be a part of your definition of done and not just an afterthought.
00:13:03.919 --> 00:13:14.559
And I don't and I don't necessarily mean observability in the sense that um I went to my Airflow pipeline, I went to the Airflow UI, it's all green.
00:13:14.720 --> 00:13:16.080
That's not observability.
00:13:16.320 --> 00:13:25.440
Observability is in my opinion, taking a step back and kind of seeing like is the output why expect?
00:13:25.759 --> 00:13:32.559
Like am I like the pipeline runs fine, but are things being silently dropped?
00:13:32.639 --> 00:13:49.440
And being very kind of curious about that, so building in observability layers where uh a great thing that's worked for us is uh dashboards built in in um in our stack that is kind of independent of the main data pipelines themselves.
00:13:49.759 --> 00:14:05.600
So it kind of does his thing, the main data pipelines do their thing, so uh we kind of know that yes, the pipeline's passed, and yes, the data that is being fetched, that whatever it's trying to do is working as intended.
00:14:05.840 --> 00:14:12.720
So that that will definitely help to keep costs uh um uh running as efficiently as possible.
00:14:12.879 --> 00:14:19.120
And the other thing as well is when you're a small team, um automation is very important.
00:14:19.279 --> 00:14:29.840
Uh so to automate and this might sound like an obvious thing, but you'd be surprised how many times I've I mean, when when I joined in Insignus, for example, uh, you know, this was happening.
00:14:30.159 --> 00:14:32.399
Automate things that take the most time.
00:14:32.799 --> 00:14:39.600
So uh it can be used intelligently as a substitution for headcount if you know what you're doing.
00:14:39.759 --> 00:14:53.039
You don't uh the the um one of the aims of the big game is to uh uh try and scale the company uh but not necessarily scale it with headcount, right?
00:14:53.120 --> 00:14:54.000
If that makes sense.
00:14:54.240 --> 00:15:01.519
So be uh mindful and be intelligent of what you can automate, what takes you the most time, and go ahead and do it.
00:15:01.600 --> 00:15:05.360
That that that will definitely help you with with kind of cost savings.
00:15:05.759 --> 00:15:27.840
Um and then I I will also say the other thing is in our data platform we used a managed airflow version, we don't host things ourselves ourselves because whilst we do have a um platform team, it's not solely dedicated to the data team.
00:15:28.480 --> 00:15:37.919
Um if you use the managed airflow version, uh like the things like the scheduler, the workers, the database is all is all kind of abstracted away from you.
00:15:38.159 --> 00:15:50.480
So that leaves you free to write your code and write your documentation so that you know you don't have to spend your time putting on your platform hat and looking after those as well.
00:15:50.559 --> 00:15:57.039
So that that will also help you to kind of manage your costs if you are a small team and you want to get started in Airflow.
00:15:59.519 --> 00:16:08.000
This episode is brought to you by Astronomer, the team behind Astro, a managed Airflow platform built for data engineering teams.
00:16:08.240 --> 00:16:12.159
From batch pipelines to training models and wiring up agents.
00:16:12.399 --> 00:16:18.399
Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
00:16:18.639 --> 00:16:21.039
Check it out at astronomer.io.
00:16:23.200 --> 00:16:24.320
Yeah, absolutely.
00:16:24.480 --> 00:16:27.919
I there's so much good stuff there that I want to double-click on.
00:16:28.080 --> 00:16:35.840
I do think the managed service, of course, we're uh very heavily biased at Astronomer, but we do think that this helps people quite a lot.
00:16:35.919 --> 00:16:39.840
Um, again, as you said, so that you don't have to worry about the airflow infrastructure.
00:16:40.000 --> 00:16:55.600
And I think that um relevant for automation as well, because I think it allows you to implement, you know, things like auto-scaling and stuff like that, where you can um just have a lot more intelligence built into the platform as opposed to trying to manually tweak those things.
00:16:55.919 --> 00:17:08.720
Uh I'm curious actually, going back to the first thing you said on observability and how it's not just, you know, looking and seeing if you had any tasks fail, but also making sure that your output looks like what it should and all of that.
00:17:09.039 --> 00:17:21.519
I, you know, oftentimes I think talk to people about that in the context of implementing things like data quality checks so that they're catching, you know, things where the task succeeded, but like something is wrong with the data.
00:17:21.759 --> 00:17:24.240
I don't know if that's exactly what you're talking about.
00:17:24.400 --> 00:17:31.359
Um I mean, I can give I can give a specific example without um without giving away company secrets.
00:17:31.440 --> 00:17:34.240
So yeah, data quality checks are definitely important.
00:17:34.880 --> 00:17:48.000
We we definitely we definitely use um a lot of them in our pipelines to make sure that the the data kind of before and and after you know it all looks and is all aligned.
00:17:48.160 --> 00:18:02.559
But what I mean, you know, a great example of this is um our customer services team chat with clients every day and we record calls and transcripts, right?
00:18:03.200 --> 00:18:13.759
Now we've got uh a data pipeline that gets calls and transcripts and every day it shows yeah, it's green, it's getting them.
00:18:14.079 --> 00:18:17.440
But how do you know that they're getting all of them?
00:18:17.599 --> 00:18:20.160
How do you know that they're not silently dropping anything?
00:18:20.480 --> 00:18:25.519
How do you know that um uh you know things are not being muddled along the way?
00:18:25.680 --> 00:18:36.319
So that's where building a separate observer, which I I I appreciate it can um it can be case specific, it can be project specific.
00:18:37.039 --> 00:18:42.960
I mean, the example I'm giving is specific to this particular pipeline, it might not apply to others.
00:18:43.119 --> 00:18:58.319
So, but we've built observable observability layers where we look at the output of what we've actually collected and then do like a kind of summary, like right, okay, um airflow pipeline is green.
00:18:58.480 --> 00:19:07.440
Ah, but like look, we've got all the calls that Ringover said we've had, we've got all the transcripts, so there doesn't seem to be any errors, nothing is being dropped silently.
00:19:07.599 --> 00:19:17.119
So that's why you know that observability layer in addition to your data quality checks for us is what has has been quite important.
00:19:17.680 --> 00:19:18.799
Yeah, makes sense.
00:19:18.960 --> 00:19:31.039
And tell me about how so, of course, having that data quality is really important for being able to make accurate business decisions, but you're also making the argument for cost optimization.
00:19:31.200 --> 00:19:34.880
So, how does having that observability layer help with cost?
00:19:35.839 --> 00:19:47.920
So it helps with costs because I think um you could argue like, well, you're building more things, so in the short term is costing you more, right?
00:19:48.400 --> 00:19:59.680
But by having that observability layer, we're not we're actually we the thing that's gonna the thing that it will do is prevent incidents from happening.
00:20:00.000 --> 00:20:05.440
So people downstream are not gonna scream like, well, I I can't see any of the ring over data, where is it?
00:20:05.680 --> 00:20:13.920
That in itself will incur a much bigger cost than the kind of short-term costs you see of building those observability layers in the first place.
00:20:14.240 --> 00:20:23.920
So the cost of having that incident downstream is gonna be much bigger than the cost, then the initial cost of building the observability layer from the start.
00:20:24.240 --> 00:20:24.880
Got it.
00:20:25.039 --> 00:20:26.559
Yeah, okay, I see.
00:20:27.119 --> 00:20:44.079
And I guess related to that potentially, are there any patterns within your DAGs themselves in terms of implementing best practices that you think have made a big difference for you in terms of cost or scalability?
00:20:44.480 --> 00:20:46.559
Maybe other than data quality checks?
00:20:46.880 --> 00:20:55.599
Yeah, other than data quality and other than using, which I've already which we've spoken about, using Airflow as an orchestration engine, not an execution one.
00:20:56.319 --> 00:21:04.880
Um I would say automating your CI in a deployment, don't be kind of tempted to leave those as an afterthought.
00:21:05.039 --> 00:21:10.160
Um, and let your CI pipeline handle as much of the automated tests as possible.
00:21:10.480 --> 00:21:16.799
Let machines pick up things so you don't have to manually kind of do it yourself.
00:21:16.960 --> 00:21:56.799
Things like linting, unit testing, um documentation, and make sure that all your DAGs are kind of consistent, they all have a level playing field, and a great way to do that is enforcing it through your CI CD pipeline, um, and making sure that you know, like your DAGs have all the kind of best practices set out by Airflow and Astronomer, things like failure callbacks, um, being mindful of where you how you put the documentation, being mindful of where you're when your DAG starts and when it ends, and making sure it doesn't, you know, it doesn't go rogue and just do loads of which I have seen happen um before.
00:21:57.039 --> 00:22:10.640
So um And the other thing as well for us is we purge our logs and our metadata because we find that the metadata table can build up very quickly.
00:22:12.000 --> 00:22:18.319
So to kind of make sure that you know things are operating at that sweet spot, we purge them periodically.
00:22:18.480 --> 00:22:24.240
Um so that you know we don't hold on to logs you know for you know that like a year old.
00:22:24.319 --> 00:22:38.000
You know, we don't we don't need to see um you know that that it's just that you know logs that are recent that that may produce a failure, those are the ones that we want to see so that we can begin to debug if we need to.
00:22:38.559 --> 00:22:42.079
Yeah, that's a recommendation we make very commonly to our customers as well.
00:22:42.160 --> 00:22:44.079
It can have big performance impacts.
00:22:44.799 --> 00:22:46.160
Yeah, definitely.
00:22:46.640 --> 00:22:55.839
Okay, and then I understand that you have also built some internal tools kind of around your Airflow platform to help keep it running efficiently.
00:22:56.079 --> 00:22:58.640
Tell me about those and how they help you.
00:22:59.119 --> 00:23:10.240
Yeah, so the first thing is um uh we built a tool to show our SQL pipeline costs and we call that tariff.
00:23:10.720 --> 00:23:17.279
Um so basically, uh Airflow is responsible for orchestrating our SQL pipelines as well.
00:23:17.519 --> 00:23:29.039
And the the reason why we built this tool is one, we you know, we just want to see in a plain and clear way as possible how much our SQL pipelines are costing.
00:23:29.279 --> 00:23:38.400
We don't want those to be creeping up and then you know um you know leaking into our airflow costs and our overall data infrastructure costs as well.
00:23:38.559 --> 00:23:43.519
That's something that we've that you know it has a front end and a back end.
00:23:43.680 --> 00:24:00.960
Um, and we find that it's definitely given us visibility and and some interesting insights, things like um, if you have kind of you know incremental mechanisms, what you might think is like, oh yeah, it's it's working really great, it's actually costing you a lot.
00:24:01.119 --> 00:24:08.400
So it's actually helped us with the optimizations um a lot in our data infrastructure to kind of keep costs down.
00:24:08.640 --> 00:24:23.039
The second thing is um we built an AI-powered tool to give recommendations on how to improve the Airflow platform in terms of like the workers, their memory, the scheduler, how big the database should be.
00:24:23.119 --> 00:24:24.240
We call that Maestro.
00:24:24.799 --> 00:24:30.640
Um previously this kind of took a long time, and it was it was for me to do.
00:24:30.880 --> 00:24:43.519
Um, you know, it would you would kind of have to analyze your DAGs, analyze how they did over a period of time, analyze how your workers in your Airflow environment are doing.
00:24:43.599 --> 00:24:46.160
You know, are they running out of memory frequently?
00:24:46.400 --> 00:24:49.680
Are they just about to run out of memory or are they doing okay?
00:24:49.759 --> 00:24:56.480
And then it was very much a finger in the air, like, oh I think the worker provision should be increased by X amount.
00:24:56.799 --> 00:25:03.920
This kind of takes it all away, um, you know, and it takes it takes minutes, you know, to do and give recommendations.
00:25:04.319 --> 00:25:11.519
Um, so no, so tools like that where it's really helped us to make sure that again that goes back to the point of automation.
00:25:12.079 --> 00:25:24.960
You know, I could have easily not easily, but it would have taken me a while to kind of look through a SQL pipeline and go, right, you know, what are what is uh costing the most here and what is costing the least and why.
00:25:25.279 --> 00:25:27.519
You know, we've kind of automated all of that.
00:25:27.759 --> 00:25:29.359
Same for Maestra as well.
00:25:30.079 --> 00:25:48.880
So it's tools like that that kind of helped us to uh you know build our data infrastructure, you know, well from the start, uh, and to make sure that it's it's not just about, like I said before, it's not just about short-term let's just get costs down, let's just spend a pound a day.
00:25:49.440 --> 00:25:54.960
The the the point I want to stress here is you want to find that sweet spot.
00:25:55.680 --> 00:26:11.279
Just you you want to give, for example, you want to give your airflow workers just enough juice so that they need they can they have what they need to do to do the work that they're asked to do, but not so much that you are it's costing you an an absolute arm and leg.
00:26:11.440 --> 00:26:21.440
So you want to find that sweet spot, and so the tools we've built there is helping us to find that sweet spot to help us you know keep our data pipelines running every day.
00:26:21.759 --> 00:26:31.039
Yeah, I think that's a great description of kind of striking the right balance between those short-term adaptations and the long-term sustainability of your platform.
00:26:31.200 --> 00:26:32.880
Yeah, very interesting.
00:26:33.519 --> 00:26:37.839
Well, on a slightly different topic, maybe, maybe it's related.
00:26:37.920 --> 00:26:45.599
Uh, I know you've also mentioned that you just started exploring AI with Airflow, which is of course a really popular topic right now.
00:26:45.759 --> 00:26:49.839
Um, I'm curious what sort of use cases are you looking at implementing there?
00:26:50.480 --> 00:26:58.480
So one of some of the obvious UK use cases is probably some analysis of the movement of money.
00:26:58.720 --> 00:27:04.559
So things like um our clients, how do they kind of invest with us?
00:27:04.720 --> 00:27:07.039
Um, it's more kind of MLOPs, actually.
00:27:07.279 --> 00:27:13.359
Um, you know, we want we want to kind of identify patterns and try and capitalize on them.
00:27:13.519 --> 00:27:22.000
Um with move like you know, and maybe things like in our customer journey, things may not be as optimized as they can be.
00:27:22.079 --> 00:27:26.880
So we what kind of you kind of you want to use AI and MLOPs to help us identify those.
00:27:27.200 --> 00:27:39.440
One of the other things I found really inspiring from the Airflow Summit, which has just gone by, is you know, the the idea of self-healing pipelines, which is a very juicy topic to me.
00:27:39.519 --> 00:27:43.440
So, you know, things like and again that comes back to the whole point, right?
00:27:43.680 --> 00:27:45.839
Automating it away as much as I can.
00:27:46.079 --> 00:28:09.200
Not automating it fully so that an AI just goes wild and on the rampage and changes things, but to help us as much as possible when there are things going wrong, you know, an AI can do the analysis much faster, looking through logs and seeing what's got what's gotten wrong, implementing uh suggesting a fix, an AI can do that much faster than a human can.
00:28:09.359 --> 00:28:12.880
So definitely in our future, a kind of self-healing pipeline.
00:28:12.960 --> 00:28:16.880
And and there was loads of cool talks, people had different ways of doing it.
00:28:17.039 --> 00:28:23.680
There was, you know, one of the talks I think I went to was a DAG does it, a DAG that listens to other DAGs.
00:28:23.839 --> 00:28:25.440
So that was cool as well.
00:28:25.839 --> 00:28:33.279
We've got some ideas of how it can be done at Insignus for us, so we're definitely very interested to leverage that.
00:28:34.240 --> 00:28:35.119
Yeah, absolutely.
00:28:35.279 --> 00:28:36.960
This is something I'm doing myself as well.
00:28:37.039 --> 00:28:41.279
Of course, Astronomer has our own perspective on how we can implement this within our platform.
00:28:41.359 --> 00:28:43.039
As you said, there's many ways of doing it.
00:28:43.200 --> 00:28:45.200
I love that the community are sharing those.
00:28:45.359 --> 00:28:53.200
But yeah, it has certainly helped me a lot in the debugging of the pipelines that you know provide metrics and stuff for my team.
00:28:53.359 --> 00:29:00.240
Um, it's often very quick at identifying any issues and fixing them compared to what it used to take me to debug them myself.
00:29:01.279 --> 00:29:02.400
Yeah, that's great.
00:29:02.559 --> 00:29:11.440
Well, I think uh in the interest of time, that's a good uh lead into my standard final question, which is what would you most like to see from the Airflow project in the near future?
00:29:11.920 --> 00:29:26.559
I think one of the things actually I'd like to see is um in terms of when DAGs get passed, um, so that when they're when when they when they show up in the UI, I'd like to see that a little bit faster.
00:29:26.799 --> 00:29:36.079
It's not a huge bottleneck for us, but it sometimes takes as long as longer than we like to see when we implement changes to then kind of see that in the UI.
00:29:36.400 --> 00:29:44.400
Um and I know it a lot of it comes down to the DAG file processing, if you have things like heavy, heavy top-level imports.
00:29:44.799 --> 00:29:51.920
Um, but I think the repassing at the moment works on an interval rather than an event-based change.
00:29:52.079 --> 00:29:55.519
So I'd love for X Airflow to explore that more.
00:29:55.839 --> 00:30:08.960
Um to you know, to make it you know a little bit more the DAC passing a little bit more event-based, so that you can kind of see changes close to instantaneous rather than kind of having waiting an interval.
00:30:09.279 --> 00:30:10.319
Yeah, absolutely.
00:30:10.480 --> 00:30:11.519
That's a great one.
00:30:11.839 --> 00:30:14.640
Okay, well, Inurita, thank you so much for coming on.
00:30:14.720 --> 00:30:16.400
This has been a really great discussion.
00:30:16.559 --> 00:30:19.599
Like I said, I think it will be very helpful for a lot of our listeners.
00:30:19.759 --> 00:30:23.119
Just to close this out, what is the best way for folks to get in touch with you?
00:30:23.440 --> 00:30:26.079
Uh you can just yeah, find me on LinkedIn, probably.
00:30:26.160 --> 00:30:27.440
I'm quite active there.
00:30:27.599 --> 00:30:32.000
Um, but yeah, feel free to also reach out to me via email.
00:30:32.319 --> 00:30:36.319
Um yeah, I'd say those are the two options.
00:30:36.640 --> 00:30:36.960
Great.
00:30:37.119 --> 00:30:39.599
We'll throw your LinkedIn in the show notes for sure.
00:30:39.759 --> 00:30:41.200
Well, thank you again for coming on.
00:30:41.279 --> 00:30:42.400
It's been great speaking with you.
00:30:42.720 --> 00:30:43.200
No problem.
00:30:43.279 --> 00:30:44.319
Yeah, thanks for having me.
00:30:44.559 --> 00:30:47.440
Thank you for listening to the Data Flowcast.
00:30:47.599 --> 00:30:51.039
Check the show notes for links to everything mentioned today.
00:30:51.200 --> 00:30:55.839
If you're enjoying the podcast, subscribe so you never miss an episode.
00:30:56.000 --> 00:30:58.319
And leave us a five star review.
00:00:00.320 --> 00:00:02.240
You are listening to the data of podcasts.
00:00:07.360 --> 00:00:09.599
We are excited to announce the law of data.
00:00:17.359 --> 00:00:19.839
We are offering Survey Respondents something special.
00:00:20.399 --> 00:00:25.920
A free code to take the new Astronomer Certified AI orchestrate fundamental text.
00:00:26.480 --> 00:00:28.239
$150 value.
00:00:28.399 --> 00:00:35.600
Just go to astronomer.link slash airflow minus survey minus 2026 or click the link in our show notes.
00:00:35.759 --> 00:00:38.000
Now let's get into the show.
00:00:38.479 --> 00:00:38.719
Okay.
00:00:38.880 --> 00:00:39.679
Hi everyone.
00:00:39.920 --> 00:00:41.600
Welcome to the Data Flowcast.
00:00:41.679 --> 00:00:48.399
I'm your host, Kenton Danis, and today I'm joined by Anaruda Sengupta, who is a staff data engineer at Insignus.
00:00:48.719 --> 00:00:50.079
Anaruda, how are you?
00:00:50.560 --> 00:00:51.200
Yeah, I'm good.
00:00:51.359 --> 00:00:51.759
Doing good.
00:00:51.840 --> 00:00:52.079
Thanks.
00:00:52.159 --> 00:00:52.560
How are you?
00:00:52.880 --> 00:00:53.359
I'm good.
00:00:53.520 --> 00:00:54.799
It's great to have you on.
00:00:54.880 --> 00:00:56.719
I'm excited to chat today.
00:00:57.600 --> 00:01:01.200
For our listeners, bit of a preview for today's podcast.
00:01:01.280 --> 00:01:16.560
We're going to discuss how Insignus runs its data platform on Airflow and some advice from the team on controlling costs and implementing best practices, which I have to say, I think are two of maybe the most popular topics ever within Airflow and orchestration.
00:01:16.719 --> 00:01:20.319
So I think, yeah, our listeners will get a lot out of this.
00:01:20.640 --> 00:01:23.760
But as always, I want to start with a little bit of background.
00:01:23.920 --> 00:01:27.200
So tell me first about the company you work for.
00:01:27.280 --> 00:01:28.799
What does Insignus do?
00:01:29.599 --> 00:01:30.319
Yeah, sure.
00:01:30.400 --> 00:01:34.400
So Insignus is a FinTech based in the UK.
00:01:34.480 --> 00:01:37.040
We've got offices in London and Cambridge.
00:01:37.840 --> 00:01:43.280
We basically operate in the cash savings industry in the UK.
00:01:43.439 --> 00:01:59.439
So we manage cash savings across businesses and people and basically make it as easy as possible for people to manage and diversify their cash holdings to get it to yield the highest possible returns.
00:01:59.680 --> 00:02:14.159
And we do that by um, you know, we've built a client platform where clients basically instruct us to move their money into differing savings accounts that yield the highest interest rates.
00:02:14.319 --> 00:02:22.639
So basically, we don't want um for charities, for schools, for example, we don't want their cash reserves sitting idle.
00:02:22.879 --> 00:02:33.120
We also want it to do the hard work so that you know it it works hard alongside them in a it's it's not just a case of it sits there and does nothing.
00:02:33.759 --> 00:02:35.360
Okay, yeah, that's great.
00:02:35.599 --> 00:02:44.479
And I would imagine for a company like that, I mean FinSurf always has a lot of data to manage, I think, lots of sensitive data that you're dealing with.
00:02:45.520 --> 00:02:46.080
Yeah, for sure.
00:02:46.319 --> 00:02:52.560
Like, yeah, like um there's lots of transactional data, um, which you can imagine flying around.
00:02:52.719 --> 00:03:12.479
Um, so yeah, it's a it's a case of um implementing uh best practices in a way so that you know we don't get bankrupted, but also keeping that uh kind of PII front and center, um, and just being aware uh of what is PII and what is not.
00:03:13.199 --> 00:03:14.319
Yeah, absolutely.
00:03:14.479 --> 00:03:21.199
And so tell me a little bit more about then how your data team is structured and kind of what you're responsible for.
00:03:21.599 --> 00:03:29.919
Yeah, so we basically um our the the core um platform is built in C sharp.
00:03:30.080 --> 00:03:38.400
And so we've got three kind of operational teams that look after that, which is you know look divided into three distinct uh areas.
00:03:38.960 --> 00:03:58.000
The data team basically sits across the entire business um to kind of consume data from everywhere uh in the Insignus engine, as I like to call it, and try and deliver data-driven insights to the rest of the business and to the C-suite.
00:03:58.479 --> 00:04:09.199
Um and uh we not only sit across the operational team, so we collect internal data, but we also collect a lot of external data as well.
00:04:09.520 --> 00:04:22.720
So our um customer services team, for example, their calls such as they have with their with clients, get recorded in Ringover, so we export data from Ringover.
00:04:22.959 --> 00:04:50.000
We've got our um CRM set up in Salesforce, so we um import data from Salesforce, and you know, so we try and build a kind of single plane of glass so that anyone in the company can just come to us with uh uh an insight they either want to know or or they want to look at data, where that single place where you can look at all the data of what's happening in the company.
00:04:50.240 --> 00:04:54.800
I like to think of us as a basically a newspaper.
00:04:54.959 --> 00:05:02.160
You you you kind of get an idea of what's happening in the business, but and we'll we'll get I'm sure we'll get into that later.
00:05:02.480 --> 00:05:11.279
What is important for us is we don't have um core business kind of logic sitting within our data platform.
00:05:11.439 --> 00:05:15.839
We are just simply there to be the newspaper for the rest of the business.
00:05:16.240 --> 00:05:17.600
Okay, interesting.
00:05:17.920 --> 00:05:27.759
I love the analogy of the newspaper, and so you're just providing all of the information, then what the other teams do with it is sort of up to them based on their use cases.
00:05:28.000 --> 00:05:34.160
Yeah, it's almost like yeah, it like and sometimes I I think it's served us very well so far.
00:05:34.240 --> 00:05:38.959
And sometimes a bit people have been like, uh, you know, why why is this broken?
00:05:39.040 --> 00:05:42.079
It's like, don't know, like you have to ask the operational teams.
00:05:42.160 --> 00:05:44.800
I'm just uh I'm almost like the messenger here.
00:05:44.879 --> 00:05:46.480
I just report on what I see.
00:05:46.800 --> 00:05:48.879
Yeah, yeah, that makes sense.
00:05:49.040 --> 00:05:56.639
And I think you have told me previously, if I'm not mistaken, that you are the only data engineer at the company.
00:05:56.800 --> 00:06:00.800
So your function, at least within that team, is of one person.
00:06:01.680 --> 00:06:02.319
Yes, correct.
00:06:02.399 --> 00:06:06.079
We've uh we've got um um another person.
00:06:06.240 --> 00:06:12.639
He's he's more of a data analyst like his job title says data engineer, but we also need a data analyst within the team.
00:06:12.720 --> 00:06:18.639
So he looks after the data analysis side, and yeah, I looked after mostly the data engineering side.
00:06:18.959 --> 00:06:19.279
Okay.
00:06:19.920 --> 00:06:24.879
And that means you probably got a lot of responsibility that's just on you.
00:06:25.040 --> 00:06:32.560
I'm curious how does that shape you know the way that you work and yeah, like in in the beginning it was tough for sure.
00:06:32.800 --> 00:06:41.040
Um I came from JLR from Jaguar Land Rover, and that had a very big data engineering team.
00:06:41.360 --> 00:06:50.959
So in the beginning, really, I didn't if I had ideas that I wanted to discuss, I couldn't really discuss them with anybody else.
00:06:51.120 --> 00:07:02.399
Um, so for me personally, AI has been really helpful with that, just as a like someone to have a back and forth, like a like to bounce ideas off.
00:07:02.560 --> 00:07:17.360
I you know, I've I've definitely done it with my software engineering colleagues, um, and they have been helpful to an extent, but to like it's different when you're talking to like a data engineering colleague because they they might not have all the context.
00:07:17.519 --> 00:07:22.720
Yeah, a software engineer might not have all the context a data engineering does, a data engineer does.
00:07:23.040 --> 00:07:35.839
Um and but but what's also important for us is the way we shape our work is we still want the data engineering pipelines to be treated as first-class citizens.
00:07:36.000 --> 00:07:42.959
So it has the same rigorous set of testing, see set of CICDs, extensive documentation.
00:07:43.040 --> 00:07:49.199
You know, it's not I certainly don't allow it to be treated like an afterthought.
00:07:49.519 --> 00:07:58.399
Um, you know, it it it deserves the same kind of um rigorousness as the other software engineering teams as well.
00:07:58.720 --> 00:07:59.920
Yeah, absolutely.
00:08:00.160 --> 00:08:06.560
Well, I'm sure that constraint will be relevant to some of our upcoming discussion about airflow as well.
00:08:06.720 --> 00:08:16.160
I have also been myself not in my current role, but in pastorals in this position of being the only analyst at a company or something like that.
00:08:16.240 --> 00:08:19.439
And yeah, it's a hard, it's a hard spot to be in sometimes, definitely.
00:08:19.680 --> 00:08:20.879
Yeah, definitely.
00:08:21.120 --> 00:08:25.120
Um yeah, so that's great that you've managed to make that work.
00:08:25.279 --> 00:08:29.680
But again, I I do want to talk about how you know Airflow fits into this.
00:08:29.839 --> 00:08:35.200
So tell me about how your data platform is structured and what role Airflow plays.
00:08:35.759 --> 00:08:41.519
Yeah, so Airflow is basically at the very heart of our data, of our data infrastructure.
00:08:41.759 --> 00:08:45.360
Um, it's our main tool for orchestrating data pipelines.
00:08:45.919 --> 00:08:58.559
So we not only use it to orchestrate when we, for example, get uh third-party data from using APIs, but we also use it as an observation layer.
00:08:58.799 --> 00:09:17.919
So we've got um a couple of real-time pipelines that we have, which is not built with Airflow, but Airflow kind of sits on top of those to build an observation layer to ensure that you know those real-time pipelines are definitely working uh as intended.
00:09:18.159 --> 00:09:36.240
And that kind of observe like observability layers we found that it's very useful with what capability Airflow provides for us, uh, you know, as well as being you know our standard tool for kind of like you know, the standard ETL job.
00:09:36.480 --> 00:09:37.840
Um, yeah.
00:09:38.080 --> 00:09:53.279
And yeah, and we also use a lot of and again I'll get into that, but yeah, like for us, and I'm sure this is not a uh not a revolutionary thing, I'm gonna say, is Airflow for us is the the orchestrator for everything.
00:09:53.679 --> 00:10:00.720
It it tells other things to do things and then returns for a result back, but doesn't actually do the things itself.
00:10:01.200 --> 00:10:11.600
Um so it yeah, so it it like it it controls everything, it's like the brain, but it doesn't actually perform any of the heavy lifting as much as possible.
00:10:12.080 --> 00:10:22.000
Other tools perform, you know, for example, SQL pipelines, they perform the head like in BigQuery, it performs the heavy lifting, and Airflow just orchestrates that.
00:10:22.480 --> 00:10:23.440
Yeah, absolutely.
00:10:23.600 --> 00:10:29.679
I think that is a really common pattern, and I do want to get into that a little more before we do, just high level.
00:10:29.840 --> 00:10:33.759
Tell me a little bit about what kinds of data pipelines you're running with Airflow.
00:10:34.320 --> 00:11:05.279
Yeah, so we've got um we've got a few operational ones, we've got a few where we use uh where we try and get data from external systems, and we try and do that as um like even if they're comp like even if we are using APIs, we try and use methods that are kind of best practices, not just to data engineering, but to software engineering, things like using uh watermarks when retrieving API results.
00:11:05.519 --> 00:11:13.279
We try in terms of kind of cost saving, we try and stay two or three steps ahead.
00:11:13.519 --> 00:11:19.919
You know, you get data from an API, you you get all the data there, it's fine, it works, you know.
00:11:20.159 --> 00:11:31.759
But we've because we don't have that lock, that luxury of having a huge budget behind us, we kind of have to think two or three steps ahead, like right, it works, but now how can I make it as efficiently?
00:11:31.919 --> 00:11:34.879
How can I get the data as efficiently as possible?
00:11:35.039 --> 00:11:46.000
And on top of that as well, we've got some we've got some operational pipelines of our own, which the businesses rely on um to use every day.
00:11:46.159 --> 00:12:09.519
So we also make sure that in our the data pipelines that we do have, the relevant observe observability layers, the relevant testing, making sure that you know things like if it fails or if it misses an SLA, it's all kind of there as in best practice to catch it so that incidents can be resolved as painlessly as possible.
00:12:09.840 --> 00:12:10.159
Okay.
00:12:10.399 --> 00:12:11.679
Yeah, got it.
00:12:12.000 --> 00:12:22.799
And then you've alluded to this a little bit, but I know you've also spoken about building, you know, a low-cost, scalable platform for airflow.
00:12:22.879 --> 00:12:28.320
And we've talked already about how you're working within the constraint of you know a very small team.
00:12:28.559 --> 00:12:28.879
Yeah.
00:12:29.120 --> 00:12:35.279
Tell me about how you have gone about implementing this, maybe anything else about kind of the importance of doing it.
00:12:35.440 --> 00:12:47.360
And I guess, you know, what advice you would give to other small teams who are trying to use Airflow, but you know, need to be very conscious of cost and things like that, regardless of where or how they're running it.
00:12:47.840 --> 00:13:03.759
Yeah, I think I think if you're in a similar position, one of the things I'd say first is um observability building in observability layers is should be a part of your definition of done and not just an afterthought.
00:13:03.919 --> 00:13:14.559
And I don't and I don't necessarily mean observability in the sense that um I went to my Airflow pipeline, I went to the Airflow UI, it's all green.
00:13:14.720 --> 00:13:16.080
That's not observability.
00:13:16.320 --> 00:13:25.440
Observability is in my opinion, taking a step back and kind of seeing like is the output why expect?
00:13:25.759 --> 00:13:32.559
Like am I like the pipeline runs fine, but are things being silently dropped?
00:13:32.639 --> 00:13:49.440
And being very kind of curious about that, so building in observability layers where uh a great thing that's worked for us is uh dashboards built in in um in our stack that is kind of independent of the main data pipelines themselves.
00:13:49.759 --> 00:14:05.600
So it kind of does his thing, the main data pipelines do their thing, so uh we kind of know that yes, the pipeline's passed, and yes, the data that is being fetched, that whatever it's trying to do is working as intended.
00:14:05.840 --> 00:14:12.720
So that that will definitely help to keep costs uh um uh running as efficiently as possible.
00:14:12.879 --> 00:14:19.120
And the other thing as well is when you're a small team, um automation is very important.
00:14:19.279 --> 00:14:29.840
Uh so to automate and this might sound like an obvious thing, but you'd be surprised how many times I've I mean, when when I joined in Insignus, for example, uh, you know, this was happening.
00:14:30.159 --> 00:14:32.399
Automate things that take the most time.
00:14:32.799 --> 00:14:39.600
So uh it can be used intelligently as a substitution for headcount if you know what you're doing.
00:14:39.759 --> 00:14:53.039
You don't uh the the um one of the aims of the big game is to uh uh try and scale the company uh but not necessarily scale it with headcount, right?
00:14:53.120 --> 00:14:54.000
If that makes sense.
00:14:54.240 --> 00:15:01.519
So be uh mindful and be intelligent of what you can automate, what takes you the most time, and go ahead and do it.
00:15:01.600 --> 00:15:05.360
That that that will definitely help you with with kind of cost savings.
00:15:05.759 --> 00:15:27.840
Um and then I I will also say the other thing is in our data platform we used a managed airflow version, we don't host things ourselves ourselves because whilst we do have a um platform team, it's not solely dedicated to the data team.
00:15:28.480 --> 00:15:37.919
Um if you use the managed airflow version, uh like the things like the scheduler, the workers, the database is all is all kind of abstracted away from you.
00:15:38.159 --> 00:15:50.480
So that leaves you free to write your code and write your documentation so that you know you don't have to spend your time putting on your platform hat and looking after those as well.
00:15:50.559 --> 00:15:57.039
So that that will also help you to kind of manage your costs if you are a small team and you want to get started in Airflow.
00:15:59.519 --> 00:16:08.000
This episode is brought to you by Astronomer, the team behind Astro, a managed Airflow platform built for data engineering teams.
00:16:08.240 --> 00:16:12.159
From batch pipelines to training models and wiring up agents.
00:16:12.399 --> 00:16:18.399
Astro handles the infrastructure, upgrades, and scaling so you can focus on building.
00:16:18.639 --> 00:16:21.039
Check it out at astronomer.io.
00:16:23.200 --> 00:16:24.320
Yeah, absolutely.
00:16:24.480 --> 00:16:27.919
I there's so much good stuff there that I want to double-click on.
00:16:28.080 --> 00:16:35.840
I do think the managed service, of course, we're uh very heavily biased at Astronomer, but we do think that this helps people quite a lot.
00:16:35.919 --> 00:16:39.840
Um, again, as you said, so that you don't have to worry about the airflow infrastructure.
00:16:40.000 --> 00:16:55.600
And I think that um relevant for automation as well, because I think it allows you to implement, you know, things like auto-scaling and stuff like that, where you can um just have a lot more intelligence built into the platform as opposed to trying to manually tweak those things.
00:16:55.919 --> 00:17:08.720
Uh I'm curious actually, going back to the first thing you said on observability and how it's not just, you know, looking and seeing if you had any tasks fail, but also making sure that your output looks like what it should and all of that.
00:17:09.039 --> 00:17:21.519
I, you know, oftentimes I think talk to people about that in the context of implementing things like data quality checks so that they're catching, you know, things where the task succeeded, but like something is wrong with the data.
00:17:21.759 --> 00:17:24.240
I don't know if that's exactly what you're talking about.
00:17:24.400 --> 00:17:31.359
Um I mean, I can give I can give a specific example without um without giving away company secrets.
00:17:31.440 --> 00:17:34.240
So yeah, data quality checks are definitely important.
00:17:34.880 --> 00:17:48.000
We we definitely we definitely use um a lot of them in our pipelines to make sure that the the data kind of before and and after you know it all looks and is all aligned.
00:17:48.160 --> 00:18:02.559
But what I mean, you know, a great example of this is um our customer services team chat with clients every day and we record calls and transcripts, right?
00:18:03.200 --> 00:18:13.759
Now we've got uh a data pipeline that gets calls and transcripts and every day it shows yeah, it's green, it's getting them.
00:18:14.079 --> 00:18:17.440
But how do you know that they're getting all of them?
00:18:17.599 --> 00:18:20.160
How do you know that they're not silently dropping anything?
00:18:20.480 --> 00:18:25.519
How do you know that um uh you know things are not being muddled along the way?
00:18:25.680 --> 00:18:36.319
So that's where building a separate observer, which I I I appreciate it can um it can be case specific, it can be project specific.
00:18:37.039 --> 00:18:42.960
I mean, the example I'm giving is specific to this particular pipeline, it might not apply to others.
00:18:43.119 --> 00:18:58.319
So, but we've built observable observability layers where we look at the output of what we've actually collected and then do like a kind of summary, like right, okay, um airflow pipeline is green.
00:18:58.480 --> 00:19:07.440
Ah, but like look, we've got all the calls that Ringover said we've had, we've got all the transcripts, so there doesn't seem to be any errors, nothing is being dropped silently.
00:19:07.599 --> 00:19:17.119
So that's why you know that observability layer in addition to your data quality checks for us is what has has been quite important.
00:19:17.680 --> 00:19:18.799
Yeah, makes sense.
00:19:18.960 --> 00:19:31.039
And tell me about how so, of course, having that data quality is really important for being able to make accurate business decisions, but you're also making the argument for cost optimization.
00:19:31.200 --> 00:19:34.880
So, how does having that observability layer help with cost?
00:19:35.839 --> 00:19:47.920
So it helps with costs because I think um you could argue like, well, you're building more things, so in the short term is costing you more, right?
00:19:48.400 --> 00:19:59.680
But by having that observability layer, we're not we're actually we the thing that's gonna the thing that it will do is prevent incidents from happening.
00:20:00.000 --> 00:20:05.440
So people downstream are not gonna scream like, well, I I can't see any of the ring over data, where is it?
00:20:05.680 --> 00:20:13.920
That in itself will incur a much bigger cost than the kind of short-term costs you see of building those observability layers in the first place.
00:20:14.240 --> 00:20:23.920
So the cost of having that incident downstream is gonna be much bigger than the cost, then the initial cost of building the observability layer from the start.
00:20:24.240 --> 00:20:24.880
Got it.
00:20:25.039 --> 00:20:26.559
Yeah, okay, I see.
00:20:27.119 --> 00:20:44.079
And I guess related to that potentially, are there any patterns within your DAGs themselves in terms of implementing best practices that you think have made a big difference for you in terms of cost or scalability?
00:20:44.480 --> 00:20:46.559
Maybe other than data quality checks?
00:20:46.880 --> 00:20:55.599
Yeah, other than data quality and other than using, which I've already which we've spoken about, using Airflow as an orchestration engine, not an execution one.
00:20:56.319 --> 00:21:04.880
Um I would say automating your CI in a deployment, don't be kind of tempted to leave those as an afterthought.
00:21:05.039 --> 00:21:10.160
Um, and let your CI pipeline handle as much of the automated tests as possible.
00:21:10.480 --> 00:21:16.799
Let machines pick up things so you don't have to manually kind of do it yourself.
00:21:16.960 --> 00:21:56.799
Things like linting, unit testing, um documentation, and make sure that all your DAGs are kind of consistent, they all have a level playing field, and a great way to do that is enforcing it through your CI CD pipeline, um, and making sure that you know, like your DAGs have all the kind of best practices set out by Airflow and Astronomer, things like failure callbacks, um, being mindful of where you how you put the documentation, being mindful of where you're when your DAG starts and when it ends, and making sure it doesn't, you know, it doesn't go rogue and just do loads of which I have seen happen um before.
00:21:57.039 --> 00:22:10.640
So um And the other thing as well for us is we purge our logs and our metadata because we find that the metadata table can build up very quickly.
00:22:12.000 --> 00:22:18.319
So to kind of make sure that you know things are operating at that sweet spot, we purge them periodically.
00:22:18.480 --> 00:22:24.240
Um so that you know we don't hold on to logs you know for you know that like a year old.
00:22:24.319 --> 00:22:38.000
You know, we don't we don't need to see um you know that that it's just that you know logs that are recent that that may produce a failure, those are the ones that we want to see so that we can begin to debug if we need to.
00:22:38.559 --> 00:22:42.079
Yeah, that's a recommendation we make very commonly to our customers as well.
00:22:42.160 --> 00:22:44.079
It can have big performance impacts.
00:22:44.799 --> 00:22:46.160
Yeah, definitely.
00:22:46.640 --> 00:22:55.839
Okay, and then I understand that you have also built some internal tools kind of around your Airflow platform to help keep it running efficiently.
00:22:56.079 --> 00:22:58.640
Tell me about those and how they help you.
00:22:59.119 --> 00:23:10.240
Yeah, so the first thing is um uh we built a tool to show our SQL pipeline costs and we call that tariff.
00:23:10.720 --> 00:23:17.279
Um so basically, uh Airflow is responsible for orchestrating our SQL pipelines as well.
00:23:17.519 --> 00:23:29.039
And the the reason why we built this tool is one, we you know, we just want to see in a plain and clear way as possible how much our SQL pipelines are costing.
00:23:29.279 --> 00:23:38.400
We don't want those to be creeping up and then you know um you know leaking into our airflow costs and our overall data infrastructure costs as well.
00:23:38.559 --> 00:23:43.519
That's something that we've that you know it has a front end and a back end.
00:23:43.680 --> 00:24:00.960
Um, and we find that it's definitely given us visibility and and some interesting insights, things like um, if you have kind of you know incremental mechanisms, what you might think is like, oh yeah, it's it's working really great, it's actually costing you a lot.
00:24:01.119 --> 00:24:08.400
So it's actually helped us with the optimizations um a lot in our data infrastructure to kind of keep costs down.
00:24:08.640 --> 00:24:23.039
The second thing is um we built an AI-powered tool to give recommendations on how to improve the Airflow platform in terms of like the workers, their memory, the scheduler, how big the database should be.
00:24:23.119 --> 00:24:24.240
We call that Maestro.
00:24:24.799 --> 00:24:30.640
Um previously this kind of took a long time, and it was it was for me to do.
00:24:30.880 --> 00:24:43.519
Um, you know, it would you would kind of have to analyze your DAGs, analyze how they did over a period of time, analyze how your workers in your Airflow environment are doing.
00:24:43.599 --> 00:24:46.160
You know, are they running out of memory frequently?
00:24:46.400 --> 00:24:49.680
Are they just about to run out of memory or are they doing okay?
00:24:49.759 --> 00:24:56.480
And then it was very much a finger in the air, like, oh I think the worker provision should be increased by X amount.
00:24:56.799 --> 00:25:03.920
This kind of takes it all away, um, you know, and it takes it takes minutes, you know, to do and give recommendations.
00:25:04.319 --> 00:25:11.519
Um, so no, so tools like that where it's really helped us to make sure that again that goes back to the point of automation.
00:25:12.079 --> 00:25:24.960
You know, I could have easily not easily, but it would have taken me a while to kind of look through a SQL pipeline and go, right, you know, what are what is uh costing the most here and what is costing the least and why.
00:25:25.279 --> 00:25:27.519
You know, we've kind of automated all of that.
00:25:27.759 --> 00:25:29.359
Same for Maestra as well.
00:25:30.079 --> 00:25:48.880
So it's tools like that that kind of helped us to uh you know build our data infrastructure, you know, well from the start, uh, and to make sure that it's it's not just about, like I said before, it's not just about short-term let's just get costs down, let's just spend a pound a day.
00:25:49.440 --> 00:25:54.960
The the the point I want to stress here is you want to find that sweet spot.
00:25:55.680 --> 00:26:11.279
Just you you want to give, for example, you want to give your airflow workers just enough juice so that they need they can they have what they need to do to do the work that they're asked to do, but not so much that you are it's costing you an an absolute arm and leg.
00:26:11.440 --> 00:26:21.440
So you want to find that sweet spot, and so the tools we've built there is helping us to find that sweet spot to help us you know keep our data pipelines running every day.
00:26:21.759 --> 00:26:31.039
Yeah, I think that's a great description of kind of striking the right balance between those short-term adaptations and the long-term sustainability of your platform.
00:26:31.200 --> 00:26:32.880
Yeah, very interesting.
00:26:33.519 --> 00:26:37.839
Well, on a slightly different topic, maybe, maybe it's related.
00:26:37.920 --> 00:26:45.599
Uh, I know you've also mentioned that you just started exploring AI with Airflow, which is of course a really popular topic right now.
00:26:45.759 --> 00:26:49.839
Um, I'm curious what sort of use cases are you looking at implementing there?
00:26:50.480 --> 00:26:58.480
So one of some of the obvious UK use cases is probably some analysis of the movement of money.
00:26:58.720 --> 00:27:04.559
So things like um our clients, how do they kind of invest with us?
00:27:04.720 --> 00:27:07.039
Um, it's more kind of MLOPs, actually.
00:27:07.279 --> 00:27:13.359
Um, you know, we want we want to kind of identify patterns and try and capitalize on them.
00:27:13.519 --> 00:27:22.000
Um with move like you know, and maybe things like in our customer journey, things may not be as optimized as they can be.
00:27:22.079 --> 00:27:26.880
So we what kind of you kind of you want to use AI and MLOPs to help us identify those.
00:27:27.200 --> 00:27:39.440
One of the other things I found really inspiring from the Airflow Summit, which has just gone by, is you know, the the idea of self-healing pipelines, which is a very juicy topic to me.
00:27:39.519 --> 00:27:43.440
So, you know, things like and again that comes back to the whole point, right?
00:27:43.680 --> 00:27:45.839
Automating it away as much as I can.
00:27:46.079 --> 00:28:09.200
Not automating it fully so that an AI just goes wild and on the rampage and changes things, but to help us as much as possible when there are things going wrong, you know, an AI can do the analysis much faster, looking through logs and seeing what's got what's gotten wrong, implementing uh suggesting a fix, an AI can do that much faster than a human can.
00:28:09.359 --> 00:28:12.880
So definitely in our future, a kind of self-healing pipeline.
00:28:12.960 --> 00:28:16.880
And and there was loads of cool talks, people had different ways of doing it.
00:28:17.039 --> 00:28:23.680
There was, you know, one of the talks I think I went to was a DAG does it, a DAG that listens to other DAGs.
00:28:23.839 --> 00:28:25.440
So that was cool as well.
00:28:25.839 --> 00:28:33.279
We've got some ideas of how it can be done at Insignus for us, so we're definitely very interested to leverage that.
00:28:34.240 --> 00:28:35.119
Yeah, absolutely.
00:28:35.279 --> 00:28:36.960
This is something I'm doing myself as well.
00:28:37.039 --> 00:28:41.279
Of course, Astronomer has our own perspective on how we can implement this within our platform.
00:28:41.359 --> 00:28:43.039
As you said, there's many ways of doing it.
00:28:43.200 --> 00:28:45.200
I love that the community are sharing those.
00:28:45.359 --> 00:28:53.200
But yeah, it has certainly helped me a lot in the debugging of the pipelines that you know provide metrics and stuff for my team.
00:28:53.359 --> 00:29:00.240
Um, it's often very quick at identifying any issues and fixing them compared to what it used to take me to debug them myself.
00:29:01.279 --> 00:29:02.400
Yeah, that's great.
00:29:02.559 --> 00:29:11.440
Well, I think uh in the interest of time, that's a good uh lead into my standard final question, which is what would you most like to see from the Airflow project in the near future?
00:29:11.920 --> 00:29:26.559
I think one of the things actually I'd like to see is um in terms of when DAGs get passed, um, so that when they're when when they when they show up in the UI, I'd like to see that a little bit faster.
00:29:26.799 --> 00:29:36.079
It's not a huge bottleneck for us, but it sometimes takes as long as longer than we like to see when we implement changes to then kind of see that in the UI.
00:29:36.400 --> 00:29:44.400
Um and I know it a lot of it comes down to the DAG file processing, if you have things like heavy, heavy top-level imports.
00:29:44.799 --> 00:29:51.920
Um, but I think the repassing at the moment works on an interval rather than an event-based change.
00:29:52.079 --> 00:29:55.519
So I'd love for X Airflow to explore that more.
00:29:55.839 --> 00:30:08.960
Um to you know, to make it you know a little bit more the DAC passing a little bit more event-based, so that you can kind of see changes close to instantaneous rather than kind of having waiting an interval.
00:30:09.279 --> 00:30:10.319
Yeah, absolutely.
00:30:10.480 --> 00:30:11.519
That's a great one.
00:30:11.839 --> 00:30:14.640
Okay, well, Inurita, thank you so much for coming on.
00:30:14.720 --> 00:30:16.400
This has been a really great discussion.
00:30:16.559 --> 00:30:19.599
Like I said, I think it will be very helpful for a lot of our listeners.
00:30:19.759 --> 00:30:23.119
Just to close this out, what is the best way for folks to get in touch with you?
00:30:23.440 --> 00:30:26.079
Uh you can just yeah, find me on LinkedIn, probably.
00:30:26.160 --> 00:30:27.440
I'm quite active there.
00:30:27.599 --> 00:30:32.000
Um, but yeah, feel free to also reach out to me via email.
00:30:32.319 --> 00:30:36.319
Um yeah, I'd say those are the two options.
00:30:36.640 --> 00:30:36.960
Great.
00:30:37.119 --> 00:30:39.599
We'll throw your LinkedIn in the show notes for sure.
00:30:39.759 --> 00:30:41.200
Well, thank you again for coming on.
00:30:41.279 --> 00:30:42.400
It's been great speaking with you.
00:30:42.720 --> 00:30:43.200
No problem.
00:30:43.279 --> 00:30:44.319
Yeah, thanks for having me.
00:30:44.559 --> 00:30:47.440
Thank you for listening to the Data Flowcast.
00:30:47.599 --> 00:30:51.039
Check the show notes for links to everything mentioned today.
00:30:51.200 --> 00:30:55.839
If you're enjoying the podcast, subscribe so you never miss an episode.
00:30:56.000 --> 00:30:58.319
And leave us a five star review.
