WEBVTT
00:00:00.160 --> 00:00:03.680
I think curiosity should lead the learning, right?
00:00:04.000 --> 00:00:12.320
So some since I was open for my curiosity, initial times it was a little hard, but then it was very fast to learn.
00:00:12.480 --> 00:00:19.120
So resource-wise, I think mix of official talks, blogs, and there are like really good crash courses and stuff like that.
00:00:19.280 --> 00:00:29.440
So when I learn something new, my equation or framework is that like watch a crash course, find what it's used for, and understand what's happening here.
00:00:34.479 --> 00:00:42.719
You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
00:00:44.880 --> 00:00:47.840
Here, Airflow community, we've got exciting news.
00:00:48.000 --> 00:00:50.880
The annual Apache Airflow survey is live.
00:00:51.039 --> 00:00:58.640
In just seven minutes, you can help shape the future of Airflow and the free Airflow 3 Fundamentals or Demethrine certification.
00:01:05.280 --> 00:01:06.319
Don't miss out.
00:01:06.480 --> 00:01:10.319
The survey link is in the show notes and closes November 21st.
00:01:12.719 --> 00:01:13.599
Hello everyone.
00:01:13.760 --> 00:01:15.280
Welcome to the Dela Flowcast.
00:01:15.359 --> 00:01:24.159
I'm your host, Mark Lumberti, and I'm super excited because today I'm joined by Nassim Shah, head of engineering at Xena Intelligence.
00:01:24.400 --> 00:01:26.400
I mean, the first question is how are you doing?
00:01:26.719 --> 00:01:27.439
Hi, it's good.
00:01:27.519 --> 00:01:29.040
I'm really excited for this.
00:01:29.120 --> 00:01:30.239
It was a really great day.
00:01:30.319 --> 00:01:34.560
It was raining and all, and it was such a beautiful day to start on.
00:01:35.040 --> 00:01:41.280
And you know how things go and had a great day at work, did a lot of problem solving and all.
00:01:41.439 --> 00:01:42.799
It's exciting stuff.
00:01:42.959 --> 00:01:44.879
So yeah, never bothered.
00:01:45.120 --> 00:01:45.680
That's awesome.
00:01:45.840 --> 00:02:02.799
Well, I'm really glad to be here with you today because we're actually gonna talk about a lot of exciting stuff, more specifically, how you and obviously your team also transformed from Chrome jobs to sophisticated LLM-powered review analysis using obviously AI.
00:02:02.959 --> 00:02:10.560
And so you will discover also how Nassim and the team is able to process millions of Amazon reviews.
00:02:10.719 --> 00:02:15.520
So this is super exciting, and obviously, everything is orchestrated using Airflow.
00:02:16.000 --> 00:02:19.840
So I would like to start with your background.
00:02:20.080 --> 00:02:26.479
Okay, so can you tell us about your journey into data engineering and what led you to discover Airflow?
00:02:26.800 --> 00:02:33.039
And also what were some of the data pipeline challenges that you were facing before adopting Airflow?
00:02:33.280 --> 00:02:44.000
So I come from a software engineering background, not really a data engineer as per se, when situations and things got into data engineering related works also.
00:02:44.159 --> 00:02:46.960
So I actually come from more of like a software engineering background.
00:02:47.039 --> 00:02:50.000
I didn't know much about the world of data engineering at the start.
00:02:50.159 --> 00:02:53.919
I joined intelligence straight out of college as a first engineering hire.
00:02:54.240 --> 00:03:06.639
So basically, as a founding engineer, so you have to gather things around and learn new things and the requirements from the stakeholders, from the founders, the CEO, director, reporting to them, directors.
00:03:06.800 --> 00:03:09.120
Like you have to find things on your own.
00:03:09.199 --> 00:03:18.319
And that gave me a lot of exploring and gave me a lot of freedom to explore and find things on my own and find it my own way.
00:03:18.479 --> 00:03:22.080
So that was really exciting for me and straight off a college.
00:03:22.159 --> 00:03:25.280
It's something that you don't get when you join an MNC.
00:03:25.360 --> 00:03:28.479
The first that's how I come into this picture.
00:03:28.719 --> 00:03:39.039
So my role earlier was to build MVPs and get product out of the ground, which ended up helping us raising funds and join accelerators like Techstars.
00:03:39.840 --> 00:03:41.280
A little bit about the company.
00:03:41.360 --> 00:03:44.560
So Xena is an AI-powered e-commerce ground platform.
00:03:44.639 --> 00:03:47.520
So everything we do is really data heavy.
00:03:47.680 --> 00:04:02.159
So we would pull in data from marketplaces like Amazon, Walmart, Shopify to generate insights for things like ad optimization, inventory planning, inventory focused, team market research, customer behavior, and all those stuff, right?
00:04:02.400 --> 00:04:06.000
So in the beginning, our pipelines were just like ground scripts, right?
00:04:06.080 --> 00:04:13.280
So honestly, it worked fine for a few customers, especially if we were like pre-revenue beta testing and all those stuff, right?
00:04:13.520 --> 00:04:18.879
So initially, my motto is start simple, time update.
00:04:18.959 --> 00:04:24.480
So you don't need to complicate into the first and waste our time and productivity on that, right?
00:04:24.720 --> 00:04:28.240
So my first approach was like start out simple.
00:04:28.319 --> 00:04:35.519
So it's few customers, handful of API calls and downloading report documents and processing it in Python.
00:04:35.600 --> 00:04:38.800
And it was very simple and it was easy, right?
00:04:38.879 --> 00:04:40.959
And we didn't waste much time on that.
00:04:41.199 --> 00:04:51.680
That's why we could ship more features when so when we started to grow and move to product market fit, those crones started to become like a nightmare.
00:04:51.839 --> 00:05:01.040
Scaling out database was easy where backends are easy, you can just containerize and it's fairly easy to scale back end APIs and stuff like that.
00:05:01.199 --> 00:05:09.279
But when it comes to the data pipelines, it was really not that easy and was a very big challenge as a fresher when I started out.
00:05:09.920 --> 00:05:16.720
We had cron jobs failing randomly, no clear visibility, which into let most data, no clear visibility.
00:05:17.199 --> 00:05:26.959
I often find myself logging in through heaps of logs at later, uh trying to find the needle in the haystack kind of situation here.
00:05:27.120 --> 00:05:30.079
So it was very hard to debug and stuff like that.
00:05:30.399 --> 00:05:34.639
So we then started to try a small upgrade.
00:05:34.800 --> 00:05:36.240
We tried small upgrade.
00:05:36.480 --> 00:05:41.120
So that's how I re-architected with again baby steps, right?
00:05:41.279 --> 00:05:42.959
SQS and easy to set up.
00:05:43.040 --> 00:05:47.360
I didn't want to complicate it very fast, but slowly.
00:05:47.519 --> 00:05:53.519
So it was interesting because we had SQS queues, so it's like Kafka queues and stuff like that.
00:05:53.680 --> 00:05:58.160
Just the message comes in and consumer gets it and process the data.
00:05:58.240 --> 00:05:59.680
And it was fairly simple.
00:05:59.839 --> 00:06:02.800
So we had one queue for one report type.
00:06:03.040 --> 00:06:06.879
And for customer, it was like five queues, five consumers.
00:06:07.199 --> 00:06:08.800
It was okay for that time.
00:06:08.879 --> 00:06:14.800
And when we had like sub-20-30 customers, it was okay for us like to go through that.
00:06:15.040 --> 00:06:21.439
But when we started grow and customers flooding in and stuff like that, it was very hard again.
00:06:21.519 --> 00:06:24.560
Like which customer data pipeline go down?
00:06:24.720 --> 00:06:26.480
Why this isn't it?
00:06:26.800 --> 00:06:32.560
Customers start comes reach out, support, and things were not uh looking good.
00:06:32.720 --> 00:06:40.480
And AWS SQS and going to the dead letter queue, myself and our team, and trying to understand.
00:06:40.639 --> 00:06:46.319
And sometimes the code works to customers, it doesn't work for others.
00:06:46.480 --> 00:06:53.120
So the lack of visibility was slowing down the scalability and our health of the platform itself.
00:06:53.279 --> 00:06:57.120
So again, it's not just the features, but we have to deliver it right.
00:06:57.839 --> 00:07:05.120
So that lack of visibility, lack of scalability, and it was not developer-friendly, right?
00:07:05.279 --> 00:07:10.639
So it cost us productivity time and all those different aspects of what we do.
00:07:10.720 --> 00:07:21.839
And we were like a very small team, like we are three engineers just managing all these data pipelines, all these features, and we have to make the stakeholders happy, and we just have to grind it.
00:07:22.000 --> 00:07:24.959
So that's started to research about data engineering.
00:07:25.040 --> 00:07:36.000
That's how we get into data engineering, and that's how I understood all there's a world like this, and there is lots of tools, and that's how I ended up in the airflow, right?
00:07:36.160 --> 00:07:39.199
Okay, it was perfect for us when you started.
00:07:39.439 --> 00:07:40.480
Yeah, wow.
00:07:40.560 --> 00:07:49.120
So I think like many teams will listen to that podcast and will realize that they are getting the same issues that you got in the past before Airflow.
00:07:49.279 --> 00:07:52.480
And some people always wonder, hey, should I use Airflow?
00:07:52.720 --> 00:08:06.160
If you listen, NASIM, you truly understand that if every day you are struggling with logs, with pipelines that you cannot properly monitor, that you cannot properly debug, and so on, obviously, Airflow will be a huge improvement for you.
00:08:06.319 --> 00:08:16.319
And honestly, at some point, when you feel that you are overwhelmed with so many issues and so on, and that your data pipelines are not reliable as they should be, I mean it's time to go with an orchestrator.
00:08:16.720 --> 00:08:22.399
And you know, Nassim, it's interesting because you said that you are a small team, right?
00:08:23.040 --> 00:08:33.679
And many people actually, I wouldn't say many, but some people believe that you need to be a huge team just to set up and run Airflow in production, which is not true.
00:08:33.840 --> 00:08:40.240
But tell me more about your experience with Airflow and more specifically, what was your experience learning Airflow?
00:08:40.320 --> 00:08:48.240
And do you have any particular resources, approaches, or things like that truly helped you in your journey with Airflow and as a newcomer to Airflow?
00:08:48.480 --> 00:08:51.919
Yeah, learning Airflow definitely felt intimidating at first.
00:08:52.159 --> 00:08:58.879
It helped a lot when I came through the software engineering lens because I understood what Docker is.
00:08:58.960 --> 00:09:00.240
I knew all these things, right?
00:09:00.399 --> 00:09:01.440
What are the containers?
00:09:01.600 --> 00:09:09.519
Why Airflow has different containers, and so it clicked me easily since I'm looking through an engineer's thing.
00:09:09.759 --> 00:09:20.000
But again, for a starter, it might be a little hard, even though I knew that it was a little intimidating at first, because what is a lot of things at first?
00:09:20.159 --> 00:09:22.080
As time moved, it was easier.
00:09:22.240 --> 00:09:23.279
So, yeah, definitely.
00:09:23.360 --> 00:09:28.799
I think I went through a lot of engineers go through what people call tutorial health, right?
00:09:28.879 --> 00:09:39.600
So I was watching crash courses, jumping into dogs, trying to piece things together, and it wasn't just clicking right away, but slowly it was the progression was very slow, right?
00:09:39.759 --> 00:09:45.279
What really helped me was stepping back and making sure if I understood the core concepts, right?
00:09:45.519 --> 00:09:58.559
What a DAG actually is, how tasks work, what are the operators, how operator works, how scheduling and dependencies tie all together, what's upstream, what's downstream.
00:09:58.799 --> 00:10:02.240
And once those fundamentals clicked, it was very easy.
00:10:02.320 --> 00:10:05.519
Like when you go through the docs, it was very easy.
00:10:05.840 --> 00:10:12.639
So after that, I was focused less on watching tutorials and practicing with real-world problems.
00:10:12.799 --> 00:10:24.480
I started practicing with real-world small things, let's say small ETL workload, make some work APIs, and just make a DAG for it and try to understand how things are.
00:10:24.639 --> 00:10:32.799
Then you discover things like XCOMs exist, like map tasks, and how do you find concurrency and all those different things?
00:10:32.960 --> 00:10:37.840
So I think curiosity should lead the learning.
00:10:38.799 --> 00:10:47.440
Since I was open for my curiosity, initial times it was a little hard, but then it was very fast to learn.
00:10:47.600 --> 00:10:55.360
So, resource-wise, I think mix of official talks, blogs, and there are like really good crash courses and stuff like that.
00:10:55.519 --> 00:11:04.960
So, when I learn something new, my equation or framework is that watch a crash course, find what's its use for, and understand what's happening here.
00:11:05.200 --> 00:11:12.480
Crash course can be like 20 minutes, one hour, but you'll get a capsule uh information of what I think it actually is.
00:11:12.639 --> 00:11:18.879
So you'll discover, for example, what Airflow has to offer, and you just know a mind map of things.
00:11:18.960 --> 00:11:19.279
Okay.
00:11:19.759 --> 00:11:22.799
Then you can start digging it a little bit later.
00:11:23.039 --> 00:11:25.440
Understand core concept directly from docs.
00:11:25.519 --> 00:11:27.200
So that's the best way to start.
00:11:27.360 --> 00:11:32.720
Then sometimes we need some little different perspective of thinking to understand things.
00:11:32.879 --> 00:11:34.799
That's how blogs help us.
00:11:35.039 --> 00:11:38.000
Then, best way to learn is by practicing.
00:11:38.240 --> 00:11:42.159
And for that, I really like the airflow in Docker containers.
00:11:42.320 --> 00:11:43.279
It was very easy.
00:11:43.360 --> 00:11:49.440
You could run a scalable Docker, a mini-scalable Docker inside your local machine and experiment.
00:11:49.600 --> 00:11:53.600
You didn't have to waste compute or resources and all those different things.
00:11:53.759 --> 00:11:58.639
So it was really easy to learn airflow moving on like that.
00:11:58.799 --> 00:12:02.799
And if you broke the Docker, you can just delete the Docker and start new.
00:12:03.039 --> 00:12:08.080
I digged into and broke Airflow a lot of times, and that's how I learned.
00:12:08.159 --> 00:12:11.840
And again, yeah, Chat GPT, Google search, and all everything helped.
00:12:12.480 --> 00:12:28.000
So a big aha moment for us, me like realizing Airflow is less about memorizing syntax and stuff like that, and more about thinking in terms of orchestration, how data moves, when it should move, and what to do when it fails.
00:12:28.159 --> 00:12:31.440
So that's you know, basic things.
00:12:33.759 --> 00:12:42.720
This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
00:12:42.879 --> 00:12:54.559
Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
00:12:54.799 --> 00:13:01.759
To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
00:13:20.720 --> 00:13:31.039
So just try to focus on what you actually need and to know that the best way, as you mentioned, is either to go on a crash course, but also just to go on the documentation.
00:13:31.279 --> 00:13:37.759
I think there is like a basic example that goes through how you write your first data pipeline, creating your first task, and so on.
00:13:37.840 --> 00:13:40.559
And that will be more than enough at least for the beginning.
00:13:40.720 --> 00:13:41.600
But this is super cool.
00:13:41.919 --> 00:13:46.559
I would love actually to know more about how did you set up your Airflow deployment?
00:13:46.720 --> 00:13:48.559
Because again, like you are a small team.
00:13:48.720 --> 00:13:52.639
What were the considerations when you decided to set up Airflow in production?
00:13:52.799 --> 00:13:54.000
What decisions did you take?
00:13:54.159 --> 00:13:57.200
Tell me more about your current infrastructure with Airflow.
00:13:57.360 --> 00:14:00.320
So, again, small team is the big point here.
00:14:00.480 --> 00:14:01.840
So, startup, right?
00:14:01.919 --> 00:14:08.320
So, startup mindset is like limited resources, and every infra choice you make is like a trade-off, right?
00:14:08.559 --> 00:14:17.519
You can have really fast DAG but high build, or you can have very slow DAG but convenient building and stuff like that.
00:14:17.759 --> 00:14:22.960
And at a startup infrastructure decisions really come down to being practical.
00:14:23.120 --> 00:14:25.279
You don't have unlimited resources, right?
00:14:25.440 --> 00:14:27.200
So every use has a trade-off.
00:14:27.440 --> 00:14:34.480
So with Airflow, the main thing for us was to find the balance between running things concurrently versus batching them, right?
00:14:34.639 --> 00:14:39.519
You can do just pull the data, and let's say for each row, you can run a map task.
00:14:39.759 --> 00:14:40.559
It's easy, right?
00:14:40.799 --> 00:14:41.840
You can orchestrate it.
00:14:42.080 --> 00:14:46.960
It's also maybe a good standard for some data pipelines and stuff like that.
00:14:47.120 --> 00:14:52.559
But again, more tasks means more complexity, more resources, and more things, right?
00:14:52.799 --> 00:15:04.720
So finding the balance between running things concurrently and batching them was really critical things I thought of when we made DACs and stuff like that.
00:15:04.960 --> 00:15:12.159
Because sure you can make everything super parallel and concurrent, but more tasks and bigger cloud width.
00:15:12.320 --> 00:15:17.440
So on the other hand, batching too much slows things down, but that's a use user experience.
00:15:17.679 --> 00:15:28.799
So we thought of strategies like okay, what comes in the user interfaces and user experience will do is more concurrently and stuff like that.
00:15:28.960 --> 00:15:36.000
But maybe for inventory forecasting and all those different problems, we'll batch it for more customers at once.
00:15:36.159 --> 00:15:49.440
So you can save a lot of resources running that it's not about everything should be in concurrent, or it's like when you when developers start, okay, everything should be fast, super optimized, super scalable, and stuff like that.
00:15:49.600 --> 00:16:00.399
But the latter is resources, build efficiency, are so my thing was like instead of over-engineering from the start, build simplest deployments that work.
00:16:00.480 --> 00:16:01.759
We are treated as we learn.
00:16:02.000 --> 00:16:10.320
So it's like questionnaire do we really need this pipeline to run every five minutes, or should we run this every week?
00:16:10.639 --> 00:16:13.759
So is our league enough or is monthly enough?
00:16:13.919 --> 00:16:23.600
So those kind of questions ask yourself, ask your team, and make trade-off tables and you and the decisions come like that.
00:16:23.840 --> 00:16:26.879
So we didn't have like fanciest deployment setup.
00:16:26.960 --> 00:16:34.480
So we had local Docker setup in everyone's machine, engineer's machine, so they can test it out on local first.
00:16:34.960 --> 00:16:45.279
Then we have this EC2 server where we have again Docker deployed, but again, it works with staging APIs and other it works with real-world data, right?
00:16:45.519 --> 00:16:47.919
Test that and concurrency and benchmark that.
00:16:48.000 --> 00:16:59.279
Then again, for production, we use Google Cloud Composer because we don't have time to every day and it takes a lot of productive time and stuff like that.
00:16:59.519 --> 00:17:04.880
In that way, we thought, okay, managed service helps in those kinds of things.
00:17:05.119 --> 00:17:10.160
And also manage service doesn't mean that your bill will be very big.
00:17:10.400 --> 00:17:15.680
Manage service means like you have to optimize for your data pipelines so that you can.
00:17:15.839 --> 00:17:23.039
So the main fear of managed pipelines is that managed services that bill might get big and stuff like that.
00:17:23.279 --> 00:17:28.160
But it's how you optimize and simplify your DAGs, that's where it matters, right?
00:17:28.480 --> 00:17:29.839
Thank you so much for your sharing.
00:17:30.079 --> 00:17:32.960
How did you come up with that infrastructure as a small team?
00:17:33.039 --> 00:17:35.519
I think that will be super useful for many listeners.
00:17:35.759 --> 00:17:39.440
But I would love to dive into one of your most interesting use cases.
00:17:39.599 --> 00:17:48.240
I don't know if it is the most interesting one, but it's definitely the one that everybody is talking about today in the engineering world, which is anything AI related.
00:17:48.400 --> 00:17:59.279
So, can you tell us about this project and how you architected the solution about orchestrating this pipeline for processing Amazon reviews using your LLM?
00:17:59.599 --> 00:18:02.240
The approach to this problem was very unique, right?
00:18:02.319 --> 00:18:09.279
So it's normally when startups I trade, we need AI, we need that, we need this, like that, and we just rest.
00:18:10.240 --> 00:18:12.319
But this use case was very unique.
00:18:12.480 --> 00:18:13.920
We had a lot of customers, right?
00:18:14.000 --> 00:18:19.599
Some are very small CPG brands, some are medium-sized brands, some are like very large companies.
00:18:19.839 --> 00:18:28.559
So we work with different kinds of startups and different kinds of businesses, and again, it's customers startup, right?
00:18:28.640 --> 00:18:32.240
So we are running on heavy data and stuff like that, right?
00:18:32.400 --> 00:18:34.799
So we had few enterprise customers.
00:18:34.960 --> 00:18:42.400
So we have these products, so that's dashboard products and stuff like that, and we also have enterprise side of things, right?
00:18:42.559 --> 00:18:50.400
So most of our DAGs are for like the actual product, so that's Xena 360 or different e-commerce.
00:18:50.480 --> 00:18:56.480
So you can connect Walmart, Amazon into our dashboard, and we pull in data insights and stuff like that, right?
00:18:56.640 --> 00:19:04.319
So that's our core providing, but we have this enterprise customers with custom and unique use cases, right?
00:19:04.480 --> 00:19:12.319
So again, one of our core ideas is like we solve problems what the brands face, right?
00:19:12.480 --> 00:19:18.079
So when one of our enterprise customers came to us and said, You're launching a new product in this market.
00:19:18.319 --> 00:19:27.759
We are not sure if we want to launch that product into that market, but we would love to know what the market looks like and should we go forward or not.
00:19:27.920 --> 00:19:31.839
So it was very simple use case and stuff like that when we started out.
00:19:32.000 --> 00:19:34.720
So we have a team of data analysts there.
00:19:34.799 --> 00:19:36.720
So we have you a couple of analysts there.
00:19:36.799 --> 00:19:41.440
So these analysts work with customers to find problems and fix them.
00:19:41.839 --> 00:19:53.279
So one of the core things in this providing was that understand the user behavior, understand the user market shopper preference, right?
00:19:53.680 --> 00:19:59.920
So immediately our first choice was uh Amazon reviews, Walmart reviews, and all those.
00:20:00.240 --> 00:20:02.400
We are working with the e-commerce company as well, right?
00:20:02.559 --> 00:20:08.880
So if a company was going to this category, what people think on the best-selling products, right?
00:20:09.200 --> 00:20:11.920
What do people think about specific competitors?
00:20:12.240 --> 00:20:15.039
So all these questions were to be answered, right?
00:20:15.279 --> 00:20:17.920
So we started pulling reviews manually.
00:20:18.079 --> 00:20:25.920
These data analysts are pulling manually, uh copy-pasted each and every review, and again it's text mining, right?
00:20:26.240 --> 00:20:34.480
If it was JSON, we could write some scripts and structure it out and run some plotly or plot some graphs and stuff.
00:20:34.559 --> 00:20:35.920
So you can understand the data.
00:20:36.240 --> 00:20:39.519
Text it was very unique thing.
00:20:39.759 --> 00:20:43.839
Like it to understand, you just need human to read and understand.
00:20:44.000 --> 00:20:51.839
But it's around the same time when Chat GPT came and the LLM boom came, and thought, okay, this is really interesting use case.
00:20:52.000 --> 00:20:57.599
Okay, people would love, and it's for this report, shopper preference report, market research.
00:20:57.759 --> 00:21:08.160
There is also demand for multiple enterprise customers ask in a time to for this, and that's how it drove us to build this solution.
00:21:08.319 --> 00:21:18.079
It was totally manual process, so we started to pull in public reviews through the web crawling with ethical crawlers, proxies, and stuff like that.
00:21:18.400 --> 00:21:29.200
And we had like a large amount of reviews per product, and we had to go through every review and find each of things, and it was interesting.
00:21:29.359 --> 00:21:39.920
Initially, we thought, okay, like post-reviews are four stars and five stars, negative than two stars, but you see a lot of negative stuff in five star reviews.
00:21:40.000 --> 00:21:43.200
I wish I could have less amount of sugar in this, right?
00:21:43.359 --> 00:21:46.240
I wish uh it had this feature, that feature.
00:21:46.559 --> 00:21:52.400
When we started out, okay, prompting with uh against a review and extracting stuff like that.
00:21:52.480 --> 00:22:04.480
So we extracted topics, we extracted themes, negative sentiments, positive sentiments, and we also extracted uh Lot of different why this review deserves four star, five star.
00:22:04.880 --> 00:22:08.079
So we invented new two-star rating.
00:22:08.240 --> 00:22:19.519
So let's say if someone started four three-star, but the review itself has much more good sentiments than we say that okay, this reserve to be 4.5 stars.
00:22:19.920 --> 00:22:26.960
And when you analyze reviews at large, we started to see patterns, right?
00:22:27.119 --> 00:22:37.519
Most of these people said this, and most of this didn't say this, and some people said this, and the importance of and related to this product was this.
00:22:37.839 --> 00:22:41.759
So LLMs was good for this kind of use case.
00:22:41.839 --> 00:22:47.519
It again text, it's context, and it can relate across memories and stuff like that.
00:22:48.000 --> 00:23:00.079
So that got us into like where this pipeline where we pull in each review and run against a standard prompt and get a JSON output of this analysis.
00:23:00.240 --> 00:23:02.880
It included positive sentiments, negative sentiments.
00:23:03.039 --> 00:23:06.559
Again, it's what it was not text, again, it was nested JSON.
00:23:06.799 --> 00:23:11.039
So why it was interesting was like again, that's how engineering comes into this type.
00:23:11.119 --> 00:23:14.799
We just don't write pipelines, we just also engineer it, right?
00:23:15.039 --> 00:23:22.640
Then that JSON structure, we directly put it into a local Postgres instance and we started to write queries.
00:23:22.799 --> 00:23:23.839
And then it made sense.
00:23:24.000 --> 00:23:34.319
So this totally unstructured data, we could structure it in such a way that we could pull out four post-each sentiments, we could group positive sentiments into one.
00:23:34.640 --> 00:23:43.279
And as we worked on this, it was for like initially we started out as like a project for data analysts to make their life easier.
00:23:43.599 --> 00:23:59.279
But through this exploration, we understood that there is a demand for on-demand review analysis and exploration and find positive sentiments, negative sentiments, and explore through the review data set and send recommendations to the customers.
00:23:59.440 --> 00:24:01.359
So we thought of to make it into a product.
00:24:01.519 --> 00:24:10.720
Why not as simple as putting in a link and we'll pull the reviews, we'll analyze it for you, and you can explore positive sentiments, negative sentiments.
00:24:10.880 --> 00:24:13.119
We'll also even suggest improvements.
00:24:13.279 --> 00:24:17.200
So one of our customers had this monitor, gaming monitor.
00:24:17.359 --> 00:24:26.400
So this gaming monitor, a lot of people were saying that, okay, I wish it would have HDR feature in this gaming monitor, right?
00:24:26.799 --> 00:24:30.000
So that was like a most requested feature.
00:24:30.240 --> 00:24:37.039
But if you just look at the Amazon page and try to read the reviews, you might not see at first glance.
00:24:37.279 --> 00:24:43.440
But when we put it in our pipeline, it was on top of the requests, right?
00:24:43.920 --> 00:24:47.039
People are saying this, people are saying that.
00:24:50.640 --> 00:24:55.119
So it drove decisions in companies with large reviews.
00:24:55.279 --> 00:25:00.559
So it also helped next iteration of the product, what to include, what not include.
00:25:00.640 --> 00:25:06.480
And this is again without interacting too directly to customer, it's there, the data is there.
00:25:06.640 --> 00:25:12.799
We just help them to get, and this was very successful at this level in the enterprise level.
00:25:12.960 --> 00:25:21.359
So we we thought out okay, we'll give it to like medium customers who have a little bit of knowledge to how to use dashboards and stuff like that.
00:25:21.519 --> 00:25:31.039
And so if they connect their Amazon account, we can pull the inventory, and this DAG will go through each product and it runs automatically.
00:25:31.279 --> 00:25:37.759
And that Airflow Rust API is helped to show ITAs and stuff like that.
00:25:38.000 --> 00:25:54.000
And essentially, how we did this DAG was pull in reviews, run the analysis on top of LLMs, then we made it into a structured format into JSON, and this JSON was converted into more structured table, and we stored it in again.
00:25:54.079 --> 00:26:05.599
We also took some decisions like store the actual responses into a parquet format so that we can save question storage and run that DB on that and run analysis on that.
00:26:05.680 --> 00:26:08.559
And it was a very interesting to work on this, right?
00:26:08.720 --> 00:26:12.000
So, in the end, airflow wasn't just scheduling, right?
00:26:12.400 --> 00:26:21.359
More of the control plane that lets us scale from manual analyst for flow to an automated system that could process millions of reviews reliably.
00:26:21.680 --> 00:26:26.000
It was really a unique set of challenges and stuff like that.
00:26:26.240 --> 00:26:26.799
That's awesome.
00:26:27.039 --> 00:26:34.000
I guess like what I love with your use case is that you prove that you can definitely use Airflow as a way to orchestrate LLMs.
00:26:34.240 --> 00:26:39.200
And like sometimes you hear that Airflow is legacy, blah blah blah, but that's not true.
00:26:39.279 --> 00:26:51.839
Like the reality is that today you can still use Airflow for many use cases, and LLM is definitely one of them, and that you just show that you can do it, even as a small team, you can definitely orchestrate LLMs using Airflow.
00:26:52.240 --> 00:26:54.960
Thank you so much for sharing that with us, Nassim.
00:26:55.119 --> 00:26:56.880
I would like to ask you one last question.
00:26:57.039 --> 00:27:02.000
What is the best way for folks to get in touch with you if they want to learn more about your use case?
00:27:02.240 --> 00:27:03.599
Because we just touched it.
00:27:03.680 --> 00:27:06.480
Honestly, there are so much things to talk about.
00:27:06.799 --> 00:27:11.599
Uh, best way is email and LinkedIn is very easy to reach me there.
00:27:11.839 --> 00:27:18.400
And yeah, I'm very uh eager to help everyone in this case, and I loved Airflow so much.
00:27:18.480 --> 00:27:19.920
I started to contribute, right?
00:27:20.000 --> 00:27:23.119
So get getting active into community is also really good.
00:27:23.200 --> 00:27:27.839
So I'm there in the Airflow Slack, in the dev channels and stuff like that.
00:27:28.000 --> 00:27:29.839
So I'm available there too.
00:27:30.000 --> 00:27:32.559
But yeah, I'm very happy to work through this.
00:27:32.720 --> 00:27:33.920
Hopefully, I can learn more.
00:27:34.000 --> 00:27:36.880
So it's knowledge sharing, knowledge transfer.
00:27:37.039 --> 00:27:43.920
When that exists, this community grows, and that's how we as a community we need this in this airflow.
00:27:44.000 --> 00:27:45.759
Okay, we can have this feature in Airflow.
00:27:45.839 --> 00:27:46.559
That's how you grow.
00:27:46.640 --> 00:27:48.559
So it's really interesting.
00:27:48.799 --> 00:27:52.400
Any small contributions to Airflow will be really great.
00:27:52.559 --> 00:27:57.279
So I would love some of the reason I start there, the contributions Airflow.
00:27:57.440 --> 00:28:05.119
And my code contributions are not a really big, very small thing, but it makes me happy that I can all contribute back to the project.
00:28:05.759 --> 00:28:06.319
That's awesome.
00:28:06.480 --> 00:28:07.440
Thank you so much, Nassim.
00:28:07.680 --> 00:28:10.640
Again, thank you so much for joining, and uh pleasure to have you.
00:28:10.799 --> 00:28:11.359
See you soon.
00:28:11.440 --> 00:28:12.160
Take care.
00:28:14.400 --> 00:28:20.880
Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
00:28:21.039 --> 00:28:24.720
You'll find links in the show notes to any resources mentioned in today's show.
00:28:24.880 --> 00:28:31.519
If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.
00:00:00.160 --> 00:00:03.680
I think curiosity should lead the learning, right?
00:00:04.000 --> 00:00:12.320
So some since I was open for my curiosity, initial times it was a little hard, but then it was very fast to learn.
00:00:12.480 --> 00:00:19.120
So resource-wise, I think mix of official talks, blogs, and there are like really good crash courses and stuff like that.
00:00:19.280 --> 00:00:29.440
So when I learn something new, my equation or framework is that like watch a crash course, find what it's used for, and understand what's happening here.
00:00:34.479 --> 00:00:42.719
You're listening to the Data Flowcast, where we unlock the secrets of mastering airflow for data engineering and AI.
00:00:44.880 --> 00:00:47.840
Here, Airflow community, we've got exciting news.
00:00:48.000 --> 00:00:50.880
The annual Apache Airflow survey is live.
00:00:51.039 --> 00:00:58.640
In just seven minutes, you can help shape the future of Airflow and the free Airflow 3 Fundamentals or Demethrine certification.
00:01:05.280 --> 00:01:06.319
Don't miss out.
00:01:06.480 --> 00:01:10.319
The survey link is in the show notes and closes November 21st.
00:01:12.719 --> 00:01:13.599
Hello everyone.
00:01:13.760 --> 00:01:15.280
Welcome to the Dela Flowcast.
00:01:15.359 --> 00:01:24.159
I'm your host, Mark Lumberti, and I'm super excited because today I'm joined by Nassim Shah, head of engineering at Xena Intelligence.
00:01:24.400 --> 00:01:26.400
I mean, the first question is how are you doing?
00:01:26.719 --> 00:01:27.439
Hi, it's good.
00:01:27.519 --> 00:01:29.040
I'm really excited for this.
00:01:29.120 --> 00:01:30.239
It was a really great day.
00:01:30.319 --> 00:01:34.560
It was raining and all, and it was such a beautiful day to start on.
00:01:35.040 --> 00:01:41.280
And you know how things go and had a great day at work, did a lot of problem solving and all.
00:01:41.439 --> 00:01:42.799
It's exciting stuff.
00:01:42.959 --> 00:01:44.879
So yeah, never bothered.
00:01:45.120 --> 00:01:45.680
That's awesome.
00:01:45.840 --> 00:02:02.799
Well, I'm really glad to be here with you today because we're actually gonna talk about a lot of exciting stuff, more specifically, how you and obviously your team also transformed from Chrome jobs to sophisticated LLM-powered review analysis using obviously AI.
00:02:02.959 --> 00:02:10.560
And so you will discover also how Nassim and the team is able to process millions of Amazon reviews.
00:02:10.719 --> 00:02:15.520
So this is super exciting, and obviously, everything is orchestrated using Airflow.
00:02:16.000 --> 00:02:19.840
So I would like to start with your background.
00:02:20.080 --> 00:02:26.479
Okay, so can you tell us about your journey into data engineering and what led you to discover Airflow?
00:02:26.800 --> 00:02:33.039
And also what were some of the data pipeline challenges that you were facing before adopting Airflow?
00:02:33.280 --> 00:02:44.000
So I come from a software engineering background, not really a data engineer as per se, when situations and things got into data engineering related works also.
00:02:44.159 --> 00:02:46.960
So I actually come from more of like a software engineering background.
00:02:47.039 --> 00:02:50.000
I didn't know much about the world of data engineering at the start.
00:02:50.159 --> 00:02:53.919
I joined intelligence straight out of college as a first engineering hire.
00:02:54.240 --> 00:03:06.639
So basically, as a founding engineer, so you have to gather things around and learn new things and the requirements from the stakeholders, from the founders, the CEO, director, reporting to them, directors.
00:03:06.800 --> 00:03:09.120
Like you have to find things on your own.
00:03:09.199 --> 00:03:18.319
And that gave me a lot of exploring and gave me a lot of freedom to explore and find things on my own and find it my own way.
00:03:18.479 --> 00:03:22.080
So that was really exciting for me and straight off a college.
00:03:22.159 --> 00:03:25.280
It's something that you don't get when you join an MNC.
00:03:25.360 --> 00:03:28.479
The first that's how I come into this picture.
00:03:28.719 --> 00:03:39.039
So my role earlier was to build MVPs and get product out of the ground, which ended up helping us raising funds and join accelerators like Techstars.
00:03:39.840 --> 00:03:41.280
A little bit about the company.
00:03:41.360 --> 00:03:44.560
So Xena is an AI-powered e-commerce ground platform.
00:03:44.639 --> 00:03:47.520
So everything we do is really data heavy.
00:03:47.680 --> 00:04:02.159
So we would pull in data from marketplaces like Amazon, Walmart, Shopify to generate insights for things like ad optimization, inventory planning, inventory focused, team market research, customer behavior, and all those stuff, right?
00:04:02.400 --> 00:04:06.000
So in the beginning, our pipelines were just like ground scripts, right?
00:04:06.080 --> 00:04:13.280
So honestly, it worked fine for a few customers, especially if we were like pre-revenue beta testing and all those stuff, right?
00:04:13.520 --> 00:04:18.879
So initially, my motto is start simple, time update.
00:04:18.959 --> 00:04:24.480
So you don't need to complicate into the first and waste our time and productivity on that, right?
00:04:24.720 --> 00:04:28.240
So my first approach was like start out simple.
00:04:28.319 --> 00:04:35.519
So it's few customers, handful of API calls and downloading report documents and processing it in Python.
00:04:35.600 --> 00:04:38.800
And it was very simple and it was easy, right?
00:04:38.879 --> 00:04:40.959
And we didn't waste much time on that.
00:04:41.199 --> 00:04:51.680
That's why we could ship more features when so when we started to grow and move to product market fit, those crones started to become like a nightmare.
00:04:51.839 --> 00:05:01.040
Scaling out database was easy where backends are easy, you can just containerize and it's fairly easy to scale back end APIs and stuff like that.
00:05:01.199 --> 00:05:09.279
But when it comes to the data pipelines, it was really not that easy and was a very big challenge as a fresher when I started out.
00:05:09.920 --> 00:05:16.720
We had cron jobs failing randomly, no clear visibility, which into let most data, no clear visibility.
00:05:17.199 --> 00:05:26.959
I often find myself logging in through heaps of logs at later, uh trying to find the needle in the haystack kind of situation here.
00:05:27.120 --> 00:05:30.079
So it was very hard to debug and stuff like that.
00:05:30.399 --> 00:05:34.639
So we then started to try a small upgrade.
00:05:34.800 --> 00:05:36.240
We tried small upgrade.
00:05:36.480 --> 00:05:41.120
So that's how I re-architected with again baby steps, right?
00:05:41.279 --> 00:05:42.959
SQS and easy to set up.
00:05:43.040 --> 00:05:47.360
I didn't want to complicate it very fast, but slowly.
00:05:47.519 --> 00:05:53.519
So it was interesting because we had SQS queues, so it's like Kafka queues and stuff like that.
00:05:53.680 --> 00:05:58.160
Just the message comes in and consumer gets it and process the data.
00:05:58.240 --> 00:05:59.680
And it was fairly simple.
00:05:59.839 --> 00:06:02.800
So we had one queue for one report type.
00:06:03.040 --> 00:06:06.879
And for customer, it was like five queues, five consumers.
00:06:07.199 --> 00:06:08.800
It was okay for that time.
00:06:08.879 --> 00:06:14.800
And when we had like sub-20-30 customers, it was okay for us like to go through that.
00:06:15.040 --> 00:06:21.439
But when we started grow and customers flooding in and stuff like that, it was very hard again.
00:06:21.519 --> 00:06:24.560
Like which customer data pipeline go down?
00:06:24.720 --> 00:06:26.480
Why this isn't it?
00:06:26.800 --> 00:06:32.560
Customers start comes reach out, support, and things were not uh looking good.
00:06:32.720 --> 00:06:40.480
And AWS SQS and going to the dead letter queue, myself and our team, and trying to understand.
00:06:40.639 --> 00:06:46.319
And sometimes the code works to customers, it doesn't work for others.
00:06:46.480 --> 00:06:53.120
So the lack of visibility was slowing down the scalability and our health of the platform itself.
00:06:53.279 --> 00:06:57.120
So again, it's not just the features, but we have to deliver it right.
00:06:57.839 --> 00:07:05.120
So that lack of visibility, lack of scalability, and it was not developer-friendly, right?
00:07:05.279 --> 00:07:10.639
So it cost us productivity time and all those different aspects of what we do.
00:07:10.720 --> 00:07:21.839
And we were like a very small team, like we are three engineers just managing all these data pipelines, all these features, and we have to make the stakeholders happy, and we just have to grind it.
00:07:22.000 --> 00:07:24.959
So that's started to research about data engineering.
00:07:25.040 --> 00:07:36.000
That's how we get into data engineering, and that's how I understood all there's a world like this, and there is lots of tools, and that's how I ended up in the airflow, right?
00:07:36.160 --> 00:07:39.199
Okay, it was perfect for us when you started.
00:07:39.439 --> 00:07:40.480
Yeah, wow.
00:07:40.560 --> 00:07:49.120
So I think like many teams will listen to that podcast and will realize that they are getting the same issues that you got in the past before Airflow.
00:07:49.279 --> 00:07:52.480
And some people always wonder, hey, should I use Airflow?
00:07:52.720 --> 00:08:06.160
If you listen, NASIM, you truly understand that if every day you are struggling with logs, with pipelines that you cannot properly monitor, that you cannot properly debug, and so on, obviously, Airflow will be a huge improvement for you.
00:08:06.319 --> 00:08:16.319
And honestly, at some point, when you feel that you are overwhelmed with so many issues and so on, and that your data pipelines are not reliable as they should be, I mean it's time to go with an orchestrator.
00:08:16.720 --> 00:08:22.399
And you know, Nassim, it's interesting because you said that you are a small team, right?
00:08:23.040 --> 00:08:33.679
And many people actually, I wouldn't say many, but some people believe that you need to be a huge team just to set up and run Airflow in production, which is not true.
00:08:33.840 --> 00:08:40.240
But tell me more about your experience with Airflow and more specifically, what was your experience learning Airflow?
00:08:40.320 --> 00:08:48.240
And do you have any particular resources, approaches, or things like that truly helped you in your journey with Airflow and as a newcomer to Airflow?
00:08:48.480 --> 00:08:51.919
Yeah, learning Airflow definitely felt intimidating at first.
00:08:52.159 --> 00:08:58.879
It helped a lot when I came through the software engineering lens because I understood what Docker is.
00:08:58.960 --> 00:09:00.240
I knew all these things, right?
00:09:00.399 --> 00:09:01.440
What are the containers?
00:09:01.600 --> 00:09:09.519
Why Airflow has different containers, and so it clicked me easily since I'm looking through an engineer's thing.
00:09:09.759 --> 00:09:20.000
But again, for a starter, it might be a little hard, even though I knew that it was a little intimidating at first, because what is a lot of things at first?
00:09:20.159 --> 00:09:22.080
As time moved, it was easier.
00:09:22.240 --> 00:09:23.279
So, yeah, definitely.
00:09:23.360 --> 00:09:28.799
I think I went through a lot of engineers go through what people call tutorial health, right?
00:09:28.879 --> 00:09:39.600
So I was watching crash courses, jumping into dogs, trying to piece things together, and it wasn't just clicking right away, but slowly it was the progression was very slow, right?
00:09:39.759 --> 00:09:45.279
What really helped me was stepping back and making sure if I understood the core concepts, right?
00:09:45.519 --> 00:09:58.559
What a DAG actually is, how tasks work, what are the operators, how operator works, how scheduling and dependencies tie all together, what's upstream, what's downstream.
00:09:58.799 --> 00:10:02.240
And once those fundamentals clicked, it was very easy.
00:10:02.320 --> 00:10:05.519
Like when you go through the docs, it was very easy.
00:10:05.840 --> 00:10:12.639
So after that, I was focused less on watching tutorials and practicing with real-world problems.
00:10:12.799 --> 00:10:24.480
I started practicing with real-world small things, let's say small ETL workload, make some work APIs, and just make a DAG for it and try to understand how things are.
00:10:24.639 --> 00:10:32.799
Then you discover things like XCOMs exist, like map tasks, and how do you find concurrency and all those different things?
00:10:32.960 --> 00:10:37.840
So I think curiosity should lead the learning.
00:10:38.799 --> 00:10:47.440
Since I was open for my curiosity, initial times it was a little hard, but then it was very fast to learn.
00:10:47.600 --> 00:10:55.360
So, resource-wise, I think mix of official talks, blogs, and there are like really good crash courses and stuff like that.
00:10:55.519 --> 00:11:04.960
So, when I learn something new, my equation or framework is that watch a crash course, find what's its use for, and understand what's happening here.
00:11:05.200 --> 00:11:12.480
Crash course can be like 20 minutes, one hour, but you'll get a capsule uh information of what I think it actually is.
00:11:12.639 --> 00:11:18.879
So you'll discover, for example, what Airflow has to offer, and you just know a mind map of things.
00:11:18.960 --> 00:11:19.279
Okay.
00:11:19.759 --> 00:11:22.799
Then you can start digging it a little bit later.
00:11:23.039 --> 00:11:25.440
Understand core concept directly from docs.
00:11:25.519 --> 00:11:27.200
So that's the best way to start.
00:11:27.360 --> 00:11:32.720
Then sometimes we need some little different perspective of thinking to understand things.
00:11:32.879 --> 00:11:34.799
That's how blogs help us.
00:11:35.039 --> 00:11:38.000
Then, best way to learn is by practicing.
00:11:38.240 --> 00:11:42.159
And for that, I really like the airflow in Docker containers.
00:11:42.320 --> 00:11:43.279
It was very easy.
00:11:43.360 --> 00:11:49.440
You could run a scalable Docker, a mini-scalable Docker inside your local machine and experiment.
00:11:49.600 --> 00:11:53.600
You didn't have to waste compute or resources and all those different things.
00:11:53.759 --> 00:11:58.639
So it was really easy to learn airflow moving on like that.
00:11:58.799 --> 00:12:02.799
And if you broke the Docker, you can just delete the Docker and start new.
00:12:03.039 --> 00:12:08.080
I digged into and broke Airflow a lot of times, and that's how I learned.
00:12:08.159 --> 00:12:11.840
And again, yeah, Chat GPT, Google search, and all everything helped.
00:12:12.480 --> 00:12:28.000
So a big aha moment for us, me like realizing Airflow is less about memorizing syntax and stuff like that, and more about thinking in terms of orchestration, how data moves, when it should move, and what to do when it fails.
00:12:28.159 --> 00:12:31.440
So that's you know, basic things.
00:12:33.759 --> 00:12:42.720
This episode is brought to you by Astronomer, creators of Astro, the leading fully managed airflow service designed for the modern data stack.
00:12:42.879 --> 00:12:54.559
Whether you're orchestrating large language models, implementing a data mesh architecture, or streamlining complex workflows, Astro empowers your team to focus on innovation, not infrastructure.
00:12:54.799 --> 00:13:01.759
To see how you can maximize Airflow's potential within your data ecosystem, visit www.astonomer.io.
00:13:20.720 --> 00:13:31.039
So just try to focus on what you actually need and to know that the best way, as you mentioned, is either to go on a crash course, but also just to go on the documentation.
00:13:31.279 --> 00:13:37.759
I think there is like a basic example that goes through how you write your first data pipeline, creating your first task, and so on.
00:13:37.840 --> 00:13:40.559
And that will be more than enough at least for the beginning.
00:13:40.720 --> 00:13:41.600
But this is super cool.
00:13:41.919 --> 00:13:46.559
I would love actually to know more about how did you set up your Airflow deployment?
00:13:46.720 --> 00:13:48.559
Because again, like you are a small team.
00:13:48.720 --> 00:13:52.639
What were the considerations when you decided to set up Airflow in production?
00:13:52.799 --> 00:13:54.000
What decisions did you take?
00:13:54.159 --> 00:13:57.200
Tell me more about your current infrastructure with Airflow.
00:13:57.360 --> 00:14:00.320
So, again, small team is the big point here.
00:14:00.480 --> 00:14:01.840
So, startup, right?
00:14:01.919 --> 00:14:08.320
So, startup mindset is like limited resources, and every infra choice you make is like a trade-off, right?
00:14:08.559 --> 00:14:17.519
You can have really fast DAG but high build, or you can have very slow DAG but convenient building and stuff like that.
00:14:17.759 --> 00:14:22.960
And at a startup infrastructure decisions really come down to being practical.
00:14:23.120 --> 00:14:25.279
You don't have unlimited resources, right?
00:14:25.440 --> 00:14:27.200
So every use has a trade-off.
00:14:27.440 --> 00:14:34.480
So with Airflow, the main thing for us was to find the balance between running things concurrently versus batching them, right?
00:14:34.639 --> 00:14:39.519
You can do just pull the data, and let's say for each row, you can run a map task.
00:14:39.759 --> 00:14:40.559
It's easy, right?
00:14:40.799 --> 00:14:41.840
You can orchestrate it.
00:14:42.080 --> 00:14:46.960
It's also maybe a good standard for some data pipelines and stuff like that.
00:14:47.120 --> 00:14:52.559
But again, more tasks means more complexity, more resources, and more things, right?
00:14:52.799 --> 00:15:04.720
So finding the balance between running things concurrently and batching them was really critical things I thought of when we made DACs and stuff like that.
00:15:04.960 --> 00:15:12.159
Because sure you can make everything super parallel and concurrent, but more tasks and bigger cloud width.
00:15:12.320 --> 00:15:17.440
So on the other hand, batching too much slows things down, but that's a use user experience.
00:15:17.679 --> 00:15:28.799
So we thought of strategies like okay, what comes in the user interfaces and user experience will do is more concurrently and stuff like that.
00:15:28.960 --> 00:15:36.000
But maybe for inventory forecasting and all those different problems, we'll batch it for more customers at once.
00:15:36.159 --> 00:15:49.440
So you can save a lot of resources running that it's not about everything should be in concurrent, or it's like when you when developers start, okay, everything should be fast, super optimized, super scalable, and stuff like that.
00:15:49.600 --> 00:16:00.399
But the latter is resources, build efficiency, are so my thing was like instead of over-engineering from the start, build simplest deployments that work.
00:16:00.480 --> 00:16:01.759
We are treated as we learn.
00:16:02.000 --> 00:16:10.320
So it's like questionnaire do we really need this pipeline to run every five minutes, or should we run this every week?
00:16:10.639 --> 00:16:13.759
So is our league enough or is monthly enough?
00:16:13.919 --> 00:16:23.600
So those kind of questions ask yourself, ask your team, and make trade-off tables and you and the decisions come like that.
00:16:23.840 --> 00:16:26.879
So we didn't have like fanciest deployment setup.
00:16:26.960 --> 00:16:34.480
So we had local Docker setup in everyone's machine, engineer's machine, so they can test it out on local first.
00:16:34.960 --> 00:16:45.279
Then we have this EC2 server where we have again Docker deployed, but again, it works with staging APIs and other it works with real-world data, right?
00:16:45.519 --> 00:16:47.919
Test that and concurrency and benchmark that.
00:16:48.000 --> 00:16:59.279
Then again, for production, we use Google Cloud Composer because we don't have time to every day and it takes a lot of productive time and stuff like that.
00:16:59.519 --> 00:17:04.880
In that way, we thought, okay, managed service helps in those kinds of things.
00:17:05.119 --> 00:17:10.160
And also manage service doesn't mean that your bill will be very big.
00:17:10.400 --> 00:17:15.680
Manage service means like you have to optimize for your data pipelines so that you can.
00:17:15.839 --> 00:17:23.039
So the main fear of managed pipelines is that managed services that bill might get big and stuff like that.
00:17:23.279 --> 00:17:28.160
But it's how you optimize and simplify your DAGs, that's where it matters, right?
00:17:28.480 --> 00:17:29.839
Thank you so much for your sharing.
00:17:30.079 --> 00:17:32.960
How did you come up with that infrastructure as a small team?
00:17:33.039 --> 00:17:35.519
I think that will be super useful for many listeners.
00:17:35.759 --> 00:17:39.440
But I would love to dive into one of your most interesting use cases.
00:17:39.599 --> 00:17:48.240
I don't know if it is the most interesting one, but it's definitely the one that everybody is talking about today in the engineering world, which is anything AI related.
00:17:48.400 --> 00:17:59.279
So, can you tell us about this project and how you architected the solution about orchestrating this pipeline for processing Amazon reviews using your LLM?
00:17:59.599 --> 00:18:02.240
The approach to this problem was very unique, right?
00:18:02.319 --> 00:18:09.279
So it's normally when startups I trade, we need AI, we need that, we need this, like that, and we just rest.
00:18:10.240 --> 00:18:12.319
But this use case was very unique.
00:18:12.480 --> 00:18:13.920
We had a lot of customers, right?
00:18:14.000 --> 00:18:19.599
Some are very small CPG brands, some are medium-sized brands, some are like very large companies.
00:18:19.839 --> 00:18:28.559
So we work with different kinds of startups and different kinds of businesses, and again, it's customers startup, right?
00:18:28.640 --> 00:18:32.240
So we are running on heavy data and stuff like that, right?
00:18:32.400 --> 00:18:34.799
So we had few enterprise customers.
00:18:34.960 --> 00:18:42.400
So we have these products, so that's dashboard products and stuff like that, and we also have enterprise side of things, right?
00:18:42.559 --> 00:18:50.400
So most of our DAGs are for like the actual product, so that's Xena 360 or different e-commerce.
00:18:50.480 --> 00:18:56.480
So you can connect Walmart, Amazon into our dashboard, and we pull in data insights and stuff like that, right?
00:18:56.640 --> 00:19:04.319
So that's our core providing, but we have this enterprise customers with custom and unique use cases, right?
00:19:04.480 --> 00:19:12.319
So again, one of our core ideas is like we solve problems what the brands face, right?
00:19:12.480 --> 00:19:18.079
So when one of our enterprise customers came to us and said, You're launching a new product in this market.
00:19:18.319 --> 00:19:27.759
We are not sure if we want to launch that product into that market, but we would love to know what the market looks like and should we go forward or not.
00:19:27.920 --> 00:19:31.839
So it was very simple use case and stuff like that when we started out.
00:19:32.000 --> 00:19:34.720
So we have a team of data analysts there.
00:19:34.799 --> 00:19:36.720
So we have you a couple of analysts there.
00:19:36.799 --> 00:19:41.440
So these analysts work with customers to find problems and fix them.
00:19:41.839 --> 00:19:53.279
So one of the core things in this providing was that understand the user behavior, understand the user market shopper preference, right?
00:19:53.680 --> 00:19:59.920
So immediately our first choice was uh Amazon reviews, Walmart reviews, and all those.
00:20:00.240 --> 00:20:02.400
We are working with the e-commerce company as well, right?
00:20:02.559 --> 00:20:08.880
So if a company was going to this category, what people think on the best-selling products, right?
00:20:09.200 --> 00:20:11.920
What do people think about specific competitors?
00:20:12.240 --> 00:20:15.039
So all these questions were to be answered, right?
00:20:15.279 --> 00:20:17.920
So we started pulling reviews manually.
00:20:18.079 --> 00:20:25.920
These data analysts are pulling manually, uh copy-pasted each and every review, and again it's text mining, right?
00:20:26.240 --> 00:20:34.480
If it was JSON, we could write some scripts and structure it out and run some plotly or plot some graphs and stuff.
00:20:34.559 --> 00:20:35.920
So you can understand the data.
00:20:36.240 --> 00:20:39.519
Text it was very unique thing.
00:20:39.759 --> 00:20:43.839
Like it to understand, you just need human to read and understand.
00:20:44.000 --> 00:20:51.839
But it's around the same time when Chat GPT came and the LLM boom came, and thought, okay, this is really interesting use case.
00:20:52.000 --> 00:20:57.599
Okay, people would love, and it's for this report, shopper preference report, market research.
00:20:57.759 --> 00:21:08.160
There is also demand for multiple enterprise customers ask in a time to for this, and that's how it drove us to build this solution.
00:21:08.319 --> 00:21:18.079
It was totally manual process, so we started to pull in public reviews through the web crawling with ethical crawlers, proxies, and stuff like that.
00:21:18.400 --> 00:21:29.200
And we had like a large amount of reviews per product, and we had to go through every review and find each of things, and it was interesting.
00:21:29.359 --> 00:21:39.920
Initially, we thought, okay, like post-reviews are four stars and five stars, negative than two stars, but you see a lot of negative stuff in five star reviews.
00:21:40.000 --> 00:21:43.200
I wish I could have less amount of sugar in this, right?
00:21:43.359 --> 00:21:46.240
I wish uh it had this feature, that feature.
00:21:46.559 --> 00:21:52.400
When we started out, okay, prompting with uh against a review and extracting stuff like that.
00:21:52.480 --> 00:22:04.480
So we extracted topics, we extracted themes, negative sentiments, positive sentiments, and we also extracted uh Lot of different why this review deserves four star, five star.
00:22:04.880 --> 00:22:08.079
So we invented new two-star rating.
00:22:08.240 --> 00:22:19.519
So let's say if someone started four three-star, but the review itself has much more good sentiments than we say that okay, this reserve to be 4.5 stars.
00:22:19.920 --> 00:22:26.960
And when you analyze reviews at large, we started to see patterns, right?
00:22:27.119 --> 00:22:37.519
Most of these people said this, and most of this didn't say this, and some people said this, and the importance of and related to this product was this.
00:22:37.839 --> 00:22:41.759
So LLMs was good for this kind of use case.
00:22:41.839 --> 00:22:47.519
It again text, it's context, and it can relate across memories and stuff like that.
00:22:48.000 --> 00:23:00.079
So that got us into like where this pipeline where we pull in each review and run against a standard prompt and get a JSON output of this analysis.
00:23:00.240 --> 00:23:02.880
It included positive sentiments, negative sentiments.
00:23:03.039 --> 00:23:06.559
Again, it's what it was not text, again, it was nested JSON.
00:23:06.799 --> 00:23:11.039
So why it was interesting was like again, that's how engineering comes into this type.
00:23:11.119 --> 00:23:14.799
We just don't write pipelines, we just also engineer it, right?
00:23:15.039 --> 00:23:22.640
Then that JSON structure, we directly put it into a local Postgres instance and we started to write queries.
00:23:22.799 --> 00:23:23.839
And then it made sense.
00:23:24.000 --> 00:23:34.319
So this totally unstructured data, we could structure it in such a way that we could pull out four post-each sentiments, we could group positive sentiments into one.
00:23:34.640 --> 00:23:43.279
And as we worked on this, it was for like initially we started out as like a project for data analysts to make their life easier.
00:23:43.599 --> 00:23:59.279
But through this exploration, we understood that there is a demand for on-demand review analysis and exploration and find positive sentiments, negative sentiments, and explore through the review data set and send recommendations to the customers.
00:23:59.440 --> 00:24:01.359
So we thought of to make it into a product.
00:24:01.519 --> 00:24:10.720
Why not as simple as putting in a link and we'll pull the reviews, we'll analyze it for you, and you can explore positive sentiments, negative sentiments.
00:24:10.880 --> 00:24:13.119
We'll also even suggest improvements.
00:24:13.279 --> 00:24:17.200
So one of our customers had this monitor, gaming monitor.
00:24:17.359 --> 00:24:26.400
So this gaming monitor, a lot of people were saying that, okay, I wish it would have HDR feature in this gaming monitor, right?
00:24:26.799 --> 00:24:30.000
So that was like a most requested feature.
00:24:30.240 --> 00:24:37.039
But if you just look at the Amazon page and try to read the reviews, you might not see at first glance.
00:24:37.279 --> 00:24:43.440
But when we put it in our pipeline, it was on top of the requests, right?
00:24:43.920 --> 00:24:47.039
People are saying this, people are saying that.
00:24:50.640 --> 00:24:55.119
So it drove decisions in companies with large reviews.
00:24:55.279 --> 00:25:00.559
So it also helped next iteration of the product, what to include, what not include.
00:25:00.640 --> 00:25:06.480
And this is again without interacting too directly to customer, it's there, the data is there.
00:25:06.640 --> 00:25:12.799
We just help them to get, and this was very successful at this level in the enterprise level.
00:25:12.960 --> 00:25:21.359
So we we thought out okay, we'll give it to like medium customers who have a little bit of knowledge to how to use dashboards and stuff like that.
00:25:21.519 --> 00:25:31.039
And so if they connect their Amazon account, we can pull the inventory, and this DAG will go through each product and it runs automatically.
00:25:31.279 --> 00:25:37.759
And that Airflow Rust API is helped to show ITAs and stuff like that.
00:25:38.000 --> 00:25:54.000
And essentially, how we did this DAG was pull in reviews, run the analysis on top of LLMs, then we made it into a structured format into JSON, and this JSON was converted into more structured table, and we stored it in again.
00:25:54.079 --> 00:26:05.599
We also took some decisions like store the actual responses into a parquet format so that we can save question storage and run that DB on that and run analysis on that.
00:26:05.680 --> 00:26:08.559
And it was a very interesting to work on this, right?
00:26:08.720 --> 00:26:12.000
So, in the end, airflow wasn't just scheduling, right?
00:26:12.400 --> 00:26:21.359
More of the control plane that lets us scale from manual analyst for flow to an automated system that could process millions of reviews reliably.
00:26:21.680 --> 00:26:26.000
It was really a unique set of challenges and stuff like that.
00:26:26.240 --> 00:26:26.799
That's awesome.
00:26:27.039 --> 00:26:34.000
I guess like what I love with your use case is that you prove that you can definitely use Airflow as a way to orchestrate LLMs.
00:26:34.240 --> 00:26:39.200
And like sometimes you hear that Airflow is legacy, blah blah blah, but that's not true.
00:26:39.279 --> 00:26:51.839
Like the reality is that today you can still use Airflow for many use cases, and LLM is definitely one of them, and that you just show that you can do it, even as a small team, you can definitely orchestrate LLMs using Airflow.
00:26:52.240 --> 00:26:54.960
Thank you so much for sharing that with us, Nassim.
00:26:55.119 --> 00:26:56.880
I would like to ask you one last question.
00:26:57.039 --> 00:27:02.000
What is the best way for folks to get in touch with you if they want to learn more about your use case?
00:27:02.240 --> 00:27:03.599
Because we just touched it.
00:27:03.680 --> 00:27:06.480
Honestly, there are so much things to talk about.
00:27:06.799 --> 00:27:11.599
Uh, best way is email and LinkedIn is very easy to reach me there.
00:27:11.839 --> 00:27:18.400
And yeah, I'm very uh eager to help everyone in this case, and I loved Airflow so much.
00:27:18.480 --> 00:27:19.920
I started to contribute, right?
00:27:20.000 --> 00:27:23.119
So get getting active into community is also really good.
00:27:23.200 --> 00:27:27.839
So I'm there in the Airflow Slack, in the dev channels and stuff like that.
00:27:28.000 --> 00:27:29.839
So I'm available there too.
00:27:30.000 --> 00:27:32.559
But yeah, I'm very happy to work through this.
00:27:32.720 --> 00:27:33.920
Hopefully, I can learn more.
00:27:34.000 --> 00:27:36.880
So it's knowledge sharing, knowledge transfer.
00:27:37.039 --> 00:27:43.920
When that exists, this community grows, and that's how we as a community we need this in this airflow.
00:27:44.000 --> 00:27:45.759
Okay, we can have this feature in Airflow.
00:27:45.839 --> 00:27:46.559
That's how you grow.
00:27:46.640 --> 00:27:48.559
So it's really interesting.
00:27:48.799 --> 00:27:52.400
Any small contributions to Airflow will be really great.
00:27:52.559 --> 00:27:57.279
So I would love some of the reason I start there, the contributions Airflow.
00:27:57.440 --> 00:28:05.119
And my code contributions are not a really big, very small thing, but it makes me happy that I can all contribute back to the project.
00:28:05.759 --> 00:28:06.319
That's awesome.
00:28:06.480 --> 00:28:07.440
Thank you so much, Nassim.
00:28:07.680 --> 00:28:10.640
Again, thank you so much for joining, and uh pleasure to have you.
00:28:10.799 --> 00:28:11.359
See you soon.
00:28:11.440 --> 00:28:12.160
Take care.
00:28:14.400 --> 00:28:20.880
Thanks for joining us for the latest episode of the Data Flowcast, mastering airflow for data engineering and AI.
00:28:21.039 --> 00:28:24.720
You'll find links in the show notes to any resources mentioned in today's show.
00:28:24.880 --> 00:28:31.519
If you're enjoying our podcast, please subscribe so you never miss an episode and leave us a five star review.
