Specialized AI for Data Engineers: Inside Astronomer’s Otto

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 08/27/2026 12:50:22
Duration: 2729.738
Channels: 1
1
00:00:11.360 --> 00:00:15.440
Hello, and welcome to the Data Engineering Podcast, the show about modern data management.
2
00:00:16.075 --> 00:00:24.314
Your host is Tobias Macey, and today I'm interviewing Yetunde Dada about Otto, astronomer's expert airflow agent. So, Yatunde, can you start by introducing yourself?
3
00:00:24.955 --> 00:00:31.035
Sure. Thank you so much for having me as well. So my name is Yetunde. I'm a senior director of product management at Astronemer.
4
00:00:31.510 --> 00:00:59.365
You'll find me across the Astronomer ecosystem because I oversee things. We're going to talk about auto today. But I also lead other initiatives like Kosmos, which is our way of converting DBT projects into Airflow decks. My background before this is many, many years in the open source space. I think there's been a previous version of your show where you actually covered Kedra, which was like a data engineering and data science framework that I worked on like many, many years ago. So it's good to see you again and be back.
5
00:01:00.040 --> 00:01:16.355
Absolutely. Yeah. No. I I definitely remember that one. Yeah. I've been doing this show for probably too long. Not long enough. Not long enough. There keeps being enough interesting things to keep it going. So, yeah, I guess, can you just give a bit of an overview about how you get started working in data and what keeps you here?
6
00:01:16.755 --> 00:01:17.635
Sure.
7
00:01:17.635 --> 00:01:19.315
So I guess my origins
8
00:01:19.315 --> 00:01:31.760
go way, way, way back. I started off as a data product manager, so more of a consumer of data in a bank in South Africa. It was there that I started to I think this was in the beginnings of Hadoop and Spark
9
00:01:32.000 --> 00:01:53.645
way back in the day when we were trying to obviously create these big data pipelines, combine all of our customer data together in the corporate investment banking setting. My journey beyond that was actually more thinking about the data engineering teams, the data analyst teams that were actually trying to work with the data and building tools to support them. So that was the journey with Kedro when I was at Quantum Black.
10
00:01:54.509 --> 00:02:07.549
I got to work quite closely with Pete on the astronomer side because there was a natural entry point where we said, Kedro, what it does is it's kind of like Django or React for data science and data engineering
11
00:02:07.710 --> 00:02:08.110
projects.
12
00:02:08.515 --> 00:02:36.345
And there's a natural entry point where you say, Okay, you've got this production ready data product, and you want to put it into production, and you want it orchestrated, so just do it with Airflow. And that was how I got to know Pete. And yeah, now you obviously see me on the astronomer side. So always been passionate about data and I guess how it actually helps people when it's it's correct, when it's timely. So, yeah. Now you're in the vanguard of the net new and yet to be explored area of the combination
13
00:02:36.345 --> 00:02:39.465
of data engineering and agentic capabilities.
14
00:02:39.465 --> 00:02:45.625
So I'm wondering if you can give a bit of overview about what auto is and some of the story behind how it came to be and why you're building it.
15
00:02:46.660 --> 00:02:57.700
So I guess for this, I mean, like, we've obviously seen how agents and, like, their use specifically in code and on the software engineering side has been, quite prolific. These tools have become incredible
16
00:02:57.700 --> 00:03:02.295
for the work that we need to do when we talk about, like, generic software engineering. But
17
00:03:02.855 --> 00:03:08.455
last year, we actually ran a survey. We're quite active participants in the state of Apache Airflow survey.
18
00:03:08.614 --> 00:03:09.735
And we did find,
19
00:03:10.455 --> 00:03:21.450
while a lot of data engineers are using a lot more of the generic coding tools and being productive with them, only 9% of them could actually speak about the quality of the code that was being produced by
20
00:03:21.610 --> 00:03:29.035
the coding tools. They often spoke about the missing context that it needed to be excellent in what it did. So there's
21
00:03:29.275 --> 00:03:49.260
obviously time spent where you have to create the context for it yourself or spend time editing the code that it was producing or doing additional steps to troubleshoot, for instance, if you're trying to use it in that use case. So it became an exam question for us where we said, Okay, cool. What does it look like if we actually help provide that context and we provide that layer
22
00:03:49.420 --> 00:03:58.055
so that we could make any agent really productive? And this spurned auto and the work that we've done to really just cement it as like astronomers
23
00:03:58.055 --> 00:03:59.015
perspective
24
00:03:59.095 --> 00:04:01.415
and way that we celebrate
25
00:04:01.575 --> 00:04:08.135
airflow and make sure that our data engineers are productive when working on the myriad of things that you'd be doing in your day to day.
26
00:04:09.090 --> 00:04:12.850
For people who are looking at auto and deciding
27
00:04:12.930 --> 00:04:24.050
whether and how to use it, what are some of the core problems that you're really tackling with the auto product and who is the target audience to sort of the characteristics of who is best served by it?
28
00:04:25.275 --> 00:04:30.075
So for this one, I'll start with a persona because, you know, product loves this question,
29
00:04:30.235 --> 00:04:35.195
but it's data engineers and the range of things that you do. When we speak about like auto's capabilities,
30
00:04:35.435 --> 00:04:42.450
it'll help you all the way from things like DAG authoring and making sure that it's encoded with Airflow based practices when writing that code,
31
00:04:42.770 --> 00:05:04.875
all the way through to doing upgrades. That's a major capability that auto has. The skill that's present in auto doesn't only do major version bumps of airflow. So we know there's obviously a big conversation around moving from Airflow two to three, because Airflow two will be reaching end of life. But it also helps with minor version updates as well. So 3.1 to 3.2,
32
00:05:04.875 --> 00:05:06.955
as well as keeping your provider
33
00:05:07.035 --> 00:05:11.240
ecosystem up to date as well, because we obviously know that there'll be new versions released.
34
00:05:11.480 --> 00:05:14.360
Auto also helps with things like doing investigations.
35
00:05:14.360 --> 00:05:16.760
So we obviously know, as a data engineer,
36
00:05:16.920 --> 00:05:29.435
your pipelines will inevitably fail because nothing is perfect and errors will pop up. But we have specific capabilities built into auto, which really help with troubleshooting and really cutting down that mean time to resolution
37
00:05:29.675 --> 00:05:33.195
as well. Additionally as well, auto also comes prepackaged
38
00:05:33.195 --> 00:05:46.310
with capabilities as well for migrating code. Some of our customers have spoken about their journeys of moving from legacy schedulers, Control M and Autosys, to Airflow because they really want to standardize on Airflow as
39
00:05:47.314 --> 00:05:53.074
the way that they do things going forward because it's open source and it can be more widely supported across the organization.
40
00:05:53.395 --> 00:05:59.955
So it really does come packaged with some of those capabilities that we expose to some of our customers when they opt in for it. There's obviously
41
00:06:00.259 --> 00:06:11.780
a host of ways that we've also found our customers still leverage and use it. One of my favorites I was hearing about recently was how our customer will use it to do health checks on data
42
00:06:11.940 --> 00:06:21.195
platform and really just find things that they need to be resolving on a day to day. So it really does try to be a partner for you across the many things that you'll be working on.
43
00:06:21.755 --> 00:06:22.795
And
44
00:06:22.795 --> 00:06:24.475
as far as the
45
00:06:24.795 --> 00:06:36.030
design and rollout of auto, obviously, you can't boil the ocean and get everything done all at once. So I'm wondering what was your process for figuring out what are the initial
46
00:06:36.030 --> 00:06:40.190
set of capabilities that we're going to target and focus on and refine,
47
00:06:40.350 --> 00:06:58.190
particularly given that agentic capabilities are still being discovered as far as how best to architect and implement them. And, also, there is the probabilistic elements. So you want to make sure that whatever you're doing, it has a high success rate and a low error rate, particularly for the target audience that you're focusing on?
48
00:06:58.910 --> 00:07:07.710
I guess it speaks about, like, the journey that we've actually taken with auto. So last year, we released a product called the Astra IDE,
49
00:07:07.870 --> 00:07:09.150
which is an in browser
50
00:07:09.694 --> 00:07:21.935
web based interface for authoring DAGs or modifying them. And we also have a workflow for testing DAGs against ephemeral deployments so that you can easily spin up the DAG, quickly test it, and then obviously commit
51
00:07:22.095 --> 00:07:51.509
the changes so that you can push them downstream and do whatever you need to there. What we started to see with this interface that we had for the Astra IDE was the type of questions that people were obviously going to be asking as they interacted with it. Obviously, the first use cases that Auto, in the end, sold for initially was the DAG altering and DAG modification use cases. But we also saw as well that a lot of folks would be using it for troubleshooting. So the case for building out specific capabilities for investigations
52
00:07:51.509 --> 00:07:52.229
became
53
00:07:52.789 --> 00:07:59.190
the next big angle that we looked at. We also did see as well that a lot of folks would also use it for upgrading
54
00:07:59.270 --> 00:08:17.625
their code at the many different levels and helping using the agent to make those code changes, especially when there's breaking changes involved with actually changing the structure of your code. So that was another use case that we decided to double down on. I think going forward as well, this does speak to where to next in terms of some of the capabilities that we want to support with auto.
55
00:08:18.210 --> 00:08:19.970
There's
56
00:08:19.970 --> 00:08:25.650
been some questions around how do we, for instance, right size our deployment on Astra.
57
00:08:25.970 --> 00:08:32.610
And we're busy looking at different ways to make those use cases come to life so that auto, in a way, can assist our customers there.
58
00:08:33.185 --> 00:08:33.904
But
59
00:08:34.145 --> 00:08:36.545
mean, we will also call out as well that
60
00:08:36.785 --> 00:08:46.305
there were high acceptance rates for the code even in the space. I think we speak about an 80% to 90% acceptance rate, straight acceptance rate for
61
00:08:46.610 --> 00:08:49.090
our customers, being able to choose
62
00:08:49.330 --> 00:08:57.250
and commit the code that Auto is producing for them as well. So yeah, that was kind of how we thought about the initial capabilities.
63
00:08:57.330 --> 00:09:01.170
I will also speak about some of the work that we've also done
64
00:09:01.505 --> 00:09:13.265
building out additional use cases as well. So this also speaks to the way that auto is constructed. Auto has, in terms of skill set and skills that are available to it as markdown files,
65
00:09:13.680 --> 00:09:22.880
we'll have a component that is present in the open source community as well. So we have on our astronomers organization on GitHub, we have a repo called,
66
00:09:24.320 --> 00:09:25.200
Agents.
67
00:09:25.360 --> 00:09:33.595
And what it does is it has 25 skills that we've seen across the data engineering ecosystem as well that do a host of things, whether it's
68
00:09:33.835 --> 00:09:47.700
the DAG authoring case to working with DBT, working with Cosmos, as I mentioned as well, our framework for converting DBT projects into Airflow DAGs. So there's always additional use cases that we see that will be cold.
69
00:09:47.860 --> 00:09:48.740
So yeah.
70
00:09:49.300 --> 00:10:02.455
In terms of the skills in particular, that is definitely one of the higher leverage capabilities for anybody who's investing in agentic engineering. It's definitely something that I and my team have been spending a lot of time on. And
71
00:10:03.095 --> 00:10:06.375
one of the challenges is making sure that it is
72
00:10:07.350 --> 00:10:15.670
useful enough without being overly prescriptive because then it guides the agent on too narrow of a path and also constrains
73
00:10:15.670 --> 00:10:16.870
the set of
74
00:10:17.270 --> 00:10:31.024
technologies that it's able to reason about. And so I'm curious what your process is for developing and designing and evaluating those skills, particularly as models and harnesses and tool chains evolve over time.
75
00:10:31.960 --> 00:10:39.320
You basically run into one of our active work streams. We're working quite closely with the university. I won't reveal just who yet because
76
00:10:39.640 --> 00:10:51.975
we need to figure out how we do the price embargo stuff. But you will see stuff coming out of the work that we're doing here. What we've basically designed is a test bench, which basically encompasses a range of data engineering
77
00:10:52.214 --> 00:10:53.015
tasks.
78
00:10:53.415 --> 00:10:59.495
We've also assessed as well the complexity and the hardness of those tasks as well when developing our bench. And
79
00:11:00.990 --> 00:11:11.390
the tasks are designed to look across use case or different activity that you would be doing with the agent and being productive with it. We're running basically auto
80
00:11:11.390 --> 00:11:14.430
versus other generic agents through the gauntlet
81
00:11:14.694 --> 00:11:23.894
of those engineering tasks, and also assessing as well the cost that it takes, the range of accuracy or completeness that it does when trying to complete the task,
82
00:11:24.214 --> 00:11:25.495
and obviously developing
83
00:11:25.654 --> 00:11:30.295
the framework there. We will be open sourcing the framework as well, the Data Engineering Benchmark,
84
00:11:31.070 --> 00:11:32.350
to make sure that
85
00:11:32.670 --> 00:11:39.310
going forward, when people look at how they develop new agents to also tackle things across the data engineering stack,
86
00:11:39.470 --> 00:11:56.375
that they can use this as an additional way of assessing its capabilities and also make things better. It's one of the ways I think that we get to celebrate in the open source community as well, building better agents so that our data engineers can be more productive. But it also makes it easier for us to also show you why auto's capabilities
87
00:11:56.455 --> 00:11:59.895
rank higher than most other agents because of the way that it's built.
88
00:12:01.310 --> 00:12:07.230
Digging now a bit more into auto and its particular relationship to Astronomer
89
00:12:07.230 --> 00:12:08.990
in terms of the
90
00:12:09.870 --> 00:12:14.110
target, it makes sense because the orchestrator is intended to be the
91
00:12:14.805 --> 00:12:24.405
core element of any data platform. It has the most visibility. It has the highest potential leverage for impacting the entire data estate of an organization,
92
00:12:24.805 --> 00:12:27.365
presuming that it's actually being properly utilized.
93
00:12:27.879 --> 00:12:31.000
And so I'm wondering if you can talk to some of the
94
00:12:31.639 --> 00:12:32.519
architectural
95
00:12:32.519 --> 00:12:33.800
aspects of
96
00:12:34.600 --> 00:12:38.120
airflow in particular that lend it to be
97
00:12:38.680 --> 00:12:48.885
something that a system like auto can work well against and just some of the peculiarities and particularities of airflow that you've had to either reconsider
98
00:12:48.885 --> 00:12:55.925
and just some of the ways that the interplay between the orchestration framework and the agent that you're developing forces
99
00:12:56.170 --> 00:12:56.970
changes
100
00:12:56.970 --> 00:13:00.730
to your overall design and approach across that boundary.
101
00:13:02.170 --> 00:13:10.250
And, you know, if I actually just build on your question, I guess it depends on how you're using Airflow as well as to whether or not, like, Airflow is truly in the
102
00:13:10.795 --> 00:13:13.035
the interception path behind
103
00:13:13.035 --> 00:13:16.235
having all the context that it needs to be able to answer that question.
104
00:13:17.115 --> 00:13:37.230
I guess this actually also does suggest as well, one of the ways that we have decided to build out auto is, especially when you're using it in the CLI, because maybe this is another thing. Auto is available via the Astra IDE, which is our in browser workflow there. It's also available via our CLI, which is the Astra CLI. So you can just get started and use it there.
105
00:13:37.855 --> 00:13:43.455
When we talk about auto and the CLI and its uses there, the fact that it's able to call in additional MCPs
106
00:13:43.455 --> 00:13:48.575
to complete the picture is one of the ways that we find that our customers
107
00:13:48.735 --> 00:13:59.560
have been, in particular, successful with it. Because to your point around the way that Airflow works, Airflow is the supreme orchestrator. We know this, and it does its job fantastically well.
108
00:13:59.960 --> 00:14:05.240
But one of its design considerations is that because it's so broad and so general,
109
00:14:05.735 --> 00:14:09.975
unless you're very specific about telling it what each individual task does
110
00:14:10.295 --> 00:14:16.615
and where all of the context around that task is located, you might not necessarily have the full picture.
111
00:14:17.015 --> 00:14:20.535
So to give you an example of what this would have meant with our
112
00:14:20.970 --> 00:14:23.290
investigation capabilities in auto,
113
00:14:24.009 --> 00:14:32.649
when it was initially designed, we had situations with our customers where they would say things like, hey, the investigation agent has told me, yes, this task has failed.
114
00:14:32.970 --> 00:14:35.449
But it's basically told me the error is somewhere
115
00:14:35.985 --> 00:14:49.425
in Databricks and kind of go figure it out. And for that, when we want to talk about our vision towards self healing pipelines, for it to stop, for the journey to stop around troubleshooting at that boundary is not enough for us,
116
00:14:50.079 --> 00:15:05.135
which is why, like, Autumn or broadly, especially in the CLI, has obviously a way for you to set up your MCP so that it can actually go forth and go and help you troubleshoot in the platform and you complete the answer there. So yeah, it does depend, obviously, on exactly
117
00:15:05.375 --> 00:15:06.975
how you're using Airflow.
118
00:15:07.135 --> 00:15:08.575
I will say as well, it does
119
00:15:09.535 --> 00:15:14.975
I also maybe I can call out why, for instance, you'd maybe use Cosmos, for instance.
120
00:15:16.010 --> 00:15:18.890
When we talk about making sure that Airflow is complete
121
00:15:19.130 --> 00:15:31.690
and has that full picture, tools like Cosmos, which obviously stretch into your DBT project and really help unfold it into it being a full Airflow DAG, is much better than you using, for instance, like a Bash operator or KPO
122
00:15:32.065 --> 00:15:40.545
to be orchestrating and calling the DBT CLI to complete that picture there. So the more information you give to Airflow and the more central it is in your pathway,
123
00:15:40.945 --> 00:15:48.300
the better time actually you have with using these agents to have a one stop answer, especially when things are not going as expected.
124
00:15:49.020 --> 00:15:50.060
And
125
00:15:50.060 --> 00:15:50.620
for
126
00:15:50.940 --> 00:15:53.980
people who are developing these pipelines,
127
00:15:54.060 --> 00:16:06.945
they're bringing auto into the mix. I'm sure a lot of them have already started experimenting with some of the more generic general purpose agents such as Cloud Code or Codex or OpenCode or Copilot, what have you.
128
00:16:07.425 --> 00:16:09.105
What are some of the
129
00:16:09.905 --> 00:16:10.785
distinctions,
130
00:16:10.945 --> 00:16:11.265
and
131
00:16:11.900 --> 00:16:15.340
what are some of the reasons that they might reach for auto specifically
132
00:16:15.340 --> 00:16:21.980
instead of their day to day driver coding agent? And what are some of the specific
133
00:16:22.060 --> 00:16:24.460
workflows and capabilities that auto
134
00:16:24.805 --> 00:16:34.725
not necessarily enforces but encourages that you can't easily replicate by just pulling in a set of skills into whatever your day to day agent harness might be.
135
00:16:35.685 --> 00:16:45.950
So for this one, like, obviously, we are the Airflow company. And we spend a lot of time curating proprietary knowledge that is not necessarily available in the open source community.
136
00:16:46.430 --> 00:16:52.270
In particular, I'll call out three branches of it. One might be exactly on how to use the Astra platform.
137
00:16:52.875 --> 00:16:57.115
And we make that obviously available to Otto as a knowledge base.
138
00:16:57.435 --> 00:17:01.835
Another one to call out is the capabilities around troubleshooting and investigating
139
00:17:02.395 --> 00:17:05.035
DAG and task failures, because we have extensive
140
00:17:05.035 --> 00:17:07.515
knowledge on how to resolve those issues.
141
00:17:08.290 --> 00:17:10.529
So we've built that into Auto2.
142
00:17:10.530 --> 00:17:14.049
And the last one to call out is that the knowledge base around upgrading
143
00:17:14.530 --> 00:17:16.610
Airflow DAGs, especially moving from
144
00:17:17.890 --> 00:17:20.610
major versions, minor versions, and then equally,
145
00:17:20.850 --> 00:17:25.245
knowledge around upgrading your provider ecosystem as well is extensive.
146
00:17:25.645 --> 00:17:33.805
And that's also made available to auto there. So what you eventually see when our customers actually talk about whether or not they're going to be using Cloud Code or
147
00:17:34.240 --> 00:17:39.519
Codecs in their workflow. They'll often reach for auto when working with Airflow
148
00:17:39.600 --> 00:17:45.200
because it is the best tool in and above it being able to load the public documentation
149
00:17:45.600 --> 00:17:55.585
and the astronomer documentation into its interface. They'll use that. And they actually talk about it just being extremely good at airflow in and above those generic tools because it has that
150
00:17:55.985 --> 00:18:09.029
we'd call it the public knowledge that any of these generic tools could use. It obviously has the Astra proprietary knowledge also built into it. And then it also has additionally, we have ways to obviously store memories
151
00:18:09.190 --> 00:18:13.190
and remember things about its interactions when you're doing things like
152
00:18:13.429 --> 00:18:19.255
perhaps it would be as simple as, this tag always fails, just retry it. And being able to store it,
153
00:18:19.815 --> 00:18:25.174
those pieces of knowledge context for auto to be using going forward. So it does get smarter over time.
154
00:18:25.975 --> 00:18:28.710
To that point of getting smarter
155
00:18:28.790 --> 00:18:31.909
and self improvement in an agent context,
156
00:18:31.910 --> 00:18:37.029
that also brings up some of the challenges and complexities when you're building any
157
00:18:37.590 --> 00:18:56.440
data engineering oriented tool is the concept of data access and data ownership and also the potential for lock in where data engineers are allergic to any potential for vendor lock in. We want our data to be easily transferred and migrated wherever we want it to as the wins and wins change.
158
00:18:56.520 --> 00:19:11.400
And so I'm wondering if you can talk to some of the ways that you've thought about that aspect of the design of auto to make sure that it is fully visible and in control of the person who's using it and avoids any potential vendor lock and then also the
159
00:19:11.825 --> 00:19:12.705
security
160
00:19:12.705 --> 00:19:28.850
and data access sensitivity element of it to make sure that there are very clear and well understood boundaries about what data is accessed when and where it might be applied so that there's no risk of training on somebody's corporate data.
161
00:19:29.650 --> 00:19:31.650
So I'll first speak to, like, how,
162
00:19:31.970 --> 00:19:47.374
you know, we obviously save additional context about you, especially as you move through your journey of, like, using auto. We do obviously adopt a lot of the open source patterns around being able to save memories to a markdown file for you to access and make it so that you can edit it as well.
163
00:19:47.695 --> 00:19:55.559
So in terms of that whole thing of like, Okay, I've been heavily reliant on auto. And for whatever reason, I want to be able to use this context in other situations.
164
00:19:55.880 --> 00:20:03.160
Let's just save to your Git repo so that you are productive already in those situations. And you can obviously point a new agent to use it there.
165
00:20:03.865 --> 00:20:05.865
We have fought extensively
166
00:20:05.865 --> 00:20:10.504
about security, data privacy, and also letting our customers know exactly
167
00:20:10.745 --> 00:20:13.304
what forms of data are consumed by auto,
168
00:20:13.784 --> 00:20:15.625
how we make sure that we're protecting
169
00:20:16.360 --> 00:20:21.159
their privacy as we use it. The way that auto is designed, at least so auto has,
170
00:20:21.799 --> 00:20:22.440
obviously,
171
00:20:23.320 --> 00:20:23.879
its
172
00:20:24.600 --> 00:20:35.875
skills based. Part of it is proprietary, and the other part of it is open source. And then all of this is connected by our LLM gateway, which allows you to choose which models you'd like to use through our model provider.
173
00:20:36.195 --> 00:20:38.595
We have thought extensively about how
174
00:20:38.595 --> 00:20:43.550
we allow our customers to access the different models. And there is heavy separation
175
00:20:43.550 --> 00:20:44.270
around
176
00:20:44.510 --> 00:21:00.415
their data that is used, because obviously we'd be accessing things like your code. We would see your prompts through the LLM gateway as well. That all of those things are air gapped between our customers as well. So there's no cross sharing at all. It's just you and the way that you interact with auto
177
00:21:00.415 --> 00:21:05.775
as opposed to it being broadly available. So the way that you find that information, if you were very curious
178
00:21:05.935 --> 00:21:09.135
for whatever reason, is we do have a very good trust website
179
00:21:09.450 --> 00:21:15.529
which actually talks about how we think about data privacy and security for our customers to make sure that
180
00:21:16.250 --> 00:21:21.049
really they can trust that we have their best interests at heart when they're using auto and these agents.
181
00:21:22.145 --> 00:21:25.424
And the other interesting aspect of any
182
00:21:25.745 --> 00:21:27.424
agent engineering
183
00:21:27.424 --> 00:21:29.825
is the model choice factor,
184
00:21:29.825 --> 00:21:56.845
where somebody maybe has an existing enterprise license with one vendor or another, maybe they focus on open weight models that they run themselves, or maybe they even have their own in house model, and that can dramatically change the capabilities and behaviors of the agent within a given harness, within a given set of skills. And so I'm wondering what are some of the challenges that you're dealing with in terms of the design and implementation to have that adaptability and element of choice for people.
185
00:21:57.485 --> 00:21:58.205
This
186
00:21:58.365 --> 00:21:59.485
actually does suggest
187
00:21:59.645 --> 00:22:04.285
what are some of the things we've learned about our customers as well as we've been on this journey. Because
188
00:22:04.660 --> 00:22:11.220
for some of our customers, to your point, they might just have a preference and have standardized on a single tool that they use in their organization.
189
00:22:11.780 --> 00:22:23.244
Or it might even be as simple as, look, it's hectic. Let's say, hectic to get AI tools approved internally in their organization. And they're worried about how do they get another one approved,
190
00:22:23.404 --> 00:22:25.404
especially when this one is a standard choice.
191
00:22:25.725 --> 00:22:39.820
With this one, we've kind of made two steps to actually solve it. And they're going be shipped over the next month or two. The first one is an ability to bring your own model. So within the LLM Gateway, we're making it possible to kind of bring your own API key
192
00:22:40.140 --> 00:22:49.454
to your model so that you can plug it into the spaces like where auto fits across the Astra platform. So whether it's through the CLI or through our IDE or even on the Astra workspace,
193
00:22:49.855 --> 00:22:54.014
and we would be able to leverage your model, especially where you have these requirements.
194
00:22:54.254 --> 00:22:56.654
And then the second thing as well is we're
195
00:22:56.975 --> 00:23:03.459
just about to re release our Astra MCP as well. And obviously, the scope of the MCP was largely around
196
00:23:03.860 --> 00:23:09.379
providing a wrapper, you could say, around the Astro API and some of the functions that you can do there.
197
00:23:10.340 --> 00:23:15.860
We will also make as well in that package a way for you to access auto skills
198
00:23:16.075 --> 00:23:19.995
as well as part of the package that you get with the Astro MCP.
199
00:23:20.075 --> 00:23:23.035
And in this world, it means, obviously, that if
200
00:23:23.195 --> 00:23:25.355
you'd standardize on a specific agent,
201
00:23:25.515 --> 00:23:36.509
you could still actually leverage at least auto skills in order to be productive with the workflow, especially in cases where you do have a preference around using a specific agent and want to do that.
202
00:23:38.110 --> 00:23:39.469
And so
203
00:23:39.630 --> 00:23:45.265
digging now into auto itself, can you describe some of the design and architecture
204
00:23:45.265 --> 00:23:48.945
of that utility and some of the ways that the
205
00:23:49.665 --> 00:23:55.260
scope and implementation have evolved from when you first started building it? So
206
00:23:55.260 --> 00:23:57.660
in terms of the way auto is constructed,
207
00:23:57.900 --> 00:23:58.860
there are
208
00:23:59.660 --> 00:24:02.780
two layers of basically proprietary
209
00:24:03.820 --> 00:24:05.180
and open source knowledge.
210
00:24:05.420 --> 00:24:15.145
The open source knowledge comes from the agents repo, which I spoke about that you should go have a look at. And then obviously, we have a proprietary knowledge base that is also part of the equation for auto.
211
00:24:15.305 --> 00:24:27.410
It is all behind an LLM gateway, which basically allows our customers to choose which model they want to use through our model providers. So they have the flexibility to do that. To your point around what
212
00:24:27.410 --> 00:24:35.570
kind of architectural decisions and choices have we made and I guess I can also speak to auto V2, which is also one of the things that we're working on now
213
00:24:36.210 --> 00:24:42.375
I think in tune with all the work that we've done on the benchmarks and really assessing auto against specific
214
00:24:42.375 --> 00:24:43.254
use cases
215
00:24:43.575 --> 00:24:47.174
and how well it stacks on being able to complete data engineering
216
00:24:48.295 --> 00:24:49.095
tasks
217
00:24:49.255 --> 00:24:51.254
in a timely way,
218
00:24:51.495 --> 00:24:52.855
using less tokens
219
00:24:53.940 --> 00:24:55.779
and making sure that, obviously,
220
00:24:55.940 --> 00:24:57.539
our customers spend less.
221
00:24:57.700 --> 00:25:01.859
Auto is built on top of the PIE framework as well. We've taken a real stab at
222
00:25:02.179 --> 00:25:06.580
really looking at what it means for which context to be applied to auto.
223
00:25:07.165 --> 00:25:13.164
These models have become very good at what they do in terms of how they know about airflow
224
00:25:13.325 --> 00:25:16.764
and have a wide understanding of it. So not all context,
225
00:25:16.925 --> 00:25:27.979
as we had it in the previous version, in the current version of auto, is necessary for it to be productive because the agents already know. So you do see us making this exercise of we're applying context,
226
00:25:27.980 --> 00:25:52.010
running that version of auto through the gauntlet of data engineering tasks, assessing if that context made auto significantly better because that's the delta, And then if not, removing that context as well. So you are going to see as well a slimmer version of auto, which only really has the stuff that really makes it exceptional against the other agents. And that's what will be packaged and released. So, yeah, those are some of the changes that we've made.
227
00:25:53.530 --> 00:25:57.050
Also, because you are bringing this into
228
00:25:57.050 --> 00:25:59.290
the workflow of an organization
229
00:25:59.290 --> 00:26:03.690
and a team, those teams have their own dynamics. They have their own preferences.
230
00:26:03.690 --> 00:26:06.225
They want to make sure that auto is
231
00:26:06.465 --> 00:26:10.065
well aligned to their stylistic guidelines,
232
00:26:10.065 --> 00:26:13.184
their data quality controls. What are the different
233
00:26:13.265 --> 00:26:18.625
axes along which they can personalize and extend auto to fit their particular situation?
234
00:26:19.810 --> 00:26:22.450
Cool. So with this one, auto does respect
235
00:26:22.450 --> 00:26:36.125
if you're using, like, Cloud MD or Skills MD, Skills markdown file, like, Cloud does respect those things. So we always think of that as, the bucket of, like, team context that you might be applying to auto to make sure that it actually is following best practices and guidelines
236
00:26:36.285 --> 00:26:37.565
with the way that it works.
237
00:26:37.885 --> 00:26:45.965
In particular, though, I will call out, when we shipped the initial version of auto in the Astra IDE, the additional thing that it used to do was obviously
238
00:26:46.605 --> 00:26:48.045
refer to the whole code base
239
00:26:48.490 --> 00:26:55.609
as the standard for how it should be mimicking and writing code as well. So it would also consider that when it was doing code generation.
240
00:26:55.850 --> 00:27:17.865
I think as much as possible, even though we don't know how the initial code base was developed and whether or not that code base is best practice, it is the one it is the code base that is most familiar to your end users and to the way that you write. So auto does try and respect some of those things as well, but it it might make suggestions on how to improve things if it does seem bad practices being done. So yeah.
241
00:27:18.860 --> 00:27:48.420
As people are starting to bring auto into the mix, what are some of the ways that it changes their overall approach or workflow or just some of the adoption curve of people, particularly if they haven't already been heavy users of agents and they just say, my Airflow system is a mess. I need to get it into better shape. I'm gonna bring in auto because it's just magic. It'll do everything I want it to do. I don't even have to think about it. Just some of the, I guess, misconceptions
242
00:27:48.420 --> 00:27:49.140
or
243
00:27:49.380 --> 00:27:50.179
proper
244
00:27:50.500 --> 00:27:59.700
approaches for how to introduce it to a team, especially if they haven't already built up that muscle of working heavily with agents for other areas of their engineering work.
245
00:28:01.175 --> 00:28:02.615
To be fair, with
246
00:28:03.095 --> 00:28:06.935
the way that agents are so prolific, there's agents everywhere. Most
247
00:28:06.935 --> 00:28:09.975
of our customers have had experience with
248
00:28:10.455 --> 00:28:23.299
some other tool when using it. And obviously, for them, when they try auto, the real differentiating factor is its knowledge on ASTRO and its knowledge on Airflow, which really does change the game for them. I can maybe speak to two
249
00:28:23.620 --> 00:28:28.260
examples of customers that have used auto for different purposes
250
00:28:28.765 --> 00:28:30.764
and really used it to
251
00:28:30.925 --> 00:28:38.365
make themselves productive in situations. So the first one is Janus Henderson Investors. They're one of the world's largest asset managers.
252
00:28:38.605 --> 00:28:39.405
And really, them,
253
00:28:40.860 --> 00:28:47.419
the use case for them was around, how do we make sure that we can reduce the mean time to resolution for our DAG failures?
254
00:28:47.660 --> 00:28:52.539
When a DAG goes down for them at 3AM in the morning, that's money lost
255
00:28:53.260 --> 00:28:54.220
for them as a company.
256
00:28:54.915 --> 00:29:00.915
And they used to have tons of on call engineers that their job was just to be maintaining these pipelines.
257
00:29:01.155 --> 00:29:03.315
So they built Auto into their Lighthouse
258
00:29:03.475 --> 00:29:04.995
product. And basically,
259
00:29:05.235 --> 00:29:07.235
focus of Auto in this situation is,
260
00:29:07.780 --> 00:29:09.620
how can we help with
261
00:29:09.620 --> 00:29:10.420
investigating
262
00:29:10.420 --> 00:29:12.500
the failure at the moment that it happens,
263
00:29:12.980 --> 00:29:16.580
creating the PR, especially if a code fix is required.
264
00:29:16.980 --> 00:29:27.025
And then also, if the answer is just retriggering the task to see if that actually does it, Auto should do that for them. And it does it via the Astralem API.
265
00:29:27.265 --> 00:29:35.345
And then if it's anything that it can't really fix, then it should hand it over to the team to come and look at it. They talk about a 95%
266
00:29:35.659 --> 00:29:36.700
reduction in the
267
00:29:37.260 --> 00:29:39.580
meantime to resolve open
268
00:29:40.380 --> 00:29:47.260
bugs and things that they have related to their DAGs as really one of the ways that they talk about success of this initiative.
269
00:29:48.025 --> 00:29:54.585
Another case might be as well, to your point, because you did bring it up when you spoke about some teams that have not really
270
00:29:54.985 --> 00:30:06.260
adopted maybe agents in their day to day. I can also give a case about a customer that needed to adopt Airflow because they were trying to do what we call a legacy tool migration.
271
00:30:06.580 --> 00:30:10.660
So they're one of The UK's largest consumer electronic companies.
272
00:30:11.140 --> 00:30:12.340
And they
273
00:30:13.144 --> 00:30:18.104
have many stores and then also have a logistics network that manages the sale of electronics.
274
00:30:18.504 --> 00:30:22.024
They spoke about wanting to sunset their jam scheduler.
275
00:30:22.424 --> 00:30:26.184
And they had about 5,000 jobs on this jam scheduler
276
00:30:26.460 --> 00:30:27.739
and needed to
277
00:30:28.059 --> 00:30:34.619
very quickly get up to speed with Airflow because that was the tool the data engineering team had selected that they were going to be moving to.
278
00:30:35.580 --> 00:30:41.340
Auto was essential in the team being able to get up to speed with Airflow knowledge
279
00:30:41.615 --> 00:30:45.375
and also migrate a host of jobs in a very short
280
00:30:46.015 --> 00:30:46.735
time scale.
281
00:30:47.135 --> 00:30:53.215
And when they started using Auto in this way, they had had experience with other tools like Copilot,
282
00:30:53.455 --> 00:31:06.889
but still found as well the quality of the code that was coming out of Auto, the fact that it was best suited to Airflow and ASTRO was actually best fit for them. So they continued to work with Auto on their journey of modernizing their
283
00:31:06.970 --> 00:31:15.184
data platform. And as they sunset more tools and move to Airflow, it really is an essential tool for them around the journeys of upskilling,
284
00:31:15.585 --> 00:31:20.784
along the journeys of DAG authoring in this case as well. And then also,
285
00:31:20.945 --> 00:31:24.625
they also use it as well for DAG improvements going forward. So yeah.
286
00:31:26.170 --> 00:31:27.610
And as people
287
00:31:28.250 --> 00:31:33.930
are investing more in using auto as part of their day to day work,
288
00:31:34.170 --> 00:31:35.850
one of the other challenges
289
00:31:35.850 --> 00:31:36.490
of
290
00:31:36.730 --> 00:31:43.235
generating new code is the challenge of validating it. And I know that that Auto has a PR review capability,
291
00:31:43.555 --> 00:31:49.154
and I'm wondering if you can talk to some of the work that went into ensuring that that was providing
292
00:31:49.475 --> 00:31:58.700
enough confidence for people to be able to actually move faster because it doesn't matter how much code you can generate if you never ship any of it and just some of the other elements of
293
00:31:59.020 --> 00:32:01.100
building useful and
294
00:32:01.260 --> 00:32:04.700
accurate validation loops into that overall
295
00:32:05.100 --> 00:32:08.140
use case of pushing more of the change through
296
00:32:08.375 --> 00:32:09.255
agents
297
00:32:09.255 --> 00:32:11.495
that are directed by human operators.
298
00:32:11.895 --> 00:32:21.495
So I guess there's actually two layers to thinking through how you validate your code, because you could validate it at the time of development, and then you can validate it at the time the PR is created.
299
00:32:22.240 --> 00:32:27.839
I'll speak to both for auto as well because it does have tools that help you with both of those things. Auto,
300
00:32:27.840 --> 00:32:31.279
especially when you're working with it through the CLI, will actually call
301
00:32:31.760 --> 00:32:33.200
something we've called AF
302
00:32:33.679 --> 00:32:36.135
CLI, the Airflow CLI. And it's a
303
00:32:36.615 --> 00:32:38.855
trimmed down version of the ASTRO CLI.
304
00:32:39.015 --> 00:32:41.495
And it was built with the intent that
305
00:32:41.895 --> 00:32:46.215
it's more agent friendly. The ASTRO CLI is built for humans to be using,
306
00:32:46.215 --> 00:32:51.989
so it has more of an interactive feel as you're working through it. And we really did want an approach where
307
00:32:52.470 --> 00:32:55.749
an agent could interact with a CLI and be productive with it.
308
00:32:56.230 --> 00:33:12.445
Because Auto has access to this tool, it's actually able to validate your code for you. So it will spin up a local Airflow instance for you. It will trigger your DAGs for you. It will troubleshoot and resolve the errors for you as well before you're ready to create, obviously, your PR and work
309
00:33:12.845 --> 00:33:24.820
from there. Now, to your point of raising the fact that auto actually helps with code review, we do actually have skills that are focused on reviewing the code against Airflow based practices,
310
00:33:25.060 --> 00:33:36.335
against things like our Airflow upgrade knowledge base as well. So even the code that you're working against, it will constantly be checking to see if this code is up to scratch. And then make suggestions
311
00:33:36.335 --> 00:33:41.054
as well on what to improve as well for the person that is reviewing the code.
312
00:33:41.135 --> 00:33:43.695
Teams will use this in and above whatever
313
00:33:44.335 --> 00:33:54.649
because some of our customers do use another code review tool that has other best practices built in. But auto is basically the layer on top that is the airflow expert, airflow expert
314
00:33:54.730 --> 00:33:55.929
that will help guide
315
00:33:56.330 --> 00:33:58.410
the PRs and the changes that they need to do there.
316
00:34:00.345 --> 00:34:01.625
As you are
317
00:34:01.785 --> 00:34:02.985
bringing auto
318
00:34:03.545 --> 00:34:05.145
into the ecosystem,
319
00:34:05.145 --> 00:34:06.505
working with your customers,
320
00:34:06.585 --> 00:34:07.465
onboarding
321
00:34:07.465 --> 00:34:08.985
engineering teams,
322
00:34:09.225 --> 00:34:13.545
what are some of the most interesting or innovative or unexpected ways that you've seen it applied?
323
00:34:14.970 --> 00:34:20.250
So I will speak to one of our customer use cases that's probably my favorite.
324
00:34:20.490 --> 00:34:29.850
They've decided to use auto as a way to do a health check across the ASTRO platform for them. What auto does in this situation is it generates a daily
325
00:34:31.025 --> 00:34:31.745
report
326
00:34:31.985 --> 00:34:37.745
of the performance of ASTRO across their deployments. So it gives a read on what is happening with their deployments,
327
00:34:37.905 --> 00:34:39.585
what is happening with their DAGs.
328
00:34:39.745 --> 00:34:43.345
And when they have this generated report, they basically triage
329
00:34:43.345 --> 00:34:50.340
specific issues that they need Auto to go resolve. And Auto is the functional partner with helping them manage their workload and
330
00:34:51.220 --> 00:34:53.380
address specific issues that they find.
331
00:34:53.620 --> 00:34:56.020
I think for this, it does speak to,
332
00:34:56.500 --> 00:35:06.825
obviously, one of the reasons why we developed this, which is that we want you to have an agent that is your partner alongside all of the work that you have to do. You will have so many competing priorities
333
00:35:07.065 --> 00:35:13.545
and things that you need to keep up to date. And if you can hand off some of the most boring pieces of your work to an agent to go and do, why not?
334
00:35:14.410 --> 00:35:19.530
So, yeah, I think probably using auto for health checks, I think, has been one of my favorite to learn.
335
00:35:20.650 --> 00:35:24.010
And as you are working in this space,
336
00:35:24.250 --> 00:35:25.210
building this system,
337
00:35:25.865 --> 00:35:36.505
helping to push the forefront of what agents can do, particularly in the challenging space of data engineering? What are some of the most interesting or unexpected or challenging lessons that you learned personally?
338
00:35:37.545 --> 00:35:41.224
This maybe also speaks to the roadmap for auto going forward. So
339
00:35:41.790 --> 00:35:46.990
one of the things that we have learned is that and you did actually allude to it when you said something around people
340
00:35:46.990 --> 00:35:52.510
have their preferences about which agents they're using, which interfaces they're also accessing the
341
00:35:52.590 --> 00:35:57.855
agents from. And it really has become one of our goals that we're able to integrate
342
00:35:58.015 --> 00:36:05.855
auto exactly where you are. So you will see things over the next few months where auto comes to the Airflow UI. Because when
343
00:36:05.855 --> 00:36:10.800
you need to troubleshoot a DAG, you're not necessarily anywhere else. You're sometimes on Airflow
344
00:36:10.880 --> 00:36:13.200
and need to be able to quickly resolve that.
345
00:36:13.520 --> 00:36:24.875
To things like we are working towards releasing an Astro Versus Code extension as well, which also brings auto to your workspace because that's where most of our customers are writing code still
346
00:36:25.035 --> 00:36:26.955
to make sure that it's
347
00:36:26.955 --> 00:36:29.195
top and present. So yeah,
348
00:36:30.235 --> 00:36:40.620
we really speak about how do we make sure that we can put auto in the places that you're working so that you can be productive there go and use it, as opposed to making you leave different tools.
349
00:36:41.340 --> 00:36:43.260
But I think going forward,
350
00:36:43.660 --> 00:36:50.620
I am very excited to see how all of this agentic work really does impact the way that we view and see
351
00:36:50.860 --> 00:36:52.620
data engineering. My hope is that,
352
00:36:53.885 --> 00:37:00.605
obviously, with auto taking care of some of the more boring and operational tasks for your work, you're really able to
353
00:37:01.005 --> 00:37:04.125
really prioritize the value creation activities upfront,
354
00:37:04.205 --> 00:37:08.205
the new work that you have around making the data platform even better,
355
00:37:08.790 --> 00:37:10.310
to even things like,
356
00:37:10.630 --> 00:37:14.870
obviously, working through the massive backlog of new DAG requests
357
00:37:14.870 --> 00:37:16.870
that you have and being productive there.
358
00:37:17.190 --> 00:37:17.670
So
359
00:37:18.310 --> 00:37:27.635
I really do see an exciting future for data engineering just purely because we have these agents that have become really good for data engineering work. They know the code.
360
00:37:27.875 --> 00:37:28.995
They know the data.
361
00:37:29.155 --> 00:37:33.235
And they know exactly how to make you productive there. So yeah.
362
00:37:34.515 --> 00:37:42.350
And on that point of knowing the data, that brings me back around to something that we didn't dig into yet where the orchestrator
363
00:37:42.910 --> 00:37:45.710
knows what it knows, but it's not necessarily
364
00:37:45.869 --> 00:37:48.750
a complete view of the entire data estate.
365
00:37:48.750 --> 00:38:02.525
And maybe you have three different Airflow instances that are all focusing on different areas of the business or different use cases. And I'm wondering what are some of the elements of things like data catalogs
366
00:38:02.525 --> 00:38:20.700
that are maybe more cross cutting that give the complete view because maybe you don't want to have one Airflow instance running everything and just some of those other aspects of additional context engineering that you need to do to make sure that auto actually has the necessary detail to properly design and scope a particular change.
367
00:38:21.575 --> 00:38:27.494
So this is where I actually alluded to the fact that, like, auto in the CLI as well, like, in particular, can access, like,
368
00:38:28.055 --> 00:38:46.150
an MCP connector. It has an MCP framework as well. This makes it obviously possible to bring in that additional context because to your point, depending on how deeply integrated you have Airflow in that system I think I gave the example of whether or not you choose to use, for instance, Cosmos, which will unpack the entire DBT project
369
00:38:46.710 --> 00:38:54.895
and have a very clear structure for it as opposed to using a Bash operator or Kapio to run the dbt CLI instead
370
00:38:55.135 --> 00:38:56.175
means that
371
00:38:56.494 --> 00:39:01.455
you're obviously able to bring in that additional context for where you need to complete the picture.
372
00:39:02.974 --> 00:39:05.535
Yeah, I do see it as the
373
00:39:05.535 --> 00:39:19.570
partnership between Airflow and the tool or the thing that it's supposed to orchestrate and how you bring in that additional layer to complete the picture. So yeah, think the only real way to do it is bring in the context where you need it. And auto can do just that.
374
00:39:20.625 --> 00:39:22.945
And for people who are
375
00:39:23.425 --> 00:39:30.065
working in the data engineering space, maybe they're already using Airflow, maybe they're using a different orchestrator
376
00:39:30.065 --> 00:39:38.720
or no orchestrator at all, or maybe they've already invested a lot into a different agent harness, what are the cases where auto is the wrong choice?
377
00:39:39.840 --> 00:39:41.280
I guess if you're
378
00:39:41.280 --> 00:39:42.160
very,
379
00:39:42.160 --> 00:39:42.720
very
380
00:39:43.360 --> 00:39:47.200
no, let's actually break this one down because I think there's ways to go with it.
381
00:39:47.680 --> 00:39:52.335
When we talk about auto as an entire product, we're talking about its
382
00:39:52.575 --> 00:40:00.655
open source skills, its proprietary knowledge base, and the fact that it's connected to our LLM gateway. And it's available within, obviously, the Astronomer platform.
383
00:40:01.055 --> 00:40:08.660
And we provide all of the additional tools on top of it, like having access to memory and storing your team conventions and the like.
384
00:40:09.220 --> 00:40:12.420
In situations where you do have to use
385
00:40:12.740 --> 00:40:38.390
your own agent because maybe your organization has picked a specific tool that you're supposed to use, I would say that maybe auto doesn't fit for your use case then, because obviously, you'd have to go through ways of getting it approved or chosen to be used in as well. And maybe you don't have the authority to do that. But the reason I thought to highlight the fact that auto is broken down into those multiple components means that we can still make the context available to you in different ways.
386
00:40:38.869 --> 00:40:49.590
Because in those situations, maybe the better fit for you is to use the Astra MTP, which we're going to be releasing soon, which has obviously access to the auto skills. So you can pull them into your agent and use them from there.
387
00:40:50.295 --> 00:40:51.335
Or if
388
00:40:52.214 --> 00:40:55.255
you need to go a step further and just use widely
389
00:40:55.255 --> 00:40:56.615
available open source
390
00:40:56.775 --> 00:41:08.560
things, then I'd recommend that you'd use the agents repo, which we have, which has an available skill set of the open source skills at the least. Doesn't necessarily have the proprietary skills, which are the special sauce for auto.
391
00:41:08.560 --> 00:41:13.600
But you have something if you're going to be using it, especially in the context of working with Airflow
392
00:41:13.680 --> 00:41:22.485
as well. So there's different ways that we think to package up parts of auto so that you can use it, like, I guess, in the situations where you need it.
393
00:41:23.365 --> 00:41:32.245
And you've mentioned some of the forward looking plans that you have for auto, but I'm wondering if there are any other aspects of the road map or just particular
394
00:41:32.720 --> 00:41:44.480
areas that you're excited to explore that you wanted to share with folks? I think maybe the I obviously spoke about, like, the surfaces that we're gonna make available for auto. So the Astro MCP, which will be shipping the Versus Code extension,
395
00:41:45.155 --> 00:41:49.395
and then also the fact that auto will be present on the Airflow UI as well,
396
00:41:49.954 --> 00:41:52.995
really as a way to help our data engineers be productive.
397
00:41:53.075 --> 00:41:55.235
Maybe the last one I'll talk about is
398
00:41:55.395 --> 00:41:57.234
the fact that we're moving towards this
399
00:41:57.579 --> 00:41:59.900
lofty vision of self healing pipelines.
400
00:41:59.900 --> 00:42:00.700
There's
401
00:42:00.700 --> 00:42:12.380
something about the Yanis Henderson investors example where we saw it was not just that Auto was able to diagnose the issue in their lighthouse design, but also able to act on behalf of the data engineers so that
402
00:42:12.895 --> 00:42:16.575
you do have this wonderful flow between there's something wrong,
403
00:42:16.895 --> 00:42:22.175
there is a fix for it, and the human just needs to approve whatever the fix is as a way to shorten time.
404
00:42:22.575 --> 00:42:30.440
And these are some of the things that we're working towards across the Astroid platform that we make this capability widely available for all of our customers.
405
00:42:30.760 --> 00:42:39.560
In the next quarter, we're kind of working on the resolution pathway for the different things that that Auto will diagnose as the issue, because it could be as simple as just retry the stack.
406
00:42:40.115 --> 00:42:48.434
It could be as complicated as there was a code change that needs to be made, and it's because of this data issue. Or maybe it's failed. We've
407
00:42:48.434 --> 00:43:00.800
realized that the data quality check has failed, and we need to try and diagnose these things. We really want to make sure that auto is really good at resolution before then extending it into ways where we could say, let's make it more automatic
408
00:43:00.960 --> 00:43:06.865
for our customers. And maybe say, for instance, that if there's an alert that goes off, that auto is immediately
409
00:43:06.865 --> 00:43:16.785
looking into resolving those things for our customers. So we're definitely on the journey towards building towards that self healing future with human approval, of course.
410
00:43:17.265 --> 00:43:20.385
And yeah, I'm excited. I'm really excited to see it brought to life.
411
00:43:21.470 --> 00:43:38.485
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling, technology, or human training that's available for building data systems today.
412
00:43:39.605 --> 00:43:40.485
You've
413
00:43:40.485 --> 00:43:44.485
actually spoken about it quite extensively, which is that context layer
414
00:43:44.565 --> 00:43:51.605
which is missing and how complete that picture is. So I think really the journey for us is not just us as astronomer
415
00:43:51.605 --> 00:43:52.725
and airflow,
416
00:43:52.885 --> 00:43:55.900
is to make sure that we have all of the pieces to answer the question.
417
00:43:56.780 --> 00:44:01.020
Because for whatever your tool stack looks like internally in your organization,
418
00:44:01.500 --> 00:44:07.500
being able to connect those dots with your agents so that they have all the right pieces is actually where you get the most value in the end.
419
00:44:08.395 --> 00:44:10.795
So yeah, I think it's just like,
420
00:44:11.115 --> 00:44:14.875
if I could also advise as well, when you're thinking of constructing
421
00:44:14.875 --> 00:44:16.155
your data platform,
422
00:44:16.395 --> 00:44:42.125
really making sure that it is accessible to these agents as well so that you can be productive. All right. Well, thank you very much for taking the time today to join me and share the work that you and your team are doing on auto and just some of the overall challenges and learnings about bringing agents into the messy world of data. So I appreciate all the time and energy you're putting into that, and I hope you enjoy the rest of day. No. This has been fantastic, and thank you so much for having us. It's been great talking to you.
423
00:44:49.640 --> 00:44:54.120
Thank you for listening, and don't forget to check out our other shows. Podcast.net
424
00:44:54.120 --> 00:45:03.320
covers the Python language, its community, and the innovative ways it is being used. And the AI engineering podcast is your guide to the fast moving world of building AI systems.
425
00:45:03.880 --> 00:45:13.855
Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project from the show, then tell us about it. Email hosts@dataengineeringpodcast.com
426
00:45:13.855 --> 00:45:20.015
with your story. Just to help other people find the show, please leave a review on Apple Podcasts and tell your friends and coworkers.