WEBVTT
00:00:00.080 --> 00:00:02.799
They're going to ask the analyst, I need this data.
00:00:02.959 --> 00:00:06.639
I need the current metrics on this given KPI.
00:00:06.719 --> 00:00:12.560
And if there's something wrong with that data, the analyst would work around that or massage it, maybe let you know.
00:00:12.880 --> 00:00:16.480
But let's say you just put an MCP layer over top of your data.
00:00:16.719 --> 00:00:18.079
They get an answer back.
00:00:18.320 --> 00:00:21.839
And the agent, it sounds like it could be the right answer.
00:00:22.000 --> 00:00:29.039
That agent might not have any idea that there is a data quality issue with that table.
00:00:44.079 --> 00:00:46.560
I'm Prakash Chundran, CEO of Zano.
00:00:47.039 --> 00:00:58.640
Today I'm joined by Heather Lutz, Director of Engineering at DataSight, a SaaS platform for the full MA lifecycle that processes some of the most sensitive financial and legal documents in the world.
00:00:58.880 --> 00:01:02.399
Heather's background is unlike most engineering leaders you'll hear.
00:01:02.560 --> 00:01:14.000
She started as an English teacher in Japan, went through a coding boot camp, got hired as a junior engineer, and over the past decade built out the core data infrastructure that backs Data Sites platform today.
00:01:14.239 --> 00:01:26.480
She introduced Kafka, Airflow, and real-time data flows to the organization, started the analytics engineering and product analytics team, and now leads data infrastructure and engineering across the US and Costa Rica.
00:01:27.599 --> 00:01:30.959
There is a line Heather said that I think every builder needs to hear.
00:01:31.439 --> 00:01:34.719
AI is going to accelerate you in one of two directions.
00:01:34.959 --> 00:01:44.719
It's either going to give you intelligence at a speed you've never had, or it's going to give you wrong answers faster without you recognizing how wrong they actually are.
00:01:45.120 --> 00:01:50.159
The difference, of course, is what's underneath the quality, governance, and structure of your data.
00:01:50.640 --> 00:01:57.599
This is a conversation about the found uh about the foundation that most teams are skipping in the rush to plug in AI.
00:01:57.760 --> 00:01:59.760
Heather, so good to have you here today.
00:01:59.840 --> 00:02:00.480
Thank you for joining.
00:02:00.959 --> 00:02:02.079
Yeah, thank you for the invite.
00:02:02.159 --> 00:02:02.719
This is great.
00:02:03.040 --> 00:02:04.000
Yeah, awesome.
00:02:04.159 --> 00:02:07.040
So I wanted to start with something I mentioned up uh at the top.
00:02:07.200 --> 00:02:15.520
You know, you went from teaching English in Sapporo, Japan to building core data infrastructure for a company that handles some of the most sensitive data in the world.
00:02:15.759 --> 00:02:19.199
Can you just at a high level walk us through that journey?
00:02:19.599 --> 00:02:19.919
Yeah.
00:02:20.159 --> 00:02:24.319
So I did, I graduated with an English degree in about 2008.
00:02:25.120 --> 00:02:32.719
And if you were around then, you know that that was a really rough time to be graduating with an English degree, pretty much any degree.
00:02:32.960 --> 00:02:41.599
Um, so I took the opportunity at the time, I had taken four years of Japanese by that point, and ended up in Japan, uh, teaching English.
00:02:41.759 --> 00:02:46.960
I absolutely loved it and thought I wanted to go into teaching and be in education forever.
00:02:47.280 --> 00:02:53.439
I still enjoy education, um, but I did start to burn out from that position relatively quickly.
00:02:53.680 --> 00:02:58.000
I was in it for a while, moved into higher ed administration.
00:02:58.080 --> 00:03:00.639
I actually have my master's in higher ed.
00:03:01.120 --> 00:03:12.960
Um, and thankfully, during that time as an administrator, I had the privilege of advising software students, folks pursuing a degree in software engineering.
00:03:13.280 --> 00:03:15.120
It was a small cohort of students.
00:03:15.280 --> 00:03:16.400
I got to know them quite well.
00:03:16.479 --> 00:03:21.039
And I also realized that I thought what they were doing was far more interesting than what I was doing.
00:03:21.360 --> 00:03:36.639
So I decided to quit my job, go to a boot camp for a little less than a half a year, try my luck, and very fortunately got picked up by the organization that became Datasite as a junior engineer.
00:03:38.000 --> 00:03:48.639
And moved through a bit, spent some time developing the MongoDB infrastructure for Datasight back when we were still a greenfield.
00:03:48.960 --> 00:03:52.719
Thankfully, my boot camp program had trained me in the mean stack.
00:03:52.879 --> 00:03:58.639
I was one of the only engineers who had even touched MongoDB on the team at that time.
00:03:59.280 --> 00:04:00.960
So I got to work with the data.
00:04:01.039 --> 00:04:03.759
I got to set up that infrastructure from scratch.
00:04:04.000 --> 00:04:16.560
Um, everything from setting up the clusters, downloading the actual instances, doing the installs, the monitoring, indexing, some of the initial schemas, all of it as a junior.
00:04:16.720 --> 00:04:17.439
Um Yeah.
00:04:17.680 --> 00:04:21.439
I remember inspiration for this, you were saying it was like within your first week.
00:04:21.519 --> 00:04:23.519
They're like, who's worked with Mongo before?
00:04:23.680 --> 00:04:30.000
And because of your mean stack trading, you raise your hand and look, it is parlayed into quite a career for you.
00:04:30.560 --> 00:04:31.680
That was that was amazing.
00:04:31.759 --> 00:04:35.439
I I am still surprised the things they let a junior engineer do.
00:04:35.680 --> 00:04:37.759
Um, set up all their data.
00:04:38.000 --> 00:04:40.720
Yeah, that was uh that was an experience.
00:04:40.879 --> 00:04:42.319
But I I did that.
00:04:42.639 --> 00:05:01.439
I was in back end engineering for most of my first part of my career, about five-ish years or so, alternating between working with the data store on the operational side and then doing coding for everything from authorization, authentication, uh document metadata related work.
00:05:01.759 --> 00:05:08.319
And eventually uh they needed somebody to manage the analytical data space.
00:05:08.560 --> 00:05:13.600
That team had spun up about a year prior, and I accepted.
00:05:13.759 --> 00:05:21.439
They also tossed in the operational store with it, and I began managing the entire data stack for the product.
00:05:21.680 --> 00:05:34.480
And eventually I brought in, as you mentioned, an full-on analytics team, an analytics engineering team, started bringing in Kafka streaming for our data, and here I am.
00:05:35.040 --> 00:05:43.279
Here you are, uh a decade later, after just that first week raising your hand in Mongo, and you you're now a data expert.
00:05:43.439 --> 00:05:45.839
Um, this is a conversation about data.
00:05:46.079 --> 00:05:51.920
Arguably, data's always been important, uh, which I know something you're a very strong proponent of.
00:05:52.639 --> 00:05:58.000
But increasingly, especially in the world of AI, uh, it is more and more important.
00:05:58.160 --> 00:06:09.120
And for those that are avid listeners to this podcast, almost every uh organization leader, organizational leader I will talk with says you it always starts with the data.
00:06:09.600 --> 00:06:14.160
So, you know, going back to the quote I mentioned, you said it's going to accelerate in one of two ways.
00:06:14.319 --> 00:06:22.800
Either you're gonna get intelligence at a speed you never thought possible, or you're gonna get the wrong answers faster than you even recognize how wrong they are.
00:06:23.279 --> 00:06:28.720
So I'd like for you, in the context of this data conversation, to unpack what you mean by that.
00:06:29.279 --> 00:06:50.879
So previously at DataSight, if somebody needed information, and when I say somebody, I typically mean a business user, somebody in sales and marketing, somebody over on the product management side, typically what they're gonna do is they're going to ask the analyst, I need this data, I need the current metrics on this given KPI.
00:06:51.040 --> 00:06:52.399
Can you tell me this information?
00:06:52.560 --> 00:06:54.000
Can you put it in a dashboard?
00:06:54.160 --> 00:06:56.639
Can you pull it out of the data store?
00:06:56.879 --> 00:06:58.399
And the analyst will go do that.
00:06:58.480 --> 00:07:08.160
And if there's something wrong with that data, or there's something wrong with that data set, or if there's known quality issues, the analyst would work around that or massage it, maybe let you know.
00:07:08.399 --> 00:07:13.600
But at the end of the day, you would have a hopefully a fairly accurate report.
00:07:13.839 --> 00:07:24.959
Assuming there's not multiple sources of the same thing in the data store, and you know, the analyst is pulling the right information and knows that data quite well.
00:07:26.160 --> 00:07:39.920
But let's say you just put an MCP layer over top of your data, and you would just allow this business leader to query it using the AI tooling of their choice.
00:07:40.160 --> 00:07:43.439
You plug in Claude, you plug in chat.
00:07:44.399 --> 00:07:46.000
They get an answer back.
00:07:46.240 --> 00:07:49.759
And the agent, it sounds like it could be the right answer.
00:07:50.000 --> 00:07:54.160
The agent seems very confident they've analyzed the response.
00:07:54.639 --> 00:08:01.439
That agent might not have any idea that there is a data quality issue with that table.
00:08:01.680 --> 00:08:12.959
That agent, depending upon how that question was asked, might not realize that there are actually three different revenue tables somewhere within the data structure, and it doesn't know which one to look at.
00:08:13.279 --> 00:08:20.639
Or it could be that they're used for two slightly different questions, and it doesn't have the context around that.
00:08:20.879 --> 00:08:32.799
So your business user just got the wrong information, but it looks awfully correct because that agent is awfully confident.
00:08:33.120 --> 00:08:41.120
I have one of my engineering leaders say that the agents right now are like the extremely intelligent but overly confident intern.
00:08:41.679 --> 00:08:46.480
Very talented, very smart, but they don't know what they don't know.
00:08:48.000 --> 00:08:49.440
They don't know what they don't know.
00:08:49.600 --> 00:09:06.080
I think that is really important because I do think there is still um, I don't I don't know if stigma is the right word, but definitely a thought that you can plug an agent into a data set, no matter how large and it will figure itself out.
00:09:06.399 --> 00:09:12.720
Sure, it might take a little bit more time to think, but it's gonna figure it out for you and it's gonna come back to you with a confident answer.
00:09:13.120 --> 00:09:16.240
You actually ran an experiment, if I remember this correctly.
00:09:16.320 --> 00:09:21.120
You said you sent an agent to Snowflake with 22,000 tables and two petabytes of data.
00:09:21.440 --> 00:09:36.159
Um talk to me a little bit more about what that experience was like, what you learned from it, and um and then from those learnings, how you decided to segment out agent access to data moving forward.
00:09:37.039 --> 00:09:37.440
Yeah.
00:09:37.600 --> 00:09:41.120
So we actually did not give it access to the full 22,000 tables.
00:09:41.279 --> 00:09:44.240
We did give it an access to a subset of it.
00:09:44.399 --> 00:09:47.600
We gave it access to our usual consumable layer.
00:09:47.679 --> 00:09:50.240
We gave it access to our analyst layer, basically.
00:09:50.480 --> 00:09:53.360
So the layer that the analysts typically use.
00:09:53.759 --> 00:10:01.360
And we let a business user set up connection to the MCP and start asking it questions.
00:10:01.600 --> 00:10:05.440
And thankfully, we paired one of our analysts with this individual.
00:10:05.840 --> 00:10:08.559
And uh honestly, even the business user knew that was wrong.
00:10:08.639 --> 00:10:11.679
That that particular business user knew his data pretty well.
00:10:11.840 --> 00:10:14.879
They look at these KPIs, they look at these numbers all day.
00:10:15.120 --> 00:10:21.279
It it it wasn't even close, mostly because there's additional filters that just aren't being applied.
00:10:21.360 --> 00:10:31.440
There's terminology that the business user might be using that, you know, that that column name, that table name's not quite the same thing.
00:10:31.840 --> 00:10:37.759
So it was very evident that the agent was making some guesses.
00:10:38.159 --> 00:10:39.759
Um it did give an answer.
00:10:40.799 --> 00:10:41.519
You got one.
00:10:41.840 --> 00:10:47.440
It was it was not the correct answer, and it would certainly not be something we would have put on a on a report.
00:10:48.000 --> 00:10:48.399
Yeah.
00:10:49.200 --> 00:11:02.159
Um you know, I think this is even though you might have a much larger data set than uh what people are traditionally used to, um, I think there's just a thought of like really kind of figuring out where to start.
00:11:02.320 --> 00:11:12.000
If people are generally aligned with the fact that like you can't just throw a large data set at an agent, well, what can you throw uh at it?
00:11:12.240 --> 00:11:28.399
And um later on, I'm sure we'll talk about like the beginnings of it, but like how do you at Data Site and as a data leader think about how to partition either different agents, different MCPs, different roles, and what they get access to?
00:11:29.200 --> 00:11:29.440
Sure.
00:11:29.759 --> 00:11:55.279
Well, first of all, if you want to put the entire landscape of, let's just say your consumable data in front of the agent, if you have somebody using a skill or somebody writing a skill who really knows what they're doing, can get detailed into it, can specifically point to the exact tables to look at, the exact information to filter on, you actually will get a half decent answer.
00:11:55.360 --> 00:11:58.960
This is assuming that the person who wrote that skill knows what they're doing.
00:11:59.200 --> 00:12:03.919
And you're probably going to repeat some of the same instructions over and over and over again.
00:12:04.080 --> 00:12:09.120
And eventually that's going to start to become a little bit untenable.
00:12:09.279 --> 00:12:12.879
Skills are typically meant to answer a how question.
00:12:13.120 --> 00:12:21.919
Um, and you're essentially forcing it to answer also, or instructing it to answer a lot of what type questions as well, in that case.
00:12:22.159 --> 00:12:32.639
So from there, how do you start going and chunking this out so that the agent doesn't have 20 something thousand tables to go and look at?
00:12:33.360 --> 00:12:37.600
Well, first of all, taking a look by domain and by role.
00:12:37.919 --> 00:12:44.960
Most people who are pooling information in an organization, they are pulling this information by some sort of domain.
00:12:45.120 --> 00:12:55.360
If you are working in product, you are probably pulling things around feature usage, metrics around the usage for the components in the application.
00:12:55.600 --> 00:13:00.240
Hopefully, you probably have some kind of OKR as you want that information out of it.
00:13:00.320 --> 00:13:03.039
You want to know how many people are using your product.
00:13:03.200 --> 00:13:07.519
Or if you're in marketing, you maybe you're doing some cross-selling information.
00:13:07.759 --> 00:13:14.000
But you are typically working out of a general bucket of information.
00:13:14.559 --> 00:13:18.240
So segment your information from a data perspective.
00:13:18.559 --> 00:13:24.960
You should have ownership by domain in a perfect world with probably some cross-dependencies underneath.
00:13:25.200 --> 00:13:28.639
But it's the same thing with the MCP and with the agent.
00:13:29.679 --> 00:13:33.840
Give it a surface of data where it's going to be the expert.
00:13:34.240 --> 00:13:40.879
I literally at one point thought to call our product agent and name it after like our analyst.
00:13:40.960 --> 00:13:44.240
I'm like, this is essentially an analyst's companion for product.
00:13:44.320 --> 00:13:45.840
That's that's what this is doing.
00:13:46.080 --> 00:13:50.000
So it's over top of this particular domain.
00:13:50.240 --> 00:13:53.679
Hopefully, that domain is also fairly well curated.
00:13:54.799 --> 00:14:03.519
If you have your gold consumable layer of data and you say this is my data product, and I don't necessarily mean it's client-facing in that sense.
00:14:03.679 --> 00:14:06.480
I mean this is the data that I care about.
00:14:06.639 --> 00:14:09.200
It's the one that I'm saying it's ready to use.
00:14:09.440 --> 00:14:12.320
It's of a quality that I am satisfied with.
00:14:12.559 --> 00:14:15.919
It is well documented, it is governed.
00:14:16.159 --> 00:14:23.919
Ideally, you have some kind of semantic layer or semantic views in the case of Snowflake on top of this.
00:14:25.200 --> 00:14:33.679
That that's a good surface to then try and put an agent over top of that or put an MCP on top of that.
00:14:33.919 --> 00:14:37.759
That also is attached typically to your RVAC solution.
00:14:38.000 --> 00:14:43.039
So your users in that area should also only have access to that date.
00:14:43.440 --> 00:14:45.919
Yeah, there's a couple threads I want to pull on there.
00:14:46.000 --> 00:15:07.440
And one of the first things that I want to recognize is that um before you define the skill file or anything like that for the special uh speciality of what that agent uh pulls and how it pulls it and what it pulls, talking about all of this pre-work that needs to be done before we even get there.
00:15:07.600 --> 00:15:10.159
You're even using words like governance, right?
00:15:10.559 --> 00:15:12.480
In an ideal world, yes.
00:15:12.879 --> 00:15:13.919
In an ideal world.
00:15:14.159 --> 00:15:17.840
But broadly speaking, you need to be able to have the data.
00:15:18.000 --> 00:15:25.759
Um, I think you said this, and probably you said you have to be able to cure curate it, maintain it, and make it consumable or accessible.
00:15:26.320 --> 00:15:27.679
It's like an API.
00:15:27.759 --> 00:15:32.159
If you're writing a piece of software, you are putting something into production.
00:15:32.639 --> 00:15:34.799
In this case, it's data.
00:15:35.039 --> 00:15:45.519
If you it's not ready for somebody to read and use, ideally it shouldn't be in that, like curate it like it's an think of it as it's an API.
00:15:45.679 --> 00:15:46.960
You are delivering a product.
00:15:47.440 --> 00:15:49.120
So I want to talk about two separate things.
00:15:49.279 --> 00:15:52.639
I want to talk about data readiness and data governance.
00:15:53.039 --> 00:15:53.360
Correct.
00:15:53.600 --> 00:15:55.200
Let's start with data readiness.
00:15:55.360 --> 00:16:00.320
When you say if the data is not ready, what does that mean to you?
00:16:00.799 --> 00:16:10.559
So to me, we're saying if the data is not ready, this means that you have some kind of CICD and testing process sitting here involved.
00:16:10.879 --> 00:16:16.159
So for instance, if you have a column that is never supposed to be null, you've checked that.
00:16:16.320 --> 00:16:20.879
You've checked that before it's even landed anywhere in production.
00:16:21.440 --> 00:16:29.519
Um, if you have a certain set of options, enums for a given set, you know what those are.
00:16:29.679 --> 00:16:32.559
You know it's not supposed to be outside of those parameters.
00:16:32.799 --> 00:16:40.480
You know that if you're joining two pieces of information, you're expecting it to be within a certain range of values.
00:16:40.720 --> 00:16:52.159
You've already checked that before your CICD process has written that into the productize version of your tables.
00:16:52.480 --> 00:16:55.840
So all of that is in there, and that's just data in motion.
00:16:55.919 --> 00:17:00.879
That's data as it's landing into that consumable environment.
00:17:01.039 --> 00:17:06.160
Ideally, you also have some data quality over top of that longer term.
00:17:06.319 --> 00:17:13.920
So you're not just monitoring, hey, I don't expect this tape column to be null, except maybe in these circumstances.
00:17:14.880 --> 00:17:18.160
Okay, great, it landed in pro, but what about over time?
00:17:18.799 --> 00:17:29.519
Like let's say all of a sudden I'm seeing more null values than I would expect for this type of value.
00:17:30.079 --> 00:17:40.319
Or hey, I'm all of a sudden seeing a great increase in this type of for the data site process, this type of project.
00:17:40.400 --> 00:17:42.000
But what does that mean?
00:17:42.160 --> 00:17:44.400
Um, so it's changing an expectation.
00:17:44.480 --> 00:17:46.640
It's going outside of what you would expect.
00:17:46.720 --> 00:17:49.039
It's outside of a standard deviation.
00:17:49.200 --> 00:17:54.480
So you're measuring it both in terms of can I put this data into prod?
00:17:55.200 --> 00:17:58.480
Can I promote this into a working layer?
00:17:58.799 --> 00:18:15.519
Plus, and this is something that you don't necessarily, I think, measure as much in typical software engineering, but over time, I'm looking and examining how my data set is changing, possibly in ways that I don't necessarily expect.
00:18:15.839 --> 00:18:21.680
And I think the interesting part there, I think when we were talking earlier, you know, you have a back-end engineering background.
00:18:22.000 --> 00:18:25.759
With back-end engineering, you have a CI CD and you have a testing process.
00:18:26.000 --> 00:18:26.160
Yes.
00:18:26.400 --> 00:18:32.319
Um, with data, that is not necessarily a standard, like the continuous testing of data quality.
00:18:32.960 --> 00:18:42.960
Um I'd love for you to talk a little bit more about that because I that was something that I remember that was surprising to me, that that was like something that you felt very strongly about.
00:18:43.200 --> 00:18:55.119
You touched on it a little bit, but I'd love for you to say more about why more people need to make it a standard, um, especially in a world where you have agents and AI accessing your data.
00:18:55.440 --> 00:18:56.000
Correct.
00:18:56.240 --> 00:19:03.039
I mean, essentially, once you have AI accessing your data, again, whatever they're accessing, that that's prod.
00:19:03.440 --> 00:19:10.240
So if you wouldn't want this committed into your application, don't put it into your data set either.
00:19:10.480 --> 00:19:14.240
One of the questions I almost always ask in an interview, and it's very general.
00:19:14.400 --> 00:19:16.559
So if you ever interview with me, take note.
00:19:17.039 --> 00:19:19.279
How do you test data?
00:19:19.680 --> 00:19:24.720
And you should have hopefully an opinionated answer or make a good one up.
00:19:25.200 --> 00:19:40.799
But you have tests that appear at various places, and you will now see this even in some of the tooling that you have, whether you're using Snowflake or Databricks, they're starting to build in data quality metrics and monitoring into their tool sets.
00:19:41.039 --> 00:19:45.039
DBT, if you're using it, has testing inside of it.
00:19:45.359 --> 00:19:55.359
Airflow, you are able to detect when a given um DAG fails, why it fails, the logs for it.
00:19:55.599 --> 00:20:01.039
Um, so you should have testing and the orchestration of the data.
00:20:01.200 --> 00:20:15.119
So as the data is, as the trigger goes off, you should be able to test as the data is then transformed and landed, transformed, joined, and landed, and you're confirming it at every point.
00:20:15.279 --> 00:20:25.119
And then you're going to essentially monitor again at the landing zone over a period of time that that data again is meeting your expectations.
00:20:25.440 --> 00:20:29.519
Like you have CI C D, the orchestration, that's your CI CD process.
00:20:29.680 --> 00:20:32.960
Did it, did it, did it also fail or something like that?
00:20:33.119 --> 00:20:41.359
Um, you can almost think of your, I guess your landed monitoring kind of like an end-to-end test or a full stack test.
00:20:41.519 --> 00:20:46.559
So you're seeing it, I view it like it is like a software.
00:20:47.039 --> 00:20:50.720
If it's important enough, I should be testing it and making certain it's okay.
00:20:51.039 --> 00:20:55.119
Well, what's really interesting, at least not that I'm aware of it, and of course I am not a data expert.
00:20:55.279 --> 00:20:57.119
I just don't think that there's a standard around this.
00:20:57.279 --> 00:21:02.400
Like you just made some like correlations around like unit tests and DBT that totally make sense to me.
00:21:02.559 --> 00:21:18.559
But like in the world of like data CICD or data pipeline, um, like the quality standard around how you continuously test that, I don't think is something that people are thinking about, but they will start thinking about it more moving forward, I think.
00:21:18.880 --> 00:21:20.640
I do hope that comes out more.
00:21:20.799 --> 00:21:26.240
I have noticed, like, you will find walkthroughs for how to create like DBT tests.
00:21:26.480 --> 00:21:32.000
You will certainly, you certainly have tools like Datadog and Monte Carlo that will monitor data for you.
00:21:32.160 --> 00:21:41.119
But you're right, I I have not seen too many references to like an end-to-end testing process for data, like actually written down.
00:21:41.279 --> 00:21:50.319
Not certainly not as many as I see for software engineering, where you have whole things like test driven development that have built out there.
00:21:50.480 --> 00:21:52.880
I I don't think it's as big as that.
00:21:53.200 --> 00:21:53.759
Yeah.
00:21:54.000 --> 00:21:54.319
Okay.
00:21:54.559 --> 00:21:59.759
So we we uh I'm sure we could talk about each one of these topics for uh much longer, but we talked.
00:22:00.160 --> 00:22:02.160
Talked a little bit about data readiness.
00:22:02.400 --> 00:22:04.240
Let's talk about data governance.
00:22:04.480 --> 00:22:04.720
Correct.
00:22:05.119 --> 00:22:06.400
What does that mean to you?
00:22:07.200 --> 00:22:20.640
So, data governance to me is the method of making your data accessible and usable and understandable and consistent to everybody.
00:22:21.039 --> 00:22:27.039
So now you're stepping outside of your own domain, and this all needs to make sense holistically.
00:22:27.359 --> 00:22:38.640
So this means that, first of all, I hope if you have data for product, you have one team that is responsible for data for product.
00:22:38.720 --> 00:22:41.839
So this goes back to the idea of domain ownership.
00:22:42.079 --> 00:22:43.759
Um, who's in charge?
00:22:44.079 --> 00:22:54.000
This also goes to make certain around, like, for instance, hey, we're talking about let's say total number of pages.
00:22:54.319 --> 00:22:55.759
A simple metric.
00:22:56.240 --> 00:22:59.599
Okay, what does total number of pages actually mean?
00:22:59.759 --> 00:23:03.200
Is that the total number of pages on a project?
00:23:03.440 --> 00:23:06.799
Is that the total number of pages that were processed?
00:23:07.039 --> 00:23:12.960
Is that the total number of pages that are available to a certain segment of users?
00:23:13.119 --> 00:23:14.559
What does that actually mean?
00:23:14.720 --> 00:23:18.559
And everybody then has to agree on that particular definition.
00:23:18.799 --> 00:23:27.599
This is one of the phenomena around how people in the organization can get a report on supposedly the same thing and get two totally different answers.
00:23:27.759 --> 00:23:35.200
That's because multiple people interpreted that information in two completely different ways.
00:23:35.440 --> 00:23:44.160
Or the person requesting the data thought they were asking for one thing, and the person filling out that request thought it was something completely different.
00:23:44.400 --> 00:23:49.119
So it's that agreement, even around how we use these terms.
00:23:50.160 --> 00:23:56.240
And then there, I think there is that overlap in data quality as well, but it's far more holistic.
00:23:56.480 --> 00:24:01.599
It's also making certain because data is actually most important when it's used together.
00:24:01.680 --> 00:24:06.000
This is actually, I think, one of the biggest differences between data and software engineering.
00:24:06.240 --> 00:24:13.519
Software engineering, you want, you want to try and keep that tight ownership, but like loosely cohesive.
00:24:13.759 --> 00:24:17.200
Data, data is most useful when it's with other data.
00:24:17.440 --> 00:24:27.200
So you also really want to make certain that you understand where that data is coming from and tracking that lineage and in some governance structures, even down to like a field level.
00:24:27.440 --> 00:24:30.559
So really understanding the origin of that data.
00:24:30.720 --> 00:24:42.640
But it's it at the heart of it, it's putting the tools in people's hands to make certain that they have enough information about the data that they are using to make very good logical business decisions.
00:24:43.119 --> 00:24:53.680
When we talk about data governance, it's, you know, I think holistically about what you're saying about governance in terms of its accessibility to everyone.
00:24:54.640 --> 00:25:04.799
But also in terms of access, you've got the platform layer, you've got the data layer, um, you know, you've got um uh the the application layer.
00:25:05.279 --> 00:25:07.359
Where does governance actually start?
00:25:07.519 --> 00:25:15.119
And especially in a world maybe we're getting into harness territory here, where do the instructions for these agents uh live, right?
00:25:15.200 --> 00:25:16.559
Like how do you think about that?
00:25:16.880 --> 00:25:19.200
Okay, well, where does governance actually start?
00:25:19.279 --> 00:25:23.039
I am, first of all, I am I am not in charge of governance.
00:25:23.359 --> 00:25:32.400
I leave leave most of that to our governance team at data site, but I am fully inspired by them and I take what they say to heart.
00:25:32.720 --> 00:25:40.000
But an ideal world, they would say governance starts as soon as that data lands in an ideal perfect world.
00:25:40.240 --> 00:25:46.319
Um, most people don't start governance as uh soon as their data is up and running.
00:25:46.400 --> 00:25:50.400
You ideally you would have started governance when your product and everything was a green field.
00:25:50.559 --> 00:25:50.720
Yeah.
00:25:50.960 --> 00:25:52.480
If you did, good job.
00:25:52.720 --> 00:25:54.640
Um that's perfect.
00:25:54.880 --> 00:25:58.240
That is not, I don't think that's most of us.
00:25:58.480 --> 00:26:05.920
Um so for us, the way we have chosen to start is from our client, even if that's internal facing layer.
00:26:06.079 --> 00:26:08.000
What's your gold standard of data?
00:26:08.240 --> 00:26:19.599
What are the top 10 metrics and reports that you are responsible for that are the most important thing to your area of the organization?
00:26:19.759 --> 00:26:20.799
Start with that.
00:26:21.440 --> 00:26:23.839
Um, start with those particular metrics.
00:26:23.920 --> 00:26:35.759
And typically you will find that there's a series of metrics, tables, structures that underpins a large portion of what you do.
00:26:36.559 --> 00:26:41.519
Document that, solidify that, and start from there.
00:26:41.839 --> 00:26:50.000
Because once you find, I think, those key structures and you start from that point, many other things become easier.
00:26:50.079 --> 00:26:59.039
And you might actually get to the point where you don't necessarily need to document, analyze, and pick apart absolutely everything.
00:26:59.279 --> 00:27:06.000
Um, like for instance, our bronze layer of data, our landed layer, we sink it.
00:27:06.079 --> 00:27:07.920
No one's gonna see that other than us.
00:27:08.319 --> 00:27:10.960
And hopefully we know what we're doing.
00:27:11.119 --> 00:27:12.160
We hope we know.
00:27:12.400 --> 00:27:15.519
Um that layer we don't really worry about.
00:27:15.680 --> 00:27:30.079
We worry predominantly about that consumable layer of data, that gold and platinum layer where we know that we have folks from other areas of the org, other analysts doing that work.
00:27:30.319 --> 00:27:35.119
Or in this case, this is where our agents are actually plugged in.
00:27:35.359 --> 00:27:44.079
So we have curated a very bounded layer where we have said everything in here is documented.
00:27:44.319 --> 00:27:50.480
There is a semantic layer in the case of Snowflake semantic views built over top of this.
00:27:50.799 --> 00:27:58.640
We actually built those views when we started designing the product itself that became this layer.
00:27:58.799 --> 00:28:06.640
So it was actually built out first before we even started coding that newer version of our layer.
00:28:06.880 --> 00:28:11.759
Um, anything new immediately has definitions coming in.
00:28:12.160 --> 00:28:21.599
So, this way, the most important things in your org plus anything going forward ends up getting governed, ends up in that consumable layer.
00:28:21.680 --> 00:28:28.480
And it's gonna be a lot easier for an agent then to roll through that and understand it.
00:28:29.359 --> 00:28:30.000
Yes.
00:28:30.240 --> 00:28:32.880
I mean, that makes a lot of sense.
00:28:33.119 --> 00:28:36.880
And I think that semantic layer is hugely critical.
00:28:36.960 --> 00:28:45.279
And I think your approach around that, you know, starting from the client, the gold standard data, uh, top 10 metrics and reports.
00:28:46.000 --> 00:28:55.039
I am still interested, though, in the cross-functional piece because giving agents access to the instructions on how to query the data.
00:28:55.680 --> 00:29:00.000
It's not like a single player game that the semantic layer can just solve on its own.
00:29:00.799 --> 00:29:04.720
How how are you handling that today at Data Site?
00:29:05.920 --> 00:29:10.160
So over top of this, the semantic layer is it helps.
00:29:10.400 --> 00:29:13.359
It helps a lot, but it's not everything.
00:29:13.680 --> 00:29:21.359
So this is where we're starting to dig into other pieces of information coupled with it.
00:29:21.519 --> 00:29:25.359
So if you're using Snowflake, you have something, for instance, called a verify query.
00:29:25.680 --> 00:29:32.079
So as we have users start to use this layer, I can't see the prompts.
00:29:32.400 --> 00:29:37.680
We can see everything that at least Snowflake is querying on.
00:29:37.839 --> 00:29:42.799
So we can actually go and say, oh, this is queried very frequently.
00:29:42.880 --> 00:29:45.200
Let's create a verified query.
00:29:45.359 --> 00:29:52.000
So essentially training it and saying, okay, when you see this request, this is what you should be running.
00:29:52.160 --> 00:29:52.480
Got it.
00:29:52.640 --> 00:29:57.920
And that will also help at them with very similar queries, similar filters.
00:29:58.079 --> 00:30:01.759
So that is a verified query where it's saying, I know this is right.
00:30:02.000 --> 00:30:04.640
This is the answer that you should then expect.
00:30:05.759 --> 00:30:06.799
Tune from there.
00:30:06.960 --> 00:30:14.000
We can also, we also monitor these agents as well for their accuracy against these verified queries.
00:30:14.160 --> 00:30:19.759
So as that's coming in, as we're deploying a new version of the agent, we actually test on that too.
00:30:19.920 --> 00:30:22.160
Like how closely did you hit these?
00:30:22.319 --> 00:30:25.039
Um, and if it fails, we won't even deploy it.
00:30:25.359 --> 00:30:32.880
Um however, even going back further than that, so you've got the verified queries, you've got your semantic layer.
00:30:33.200 --> 00:30:37.680
We have also started to dig into skills as well.
00:30:38.160 --> 00:30:45.599
So we do have skills out for our um individual sets of users.
00:30:45.759 --> 00:30:57.119
So certain filtering requirements, certain things that almost like what the analyst is trained to ask to they include this in the request?
00:30:57.279 --> 00:31:00.559
If they didn't, please prompt them for it because it'll make it better.
00:31:00.799 --> 00:31:03.519
Um, so that would be like certain filters.
00:31:03.680 --> 00:31:06.880
We also train um the agent itself on synonyms, too.
00:31:07.039 --> 00:31:11.279
Like certain synonyms, like we might call it a project, we might call it a data room.
00:31:11.359 --> 00:31:14.000
Those are synonyms they could use either one.
00:31:14.400 --> 00:31:16.720
So that's layered on top of it.
00:31:16.960 --> 00:31:23.200
There's also potentially another layer we're considering more on like the langsmith side.
00:31:23.599 --> 00:31:33.279
Um, there was a thought, once again, coming from our lead developer, of what we call the data doorman, which is maybe you actually don't need Snowflake.
00:31:33.759 --> 00:31:36.559
Maybe you don't actually need that data set.
00:31:36.720 --> 00:31:40.240
Maybe what you're actually looking for is click data.
00:31:40.480 --> 00:31:43.119
Maybe what you need is actually pendo data.
00:31:43.200 --> 00:31:51.359
Or, you know what, if you're acting about an active project, maybe I could actually send you to our operational data store and that would be more efficient.
00:31:51.839 --> 00:32:04.559
Dorman will help you with that question so that I can direct you to the correct place and you don't even have to ask us anymore which data sets and which tools you should be looking at.
00:32:04.720 --> 00:32:06.640
We'll just direct you there.
00:32:06.799 --> 00:32:10.319
Um, so that's another thing we are experimenting with.
00:32:10.480 --> 00:32:13.680
So there are multiple layers in this.
00:32:14.480 --> 00:32:22.799
And we tried to keep the snowflake agent portion very specific to the snowflake data itself.
00:32:23.039 --> 00:32:23.599
Got it.
00:32:23.839 --> 00:32:27.440
It's about the snowflake data, about the semantic.
00:32:27.599 --> 00:32:28.559
It's there.
00:32:28.880 --> 00:32:38.960
The layer before that, that skill layer, has tended to be more about the business user, how they're asking, how they're interacting.
00:32:39.200 --> 00:33:01.599
And then there's yet another layer around, okay, well, maybe we need to even redirect you to a completely different location, or perhaps between a different agent in Snowflake, because maybe you're an executive and you're using this agent normally, but you need to ask about something in product, we'll just redirect you to the product agent.
00:33:01.920 --> 00:33:04.319
And you're an executive, so you can know that information.
00:33:04.640 --> 00:33:04.880
Yeah.
00:33:05.039 --> 00:33:05.359
Yeah.
00:33:05.519 --> 00:33:09.680
I I love um I love the way that you are approaching it.
00:33:09.759 --> 00:33:17.039
And I love the concept of a data doorman that kind of ushers you into the right kind of uh set to be looking at.
00:33:17.279 --> 00:33:19.200
Um who owns this?
00:33:19.359 --> 00:33:27.200
Like because it because it is so cross-functional, um who is currently owning this at DataSight if you were allowed to share?
00:33:27.599 --> 00:33:30.799
Ah that starts to get a little complicated.
00:33:30.880 --> 00:33:34.079
So, but let's go into ideally who owns it.
00:33:34.240 --> 00:33:34.480
Yeah.
00:33:34.640 --> 00:33:40.240
So if you own a domain, you've curated the domain, guess what?
00:33:40.319 --> 00:33:41.359
You own the agent.
00:33:42.000 --> 00:33:46.240
It's it's it is for your analysts, for your services.
00:33:46.400 --> 00:33:47.920
You are the best person to train it.
00:33:48.000 --> 00:33:48.960
Ideally, you own it.
00:33:49.039 --> 00:33:57.759
Or if you don't own it, you're at least heavily directing and having influence on the engineering team that is doing it, would be my opinion on it.
00:33:58.079 --> 00:34:01.839
Um, however, we are not alone in this.
00:34:02.160 --> 00:34:05.279
Um, like for instance, we're working on this doorman concept.
00:34:05.359 --> 00:34:06.480
We don't own Lang Smith.
00:34:06.640 --> 00:34:08.800
That actually belongs to our AI platform team.
00:34:08.960 --> 00:34:17.039
They they have the AI platform and we work with them in concert to get features and functionality like this out the door.
00:34:17.519 --> 00:34:21.599
So at a really high level, that's that's the ownership.
00:34:21.679 --> 00:34:27.840
But we are still in such an experimental phase that we have not fully fleshed it out.
00:34:27.920 --> 00:34:29.360
We are still testing this.
00:34:29.519 --> 00:34:36.320
And currently we have released it to just internal product clients who are trying it out.
00:34:36.880 --> 00:34:41.360
Well, one thing is for sure, everyone is still experimenting and evolving.
00:34:41.440 --> 00:34:49.119
But at least you have some opinionated or prescriptive ways of what you've seen work and certainly what you have seen not work.
00:34:49.360 --> 00:35:10.559
Um, and in that spirit, I think you made you know a comment earlier that, you know, obviously it feels easy to build right when you're starting from scratch and it's green from but a lot of um application development leaders might be listening to this and they might be part of organizations that have been collecting data for years, decades, right?
00:35:11.039 --> 00:35:20.480
So I I'm curious if you have a perspective around what do they start implementing now to start gaining clarity?
00:35:20.639 --> 00:35:23.599
Like where would you recommend that they begin?
00:35:24.719 --> 00:35:29.760
This may seem like a really obvious spot, but do you know what you're asking for?
00:35:30.719 --> 00:35:32.719
Would be the first thing I would say.
00:35:32.960 --> 00:35:46.159
Do you have a good handle on what metrics you need, what those metrics actually represent, and what the metrics, the definition of those metrics are?
00:35:46.400 --> 00:35:49.840
Do you know what you should be tracking regularly?
00:35:50.079 --> 00:35:57.679
Do you know what you should be tracking as potentially new functionality comes out, or as functionality shifts?
00:35:57.920 --> 00:36:05.199
Um, so having a good handle on that is really helpful for just everybody involved.
00:36:05.360 --> 00:36:19.920
So as we're building out the data sets, as we're building out these layers, getting a good grasp on what actually we're supposed to be measuring, because you also have limited space and time for semantic layers, et cetera.
00:36:20.000 --> 00:36:21.599
Like so they can only be so large.
00:36:21.760 --> 00:36:26.400
They can only hold so much before you have to divide them up again, or they won't perform very well.
00:36:26.719 --> 00:36:34.559
Um so making certain you have a handle on that would actually be one of the first things I would say to do.
00:36:35.119 --> 00:36:48.400
Um and then outside of that, once you have a good handle on the metrics themselves, I would say having an understanding of a flow of how you want to interact with those metrics.
00:36:48.559 --> 00:36:54.880
So, like I'm starting on the front end, the front, the front end, um, like this would be an application.
00:36:55.599 --> 00:37:00.639
Do you see yourself interacting with an agent and asking these questions via MCP?
00:37:00.719 --> 00:37:02.880
Or actually, would you prefer a dashboard?
00:37:03.119 --> 00:37:10.559
Um, one of the things our doorman actually does is if metrics exist in a dashboard, we'll just send you to the dashboard.
00:37:10.639 --> 00:37:12.079
Why waste the tokens?
00:37:12.559 --> 00:37:14.639
Why, why consume more than we have to?
00:37:14.880 --> 00:37:23.360
We will send you to where those metrics are actually persisted, and you can just go and look at it rather than spending the extra money.
00:37:23.519 --> 00:37:30.719
Um, but even then, having an idea of that flow also figures okay, well, what kind of a skill am I building out?
00:37:30.880 --> 00:37:35.519
What kind of functionality am I building out for you to support you in doing this?
00:37:35.840 --> 00:37:38.400
Um that does help.
00:37:38.719 --> 00:37:53.679
And then if you haven't created a curated layer of data, if you threw it all in the data layer and hoped for the best, um, creating that layer once again, starting with, okay, well, here's my most important structures to have.
00:37:53.840 --> 00:37:56.159
Here's my most important things to surface.
00:37:56.320 --> 00:38:28.400
And even if you're manually putting it out there, if you don't have any kind of CICD process or you're triggering a some kind of cron job for DBT or whatever transformation you're doing, um just making certain that you are getting your monitoring and hands and your quality around that key layer, building it up, setting up the basic monitoring, even the basic testing around landing it in prod, as we discussed, is is helpful.
00:38:28.559 --> 00:38:31.840
And you'll slowly start building up the rest of it from there.
00:38:32.320 --> 00:38:32.960
Yeah.
00:38:33.360 --> 00:38:40.800
I think one of the key things there is like, you know, you can work with like a bounded set, right?
00:38:40.880 --> 00:38:42.719
It doesn't have to be everything all at once.
00:38:43.119 --> 00:38:44.159
Don't work with the ocean.
00:38:44.239 --> 00:38:45.920
It's I actually recommend not.
00:38:46.400 --> 00:38:48.159
Yes, don't don't work with the ocean.
00:38:48.239 --> 00:38:50.159
That's good right, that's a good recommendation.
00:38:50.400 --> 00:38:58.159
Um, and then you know, uh in that bounded set, I think you can do a lot of the recommendations that we've discussed here today.
00:38:58.480 --> 00:39:05.760
Make sure that that set is is governed, like that readiness that we were talking about, the governance that we were talking about.
00:39:05.920 --> 00:39:12.719
And then also on top of that, you you kind of started with this, but like what are you asking for, which bleeds into the definitions?
00:39:12.800 --> 00:39:16.880
Because if you're confused, I think the agent's gonna be confused, right?
00:39:17.280 --> 00:39:17.840
Yes.
00:39:18.079 --> 00:39:23.280
If you're confused, the agent is not really gonna necessarily make it any clearer.
00:39:23.440 --> 00:39:26.559
In fact, it it might lead you to further wrong answers.
00:39:26.719 --> 00:39:39.360
So the more pinpointed you can be in your requests, in your definitions, in what you are asking for, the more clarity we are going to be able to give back to you.
00:39:39.519 --> 00:39:52.719
And honestly, that even applies, it's more important for regenics, I think, and for these uh new processes as you have the business users themselves starting to interact with the data more.
00:39:52.960 --> 00:40:04.320
But even prior to that, it's still helpful for even an analytics team or folks who are just creating dashboards, having a really clear idea on what you're asking for, realizing that it's important.
00:40:04.400 --> 00:40:07.760
It's just not something you're gonna ask for once and toss away.
00:40:07.920 --> 00:40:19.280
It it is something important for you to track over time and having a good grasp on what those metrics are versus the things that I'm just gonna check once and then turn around and that's it.
00:40:19.599 --> 00:40:20.239
Yeah.
00:40:20.719 --> 00:40:40.400
Um, you know, as we start to wrap up, I'd love to hear a little bit more on what you are excited about in the landscape of data, in the landscape of analytics and, you know, data hygiene, especially when it comes to I actually think data, this is gonna sound weird.
00:40:40.559 --> 00:40:41.679
It's like having a renaissance.
00:40:41.760 --> 00:40:49.679
Like I think people are realizing the importance of this data hygiene, you know, at the foundation of what's going to power AI.
00:40:49.920 --> 00:40:55.519
I don't think I've ever had so many conversations about like foundationally your data has to be, right?
00:40:55.599 --> 00:40:56.400
What is that saying?
00:40:56.559 --> 00:40:58.719
Crap in, crap out type of thing, right?
00:40:59.039 --> 00:41:00.559
Garbage in, garbage out.
00:41:00.719 --> 00:41:01.360
Still applies.
00:41:01.679 --> 00:41:02.239
There it is.
00:41:02.400 --> 00:41:05.119
Because you live in this world, what are what are you looking forward to?
00:41:05.280 --> 00:41:06.159
What are you thinking about?
00:41:06.239 --> 00:41:07.760
What are you excited about?
00:41:08.320 --> 00:41:21.360
I think the thing that excites me most is just people getting great greater clarity on their data itself and really understanding their own needs and what they're asking for.
00:41:21.599 --> 00:41:34.079
Um it's it's really not easy, but it's really common to have organizations where data is very siloed, it's kind of all over the place.
00:41:34.239 --> 00:41:38.639
You have to really shift and sift through it to find its value.
00:41:38.800 --> 00:42:02.800
I'm really looking forward to people putting effort into their data, into their governance, um, and actually having data provide the clarity and insight that it is fully capable of without having to worry about whether or not that those are the right, right metrics, right information.
00:42:03.199 --> 00:42:08.960
Um, it's just I'm really looking and excited about that level of confidence and clarity.
00:42:09.440 --> 00:42:23.280
Um, it is oftentimes historically very difficult to get people excited about doing data definitions and doing um data cleanliness.
00:42:23.679 --> 00:42:34.480
I still don't think people are going to be absolutely excited about it, but I think the need for it will at least um convince people to do it.
00:42:35.039 --> 00:42:35.679
In essential.
00:42:36.559 --> 00:42:46.960
We've covered so much today, a lot of tactical advice around like how to get started with data cleanliness, especially if you're dealing with a world of a legacy data set.
00:42:47.360 --> 00:42:58.960
And I think that what it sounds like to me is that you have kind of a data governance council, maybe it's not necessarily called that, around what are the definitions?
00:42:59.280 --> 00:43:03.039
Where what the what role does the uh semantic layer play?
00:43:03.360 --> 00:43:11.760
This whole introduction of the doorman and where where that fits into the stack, that's probably like a cost savings uh you know initiative on its own.
00:43:12.079 --> 00:43:14.639
Where are those conversations being held?
00:43:14.800 --> 00:43:22.079
And is there a you know a group that you can recommend that other people can maybe adapt from what works well at DataSite?
00:43:22.480 --> 00:43:22.880
Sure.
00:43:23.280 --> 00:43:29.280
Um, so we do have a bit of data by committee, but it is a centralized committee.
00:43:29.360 --> 00:43:32.239
So we have a data governance department.
00:43:32.559 --> 00:43:40.239
Um, they are the ones who have counseled all of us in what data governance is, its importance.
00:43:40.480 --> 00:43:49.199
They actually have some of the best documentation on using their tools and finding the information in our in our data sets.
00:43:49.360 --> 00:43:52.159
We have in our metadata tool, for instance.
00:43:52.320 --> 00:43:55.920
Um, they are great at putting that information together.
00:43:56.079 --> 00:43:59.440
We do also currently have a data management office.
00:43:59.760 --> 00:44:08.320
They lead and have essentially regular meetings for data folks in the org to contribute to conversations.
00:44:09.039 --> 00:44:13.440
Um because I do work in product RD.
00:44:13.760 --> 00:44:25.280
A lot of the agenics for that we're working on came from our team, but there are other folks working in different areas of the org and doing other experiments.
00:44:25.519 --> 00:44:31.280
So we do have that centralized area to share that information.
00:44:31.679 --> 00:44:51.519
Um, in RD itself, we have a concept of communities of practice at DataSight, which has been absolutely invaluable for sharing the research and insights and experiments that folks throughout the product organization are doing.
00:44:51.760 --> 00:45:10.559
We have communities of practice on data, on AI, on Python, on back-end engineering, where we are doing demos, sharing thoughts, asking for feedback, and it is cross-team, oftentimes cross-org, where anybody is welcome to join.
00:45:10.800 --> 00:45:22.960
And it is for the purpose of sharing information, sharing philosophies, and sharing methods of practice and determining how we want to carry things out.
00:45:23.199 --> 00:45:32.079
So those are kind of the mechanisms that we have used for collaboration that have been pretty effective.
00:45:32.480 --> 00:45:33.840
I think that's extremely helpful.
00:45:34.000 --> 00:45:40.960
I mean, I think there is the the communities of practice model, which is like people that are actually building what's working, what's not working.
00:45:41.199 --> 00:45:51.360
And then when there's alignment uh and an agreed upon uh pond standard, that can then be, you know, bubbled up into like a more kind of formalized adoption within the organization.
00:45:51.760 --> 00:45:53.199
I don't care what it does.
00:45:53.599 --> 00:45:53.840
Yeah.
00:45:54.000 --> 00:45:57.199
Well, Heather, this has been a fascinating conversation for me.
00:45:57.360 --> 00:45:58.559
I've I've learned a lot.
00:45:58.719 --> 00:46:01.039
I know that others will take away a lot from it.
00:46:01.199 --> 00:46:09.360
If um there's one thing that you wanted to leave the audience or anything you wanted to share uh before we close, what would that be?
00:46:10.400 --> 00:46:17.519
I mean, it's definitely a period of experimentation and excitement.
00:46:17.760 --> 00:46:19.360
It can seem overwhelming.
00:46:19.519 --> 00:46:23.039
There's a lot happening all at the same time.
00:46:23.760 --> 00:46:29.760
Don't hesitate to experiment in a sandbox, not in prod, in a sandbox.
00:46:30.000 --> 00:46:43.920
Share the information, but don't forget about some of the core foundational data concepts, like your governance standards, like your cleanliness, like your data quality that are more important now than ever.
00:00:00.080 --> 00:00:02.799
They're going to ask the analyst, I need this data.
00:00:02.959 --> 00:00:06.639
I need the current metrics on this given KPI.
00:00:06.719 --> 00:00:12.560
And if there's something wrong with that data, the analyst would work around that or massage it, maybe let you know.
00:00:12.880 --> 00:00:16.480
But let's say you just put an MCP layer over top of your data.
00:00:16.719 --> 00:00:18.079
They get an answer back.
00:00:18.320 --> 00:00:21.839
And the agent, it sounds like it could be the right answer.
00:00:22.000 --> 00:00:29.039
That agent might not have any idea that there is a data quality issue with that table.
00:00:44.079 --> 00:00:46.560
I'm Prakash Chundran, CEO of Zano.
00:00:47.039 --> 00:00:58.640
Today I'm joined by Heather Lutz, Director of Engineering at DataSight, a SaaS platform for the full MA lifecycle that processes some of the most sensitive financial and legal documents in the world.
00:00:58.880 --> 00:01:02.399
Heather's background is unlike most engineering leaders you'll hear.
00:01:02.560 --> 00:01:14.000
She started as an English teacher in Japan, went through a coding boot camp, got hired as a junior engineer, and over the past decade built out the core data infrastructure that backs Data Sites platform today.
00:01:14.239 --> 00:01:26.480
She introduced Kafka, Airflow, and real-time data flows to the organization, started the analytics engineering and product analytics team, and now leads data infrastructure and engineering across the US and Costa Rica.
00:01:27.599 --> 00:01:30.959
There is a line Heather said that I think every builder needs to hear.
00:01:31.439 --> 00:01:34.719
AI is going to accelerate you in one of two directions.
00:01:34.959 --> 00:01:44.719
It's either going to give you intelligence at a speed you've never had, or it's going to give you wrong answers faster without you recognizing how wrong they actually are.
00:01:45.120 --> 00:01:50.159
The difference, of course, is what's underneath the quality, governance, and structure of your data.
00:01:50.640 --> 00:01:57.599
This is a conversation about the found uh about the foundation that most teams are skipping in the rush to plug in AI.
00:01:57.760 --> 00:01:59.760
Heather, so good to have you here today.
00:01:59.840 --> 00:02:00.480
Thank you for joining.
00:02:00.959 --> 00:02:02.079
Yeah, thank you for the invite.
00:02:02.159 --> 00:02:02.719
This is great.
00:02:03.040 --> 00:02:04.000
Yeah, awesome.
00:02:04.159 --> 00:02:07.040
So I wanted to start with something I mentioned up uh at the top.
00:02:07.200 --> 00:02:15.520
You know, you went from teaching English in Sapporo, Japan to building core data infrastructure for a company that handles some of the most sensitive data in the world.
00:02:15.759 --> 00:02:19.199
Can you just at a high level walk us through that journey?
00:02:19.599 --> 00:02:19.919
Yeah.
00:02:20.159 --> 00:02:24.319
So I did, I graduated with an English degree in about 2008.
00:02:25.120 --> 00:02:32.719
And if you were around then, you know that that was a really rough time to be graduating with an English degree, pretty much any degree.
00:02:32.960 --> 00:02:41.599
Um, so I took the opportunity at the time, I had taken four years of Japanese by that point, and ended up in Japan, uh, teaching English.
00:02:41.759 --> 00:02:46.960
I absolutely loved it and thought I wanted to go into teaching and be in education forever.
00:02:47.280 --> 00:02:53.439
I still enjoy education, um, but I did start to burn out from that position relatively quickly.
00:02:53.680 --> 00:02:58.000
I was in it for a while, moved into higher ed administration.
00:02:58.080 --> 00:03:00.639
I actually have my master's in higher ed.
00:03:01.120 --> 00:03:12.960
Um, and thankfully, during that time as an administrator, I had the privilege of advising software students, folks pursuing a degree in software engineering.
00:03:13.280 --> 00:03:15.120
It was a small cohort of students.
00:03:15.280 --> 00:03:16.400
I got to know them quite well.
00:03:16.479 --> 00:03:21.039
And I also realized that I thought what they were doing was far more interesting than what I was doing.
00:03:21.360 --> 00:03:36.639
So I decided to quit my job, go to a boot camp for a little less than a half a year, try my luck, and very fortunately got picked up by the organization that became Datasite as a junior engineer.
00:03:38.000 --> 00:03:48.639
And moved through a bit, spent some time developing the MongoDB infrastructure for Datasight back when we were still a greenfield.
00:03:48.960 --> 00:03:52.719
Thankfully, my boot camp program had trained me in the mean stack.
00:03:52.879 --> 00:03:58.639
I was one of the only engineers who had even touched MongoDB on the team at that time.
00:03:59.280 --> 00:04:00.960
So I got to work with the data.
00:04:01.039 --> 00:04:03.759
I got to set up that infrastructure from scratch.
00:04:04.000 --> 00:04:16.560
Um, everything from setting up the clusters, downloading the actual instances, doing the installs, the monitoring, indexing, some of the initial schemas, all of it as a junior.
00:04:16.720 --> 00:04:17.439
Um Yeah.
00:04:17.680 --> 00:04:21.439
I remember inspiration for this, you were saying it was like within your first week.
00:04:21.519 --> 00:04:23.519
They're like, who's worked with Mongo before?
00:04:23.680 --> 00:04:30.000
And because of your mean stack trading, you raise your hand and look, it is parlayed into quite a career for you.
00:04:30.560 --> 00:04:31.680
That was that was amazing.
00:04:31.759 --> 00:04:35.439
I I am still surprised the things they let a junior engineer do.
00:04:35.680 --> 00:04:37.759
Um, set up all their data.
00:04:38.000 --> 00:04:40.720
Yeah, that was uh that was an experience.
00:04:40.879 --> 00:04:42.319
But I I did that.
00:04:42.639 --> 00:05:01.439
I was in back end engineering for most of my first part of my career, about five-ish years or so, alternating between working with the data store on the operational side and then doing coding for everything from authorization, authentication, uh document metadata related work.
00:05:01.759 --> 00:05:08.319
And eventually uh they needed somebody to manage the analytical data space.
00:05:08.560 --> 00:05:13.600
That team had spun up about a year prior, and I accepted.
00:05:13.759 --> 00:05:21.439
They also tossed in the operational store with it, and I began managing the entire data stack for the product.
00:05:21.680 --> 00:05:34.480
And eventually I brought in, as you mentioned, an full-on analytics team, an analytics engineering team, started bringing in Kafka streaming for our data, and here I am.
00:05:35.040 --> 00:05:43.279
Here you are, uh a decade later, after just that first week raising your hand in Mongo, and you you're now a data expert.
00:05:43.439 --> 00:05:45.839
Um, this is a conversation about data.
00:05:46.079 --> 00:05:51.920
Arguably, data's always been important, uh, which I know something you're a very strong proponent of.
00:05:52.639 --> 00:05:58.000
But increasingly, especially in the world of AI, uh, it is more and more important.
00:05:58.160 --> 00:06:09.120
And for those that are avid listeners to this podcast, almost every uh organization leader, organizational leader I will talk with says you it always starts with the data.
00:06:09.600 --> 00:06:14.160
So, you know, going back to the quote I mentioned, you said it's going to accelerate in one of two ways.
00:06:14.319 --> 00:06:22.800
Either you're gonna get intelligence at a speed you never thought possible, or you're gonna get the wrong answers faster than you even recognize how wrong they are.
00:06:23.279 --> 00:06:28.720
So I'd like for you, in the context of this data conversation, to unpack what you mean by that.
00:06:29.279 --> 00:06:50.879
So previously at DataSight, if somebody needed information, and when I say somebody, I typically mean a business user, somebody in sales and marketing, somebody over on the product management side, typically what they're gonna do is they're going to ask the analyst, I need this data, I need the current metrics on this given KPI.
00:06:51.040 --> 00:06:52.399
Can you tell me this information?
00:06:52.560 --> 00:06:54.000
Can you put it in a dashboard?
00:06:54.160 --> 00:06:56.639
Can you pull it out of the data store?
00:06:56.879 --> 00:06:58.399
And the analyst will go do that.
00:06:58.480 --> 00:07:08.160
And if there's something wrong with that data, or there's something wrong with that data set, or if there's known quality issues, the analyst would work around that or massage it, maybe let you know.
00:07:08.399 --> 00:07:13.600
But at the end of the day, you would have a hopefully a fairly accurate report.
00:07:13.839 --> 00:07:24.959
Assuming there's not multiple sources of the same thing in the data store, and you know, the analyst is pulling the right information and knows that data quite well.
00:07:26.160 --> 00:07:39.920
But let's say you just put an MCP layer over top of your data, and you would just allow this business leader to query it using the AI tooling of their choice.
00:07:40.160 --> 00:07:43.439
You plug in Claude, you plug in chat.
00:07:44.399 --> 00:07:46.000
They get an answer back.
00:07:46.240 --> 00:07:49.759
And the agent, it sounds like it could be the right answer.
00:07:50.000 --> 00:07:54.160
The agent seems very confident they've analyzed the response.
00:07:54.639 --> 00:08:01.439
That agent might not have any idea that there is a data quality issue with that table.
00:08:01.680 --> 00:08:12.959
That agent, depending upon how that question was asked, might not realize that there are actually three different revenue tables somewhere within the data structure, and it doesn't know which one to look at.
00:08:13.279 --> 00:08:20.639
Or it could be that they're used for two slightly different questions, and it doesn't have the context around that.
00:08:20.879 --> 00:08:32.799
So your business user just got the wrong information, but it looks awfully correct because that agent is awfully confident.
00:08:33.120 --> 00:08:41.120
I have one of my engineering leaders say that the agents right now are like the extremely intelligent but overly confident intern.
00:08:41.679 --> 00:08:46.480
Very talented, very smart, but they don't know what they don't know.
00:08:48.000 --> 00:08:49.440
They don't know what they don't know.
00:08:49.600 --> 00:09:06.080
I think that is really important because I do think there is still um, I don't I don't know if stigma is the right word, but definitely a thought that you can plug an agent into a data set, no matter how large and it will figure itself out.
00:09:06.399 --> 00:09:12.720
Sure, it might take a little bit more time to think, but it's gonna figure it out for you and it's gonna come back to you with a confident answer.
00:09:13.120 --> 00:09:16.240
You actually ran an experiment, if I remember this correctly.
00:09:16.320 --> 00:09:21.120
You said you sent an agent to Snowflake with 22,000 tables and two petabytes of data.
00:09:21.440 --> 00:09:36.159
Um talk to me a little bit more about what that experience was like, what you learned from it, and um and then from those learnings, how you decided to segment out agent access to data moving forward.
00:09:37.039 --> 00:09:37.440
Yeah.
00:09:37.600 --> 00:09:41.120
So we actually did not give it access to the full 22,000 tables.
00:09:41.279 --> 00:09:44.240
We did give it an access to a subset of it.
00:09:44.399 --> 00:09:47.600
We gave it access to our usual consumable layer.
00:09:47.679 --> 00:09:50.240
We gave it access to our analyst layer, basically.
00:09:50.480 --> 00:09:53.360
So the layer that the analysts typically use.
00:09:53.759 --> 00:10:01.360
And we let a business user set up connection to the MCP and start asking it questions.
00:10:01.600 --> 00:10:05.440
And thankfully, we paired one of our analysts with this individual.
00:10:05.840 --> 00:10:08.559
And uh honestly, even the business user knew that was wrong.
00:10:08.639 --> 00:10:11.679
That that particular business user knew his data pretty well.
00:10:11.840 --> 00:10:14.879
They look at these KPIs, they look at these numbers all day.
00:10:15.120 --> 00:10:21.279
It it it wasn't even close, mostly because there's additional filters that just aren't being applied.
00:10:21.360 --> 00:10:31.440
There's terminology that the business user might be using that, you know, that that column name, that table name's not quite the same thing.
00:10:31.840 --> 00:10:37.759
So it was very evident that the agent was making some guesses.
00:10:38.159 --> 00:10:39.759
Um it did give an answer.
00:10:40.799 --> 00:10:41.519
You got one.
00:10:41.840 --> 00:10:47.440
It was it was not the correct answer, and it would certainly not be something we would have put on a on a report.
00:10:48.000 --> 00:10:48.399
Yeah.
00:10:49.200 --> 00:11:02.159
Um you know, I think this is even though you might have a much larger data set than uh what people are traditionally used to, um, I think there's just a thought of like really kind of figuring out where to start.
00:11:02.320 --> 00:11:12.000
If people are generally aligned with the fact that like you can't just throw a large data set at an agent, well, what can you throw uh at it?
00:11:12.240 --> 00:11:28.399
And um later on, I'm sure we'll talk about like the beginnings of it, but like how do you at Data Site and as a data leader think about how to partition either different agents, different MCPs, different roles, and what they get access to?
00:11:29.200 --> 00:11:29.440
Sure.
00:11:29.759 --> 00:11:55.279
Well, first of all, if you want to put the entire landscape of, let's just say your consumable data in front of the agent, if you have somebody using a skill or somebody writing a skill who really knows what they're doing, can get detailed into it, can specifically point to the exact tables to look at, the exact information to filter on, you actually will get a half decent answer.
00:11:55.360 --> 00:11:58.960
This is assuming that the person who wrote that skill knows what they're doing.
00:11:59.200 --> 00:12:03.919
And you're probably going to repeat some of the same instructions over and over and over again.
00:12:04.080 --> 00:12:09.120
And eventually that's going to start to become a little bit untenable.
00:12:09.279 --> 00:12:12.879
Skills are typically meant to answer a how question.
00:12:13.120 --> 00:12:21.919
Um, and you're essentially forcing it to answer also, or instructing it to answer a lot of what type questions as well, in that case.
00:12:22.159 --> 00:12:32.639
So from there, how do you start going and chunking this out so that the agent doesn't have 20 something thousand tables to go and look at?
00:12:33.360 --> 00:12:37.600
Well, first of all, taking a look by domain and by role.
00:12:37.919 --> 00:12:44.960
Most people who are pooling information in an organization, they are pulling this information by some sort of domain.
00:12:45.120 --> 00:12:55.360
If you are working in product, you are probably pulling things around feature usage, metrics around the usage for the components in the application.
00:12:55.600 --> 00:13:00.240
Hopefully, you probably have some kind of OKR as you want that information out of it.
00:13:00.320 --> 00:13:03.039
You want to know how many people are using your product.
00:13:03.200 --> 00:13:07.519
Or if you're in marketing, you maybe you're doing some cross-selling information.
00:13:07.759 --> 00:13:14.000
But you are typically working out of a general bucket of information.
00:13:14.559 --> 00:13:18.240
So segment your information from a data perspective.
00:13:18.559 --> 00:13:24.960
You should have ownership by domain in a perfect world with probably some cross-dependencies underneath.
00:13:25.200 --> 00:13:28.639
But it's the same thing with the MCP and with the agent.
00:13:29.679 --> 00:13:33.840
Give it a surface of data where it's going to be the expert.
00:13:34.240 --> 00:13:40.879
I literally at one point thought to call our product agent and name it after like our analyst.
00:13:40.960 --> 00:13:44.240
I'm like, this is essentially an analyst's companion for product.
00:13:44.320 --> 00:13:45.840
That's that's what this is doing.
00:13:46.080 --> 00:13:50.000
So it's over top of this particular domain.
00:13:50.240 --> 00:13:53.679
Hopefully, that domain is also fairly well curated.
00:13:54.799 --> 00:14:03.519
If you have your gold consumable layer of data and you say this is my data product, and I don't necessarily mean it's client-facing in that sense.
00:14:03.679 --> 00:14:06.480
I mean this is the data that I care about.
00:14:06.639 --> 00:14:09.200
It's the one that I'm saying it's ready to use.
00:14:09.440 --> 00:14:12.320
It's of a quality that I am satisfied with.
00:14:12.559 --> 00:14:15.919
It is well documented, it is governed.
00:14:16.159 --> 00:14:23.919
Ideally, you have some kind of semantic layer or semantic views in the case of Snowflake on top of this.
00:14:25.200 --> 00:14:33.679
That that's a good surface to then try and put an agent over top of that or put an MCP on top of that.
00:14:33.919 --> 00:14:37.759
That also is attached typically to your RVAC solution.
00:14:38.000 --> 00:14:43.039
So your users in that area should also only have access to that date.
00:14:43.440 --> 00:14:45.919
Yeah, there's a couple threads I want to pull on there.
00:14:46.000 --> 00:15:07.440
And one of the first things that I want to recognize is that um before you define the skill file or anything like that for the special uh speciality of what that agent uh pulls and how it pulls it and what it pulls, talking about all of this pre-work that needs to be done before we even get there.
00:15:07.600 --> 00:15:10.159
You're even using words like governance, right?
00:15:10.559 --> 00:15:12.480
In an ideal world, yes.
00:15:12.879 --> 00:15:13.919
In an ideal world.
00:15:14.159 --> 00:15:17.840
But broadly speaking, you need to be able to have the data.
00:15:18.000 --> 00:15:25.759
Um, I think you said this, and probably you said you have to be able to cure curate it, maintain it, and make it consumable or accessible.
00:15:26.320 --> 00:15:27.679
It's like an API.
00:15:27.759 --> 00:15:32.159
If you're writing a piece of software, you are putting something into production.
00:15:32.639 --> 00:15:34.799
In this case, it's data.
00:15:35.039 --> 00:15:45.519
If you it's not ready for somebody to read and use, ideally it shouldn't be in that, like curate it like it's an think of it as it's an API.
00:15:45.679 --> 00:15:46.960
You are delivering a product.
00:15:47.440 --> 00:15:49.120
So I want to talk about two separate things.
00:15:49.279 --> 00:15:52.639
I want to talk about data readiness and data governance.
00:15:53.039 --> 00:15:53.360
Correct.
00:15:53.600 --> 00:15:55.200
Let's start with data readiness.
00:15:55.360 --> 00:16:00.320
When you say if the data is not ready, what does that mean to you?
00:16:00.799 --> 00:16:10.559
So to me, we're saying if the data is not ready, this means that you have some kind of CICD and testing process sitting here involved.
00:16:10.879 --> 00:16:16.159
So for instance, if you have a column that is never supposed to be null, you've checked that.
00:16:16.320 --> 00:16:20.879
You've checked that before it's even landed anywhere in production.
00:16:21.440 --> 00:16:29.519
Um, if you have a certain set of options, enums for a given set, you know what those are.
00:16:29.679 --> 00:16:32.559
You know it's not supposed to be outside of those parameters.
00:16:32.799 --> 00:16:40.480
You know that if you're joining two pieces of information, you're expecting it to be within a certain range of values.
00:16:40.720 --> 00:16:52.159
You've already checked that before your CICD process has written that into the productize version of your tables.
00:16:52.480 --> 00:16:55.840
So all of that is in there, and that's just data in motion.
00:16:55.919 --> 00:17:00.879
That's data as it's landing into that consumable environment.
00:17:01.039 --> 00:17:06.160
Ideally, you also have some data quality over top of that longer term.
00:17:06.319 --> 00:17:13.920
So you're not just monitoring, hey, I don't expect this tape column to be null, except maybe in these circumstances.
00:17:14.880 --> 00:17:18.160
Okay, great, it landed in pro, but what about over time?
00:17:18.799 --> 00:17:29.519
Like let's say all of a sudden I'm seeing more null values than I would expect for this type of value.
00:17:30.079 --> 00:17:40.319
Or hey, I'm all of a sudden seeing a great increase in this type of for the data site process, this type of project.
00:17:40.400 --> 00:17:42.000
But what does that mean?
00:17:42.160 --> 00:17:44.400
Um, so it's changing an expectation.
00:17:44.480 --> 00:17:46.640
It's going outside of what you would expect.
00:17:46.720 --> 00:17:49.039
It's outside of a standard deviation.
00:17:49.200 --> 00:17:54.480
So you're measuring it both in terms of can I put this data into prod?
00:17:55.200 --> 00:17:58.480
Can I promote this into a working layer?
00:17:58.799 --> 00:18:15.519
Plus, and this is something that you don't necessarily, I think, measure as much in typical software engineering, but over time, I'm looking and examining how my data set is changing, possibly in ways that I don't necessarily expect.
00:18:15.839 --> 00:18:21.680
And I think the interesting part there, I think when we were talking earlier, you know, you have a back-end engineering background.
00:18:22.000 --> 00:18:25.759
With back-end engineering, you have a CI CD and you have a testing process.
00:18:26.000 --> 00:18:26.160
Yes.
00:18:26.400 --> 00:18:32.319
Um, with data, that is not necessarily a standard, like the continuous testing of data quality.
00:18:32.960 --> 00:18:42.960
Um I'd love for you to talk a little bit more about that because I that was something that I remember that was surprising to me, that that was like something that you felt very strongly about.
00:18:43.200 --> 00:18:55.119
You touched on it a little bit, but I'd love for you to say more about why more people need to make it a standard, um, especially in a world where you have agents and AI accessing your data.
00:18:55.440 --> 00:18:56.000
Correct.
00:18:56.240 --> 00:19:03.039
I mean, essentially, once you have AI accessing your data, again, whatever they're accessing, that that's prod.
00:19:03.440 --> 00:19:10.240
So if you wouldn't want this committed into your application, don't put it into your data set either.
00:19:10.480 --> 00:19:14.240
One of the questions I almost always ask in an interview, and it's very general.
00:19:14.400 --> 00:19:16.559
So if you ever interview with me, take note.
00:19:17.039 --> 00:19:19.279
How do you test data?
00:19:19.680 --> 00:19:24.720
And you should have hopefully an opinionated answer or make a good one up.
00:19:25.200 --> 00:19:40.799
But you have tests that appear at various places, and you will now see this even in some of the tooling that you have, whether you're using Snowflake or Databricks, they're starting to build in data quality metrics and monitoring into their tool sets.
00:19:41.039 --> 00:19:45.039
DBT, if you're using it, has testing inside of it.
00:19:45.359 --> 00:19:55.359
Airflow, you are able to detect when a given um DAG fails, why it fails, the logs for it.
00:19:55.599 --> 00:20:01.039
Um, so you should have testing and the orchestration of the data.
00:20:01.200 --> 00:20:15.119
So as the data is, as the trigger goes off, you should be able to test as the data is then transformed and landed, transformed, joined, and landed, and you're confirming it at every point.
00:20:15.279 --> 00:20:25.119
And then you're going to essentially monitor again at the landing zone over a period of time that that data again is meeting your expectations.
00:20:25.440 --> 00:20:29.519
Like you have CI C D, the orchestration, that's your CI CD process.
00:20:29.680 --> 00:20:32.960
Did it, did it, did it also fail or something like that?
00:20:33.119 --> 00:20:41.359
Um, you can almost think of your, I guess your landed monitoring kind of like an end-to-end test or a full stack test.
00:20:41.519 --> 00:20:46.559
So you're seeing it, I view it like it is like a software.
00:20:47.039 --> 00:20:50.720
If it's important enough, I should be testing it and making certain it's okay.
00:20:51.039 --> 00:20:55.119
Well, what's really interesting, at least not that I'm aware of it, and of course I am not a data expert.
00:20:55.279 --> 00:20:57.119
I just don't think that there's a standard around this.
00:20:57.279 --> 00:21:02.400
Like you just made some like correlations around like unit tests and DBT that totally make sense to me.
00:21:02.559 --> 00:21:18.559
But like in the world of like data CICD or data pipeline, um, like the quality standard around how you continuously test that, I don't think is something that people are thinking about, but they will start thinking about it more moving forward, I think.
00:21:18.880 --> 00:21:20.640
I do hope that comes out more.
00:21:20.799 --> 00:21:26.240
I have noticed, like, you will find walkthroughs for how to create like DBT tests.
00:21:26.480 --> 00:21:32.000
You will certainly, you certainly have tools like Datadog and Monte Carlo that will monitor data for you.
00:21:32.160 --> 00:21:41.119
But you're right, I I have not seen too many references to like an end-to-end testing process for data, like actually written down.
00:21:41.279 --> 00:21:50.319
Not certainly not as many as I see for software engineering, where you have whole things like test driven development that have built out there.
00:21:50.480 --> 00:21:52.880
I I don't think it's as big as that.
00:21:53.200 --> 00:21:53.759
Yeah.
00:21:54.000 --> 00:21:54.319
Okay.
00:21:54.559 --> 00:21:59.759
So we we uh I'm sure we could talk about each one of these topics for uh much longer, but we talked.
00:22:00.160 --> 00:22:02.160
Talked a little bit about data readiness.
00:22:02.400 --> 00:22:04.240
Let's talk about data governance.
00:22:04.480 --> 00:22:04.720
Correct.
00:22:05.119 --> 00:22:06.400
What does that mean to you?
00:22:07.200 --> 00:22:20.640
So, data governance to me is the method of making your data accessible and usable and understandable and consistent to everybody.
00:22:21.039 --> 00:22:27.039
So now you're stepping outside of your own domain, and this all needs to make sense holistically.
00:22:27.359 --> 00:22:38.640
So this means that, first of all, I hope if you have data for product, you have one team that is responsible for data for product.
00:22:38.720 --> 00:22:41.839
So this goes back to the idea of domain ownership.
00:22:42.079 --> 00:22:43.759
Um, who's in charge?
00:22:44.079 --> 00:22:54.000
This also goes to make certain around, like, for instance, hey, we're talking about let's say total number of pages.
00:22:54.319 --> 00:22:55.759
A simple metric.
00:22:56.240 --> 00:22:59.599
Okay, what does total number of pages actually mean?
00:22:59.759 --> 00:23:03.200
Is that the total number of pages on a project?
00:23:03.440 --> 00:23:06.799
Is that the total number of pages that were processed?
00:23:07.039 --> 00:23:12.960
Is that the total number of pages that are available to a certain segment of users?
00:23:13.119 --> 00:23:14.559
What does that actually mean?
00:23:14.720 --> 00:23:18.559
And everybody then has to agree on that particular definition.
00:23:18.799 --> 00:23:27.599
This is one of the phenomena around how people in the organization can get a report on supposedly the same thing and get two totally different answers.
00:23:27.759 --> 00:23:35.200
That's because multiple people interpreted that information in two completely different ways.
00:23:35.440 --> 00:23:44.160
Or the person requesting the data thought they were asking for one thing, and the person filling out that request thought it was something completely different.
00:23:44.400 --> 00:23:49.119
So it's that agreement, even around how we use these terms.
00:23:50.160 --> 00:23:56.240
And then there, I think there is that overlap in data quality as well, but it's far more holistic.
00:23:56.480 --> 00:24:01.599
It's also making certain because data is actually most important when it's used together.
00:24:01.680 --> 00:24:06.000
This is actually, I think, one of the biggest differences between data and software engineering.
00:24:06.240 --> 00:24:13.519
Software engineering, you want, you want to try and keep that tight ownership, but like loosely cohesive.
00:24:13.759 --> 00:24:17.200
Data, data is most useful when it's with other data.
00:24:17.440 --> 00:24:27.200
So you also really want to make certain that you understand where that data is coming from and tracking that lineage and in some governance structures, even down to like a field level.
00:24:27.440 --> 00:24:30.559
So really understanding the origin of that data.
00:24:30.720 --> 00:24:42.640
But it's it at the heart of it, it's putting the tools in people's hands to make certain that they have enough information about the data that they are using to make very good logical business decisions.
00:24:43.119 --> 00:24:53.680
When we talk about data governance, it's, you know, I think holistically about what you're saying about governance in terms of its accessibility to everyone.
00:24:54.640 --> 00:25:04.799
But also in terms of access, you've got the platform layer, you've got the data layer, um, you know, you've got um uh the the application layer.
00:25:05.279 --> 00:25:07.359
Where does governance actually start?
00:25:07.519 --> 00:25:15.119
And especially in a world maybe we're getting into harness territory here, where do the instructions for these agents uh live, right?
00:25:15.200 --> 00:25:16.559
Like how do you think about that?
00:25:16.880 --> 00:25:19.200
Okay, well, where does governance actually start?
00:25:19.279 --> 00:25:23.039
I am, first of all, I am I am not in charge of governance.
00:25:23.359 --> 00:25:32.400
I leave leave most of that to our governance team at data site, but I am fully inspired by them and I take what they say to heart.
00:25:32.720 --> 00:25:40.000
But an ideal world, they would say governance starts as soon as that data lands in an ideal perfect world.
00:25:40.240 --> 00:25:46.319
Um, most people don't start governance as uh soon as their data is up and running.
00:25:46.400 --> 00:25:50.400
You ideally you would have started governance when your product and everything was a green field.
00:25:50.559 --> 00:25:50.720
Yeah.
00:25:50.960 --> 00:25:52.480
If you did, good job.
00:25:52.720 --> 00:25:54.640
Um that's perfect.
00:25:54.880 --> 00:25:58.240
That is not, I don't think that's most of us.
00:25:58.480 --> 00:26:05.920
Um so for us, the way we have chosen to start is from our client, even if that's internal facing layer.
00:26:06.079 --> 00:26:08.000
What's your gold standard of data?
00:26:08.240 --> 00:26:19.599
What are the top 10 metrics and reports that you are responsible for that are the most important thing to your area of the organization?
00:26:19.759 --> 00:26:20.799
Start with that.
00:26:21.440 --> 00:26:23.839
Um, start with those particular metrics.
00:26:23.920 --> 00:26:35.759
And typically you will find that there's a series of metrics, tables, structures that underpins a large portion of what you do.
00:26:36.559 --> 00:26:41.519
Document that, solidify that, and start from there.
00:26:41.839 --> 00:26:50.000
Because once you find, I think, those key structures and you start from that point, many other things become easier.
00:26:50.079 --> 00:26:59.039
And you might actually get to the point where you don't necessarily need to document, analyze, and pick apart absolutely everything.
00:26:59.279 --> 00:27:06.000
Um, like for instance, our bronze layer of data, our landed layer, we sink it.
00:27:06.079 --> 00:27:07.920
No one's gonna see that other than us.
00:27:08.319 --> 00:27:10.960
And hopefully we know what we're doing.
00:27:11.119 --> 00:27:12.160
We hope we know.
00:27:12.400 --> 00:27:15.519
Um that layer we don't really worry about.
00:27:15.680 --> 00:27:30.079
We worry predominantly about that consumable layer of data, that gold and platinum layer where we know that we have folks from other areas of the org, other analysts doing that work.
00:27:30.319 --> 00:27:35.119
Or in this case, this is where our agents are actually plugged in.
00:27:35.359 --> 00:27:44.079
So we have curated a very bounded layer where we have said everything in here is documented.
00:27:44.319 --> 00:27:50.480
There is a semantic layer in the case of Snowflake semantic views built over top of this.
00:27:50.799 --> 00:27:58.640
We actually built those views when we started designing the product itself that became this layer.
00:27:58.799 --> 00:28:06.640
So it was actually built out first before we even started coding that newer version of our layer.
00:28:06.880 --> 00:28:11.759
Um, anything new immediately has definitions coming in.
00:28:12.160 --> 00:28:21.599
So, this way, the most important things in your org plus anything going forward ends up getting governed, ends up in that consumable layer.
00:28:21.680 --> 00:28:28.480
And it's gonna be a lot easier for an agent then to roll through that and understand it.
00:28:29.359 --> 00:28:30.000
Yes.
00:28:30.240 --> 00:28:32.880
I mean, that makes a lot of sense.
00:28:33.119 --> 00:28:36.880
And I think that semantic layer is hugely critical.
00:28:36.960 --> 00:28:45.279
And I think your approach around that, you know, starting from the client, the gold standard data, uh, top 10 metrics and reports.
00:28:46.000 --> 00:28:55.039
I am still interested, though, in the cross-functional piece because giving agents access to the instructions on how to query the data.
00:28:55.680 --> 00:29:00.000
It's not like a single player game that the semantic layer can just solve on its own.
00:29:00.799 --> 00:29:04.720
How how are you handling that today at Data Site?
00:29:05.920 --> 00:29:10.160
So over top of this, the semantic layer is it helps.
00:29:10.400 --> 00:29:13.359
It helps a lot, but it's not everything.
00:29:13.680 --> 00:29:21.359
So this is where we're starting to dig into other pieces of information coupled with it.
00:29:21.519 --> 00:29:25.359
So if you're using Snowflake, you have something, for instance, called a verify query.
00:29:25.680 --> 00:29:32.079
So as we have users start to use this layer, I can't see the prompts.
00:29:32.400 --> 00:29:37.680
We can see everything that at least Snowflake is querying on.
00:29:37.839 --> 00:29:42.799
So we can actually go and say, oh, this is queried very frequently.
00:29:42.880 --> 00:29:45.200
Let's create a verified query.
00:29:45.359 --> 00:29:52.000
So essentially training it and saying, okay, when you see this request, this is what you should be running.
00:29:52.160 --> 00:29:52.480
Got it.
00:29:52.640 --> 00:29:57.920
And that will also help at them with very similar queries, similar filters.
00:29:58.079 --> 00:30:01.759
So that is a verified query where it's saying, I know this is right.
00:30:02.000 --> 00:30:04.640
This is the answer that you should then expect.
00:30:05.759 --> 00:30:06.799
Tune from there.
00:30:06.960 --> 00:30:14.000
We can also, we also monitor these agents as well for their accuracy against these verified queries.
00:30:14.160 --> 00:30:19.759
So as that's coming in, as we're deploying a new version of the agent, we actually test on that too.
00:30:19.920 --> 00:30:22.160
Like how closely did you hit these?
00:30:22.319 --> 00:30:25.039
Um, and if it fails, we won't even deploy it.
00:30:25.359 --> 00:30:32.880
Um however, even going back further than that, so you've got the verified queries, you've got your semantic layer.
00:30:33.200 --> 00:30:37.680
We have also started to dig into skills as well.
00:30:38.160 --> 00:30:45.599
So we do have skills out for our um individual sets of users.
00:30:45.759 --> 00:30:57.119
So certain filtering requirements, certain things that almost like what the analyst is trained to ask to they include this in the request?
00:30:57.279 --> 00:31:00.559
If they didn't, please prompt them for it because it'll make it better.
00:31:00.799 --> 00:31:03.519
Um, so that would be like certain filters.
00:31:03.680 --> 00:31:06.880
We also train um the agent itself on synonyms, too.
00:31:07.039 --> 00:31:11.279
Like certain synonyms, like we might call it a project, we might call it a data room.
00:31:11.359 --> 00:31:14.000
Those are synonyms they could use either one.
00:31:14.400 --> 00:31:16.720
So that's layered on top of it.
00:31:16.960 --> 00:31:23.200
There's also potentially another layer we're considering more on like the langsmith side.
00:31:23.599 --> 00:31:33.279
Um, there was a thought, once again, coming from our lead developer, of what we call the data doorman, which is maybe you actually don't need Snowflake.
00:31:33.759 --> 00:31:36.559
Maybe you don't actually need that data set.
00:31:36.720 --> 00:31:40.240
Maybe what you're actually looking for is click data.
00:31:40.480 --> 00:31:43.119
Maybe what you need is actually pendo data.
00:31:43.200 --> 00:31:51.359
Or, you know what, if you're acting about an active project, maybe I could actually send you to our operational data store and that would be more efficient.
00:31:51.839 --> 00:32:04.559
Dorman will help you with that question so that I can direct you to the correct place and you don't even have to ask us anymore which data sets and which tools you should be looking at.
00:32:04.720 --> 00:32:06.640
We'll just direct you there.
00:32:06.799 --> 00:32:10.319
Um, so that's another thing we are experimenting with.
00:32:10.480 --> 00:32:13.680
So there are multiple layers in this.
00:32:14.480 --> 00:32:22.799
And we tried to keep the snowflake agent portion very specific to the snowflake data itself.
00:32:23.039 --> 00:32:23.599
Got it.
00:32:23.839 --> 00:32:27.440
It's about the snowflake data, about the semantic.
00:32:27.599 --> 00:32:28.559
It's there.
00:32:28.880 --> 00:32:38.960
The layer before that, that skill layer, has tended to be more about the business user, how they're asking, how they're interacting.
00:32:39.200 --> 00:33:01.599
And then there's yet another layer around, okay, well, maybe we need to even redirect you to a completely different location, or perhaps between a different agent in Snowflake, because maybe you're an executive and you're using this agent normally, but you need to ask about something in product, we'll just redirect you to the product agent.
00:33:01.920 --> 00:33:04.319
And you're an executive, so you can know that information.
00:33:04.640 --> 00:33:04.880
Yeah.
00:33:05.039 --> 00:33:05.359
Yeah.
00:33:05.519 --> 00:33:09.680
I I love um I love the way that you are approaching it.
00:33:09.759 --> 00:33:17.039
And I love the concept of a data doorman that kind of ushers you into the right kind of uh set to be looking at.
00:33:17.279 --> 00:33:19.200
Um who owns this?
00:33:19.359 --> 00:33:27.200
Like because it because it is so cross-functional, um who is currently owning this at DataSight if you were allowed to share?
00:33:27.599 --> 00:33:30.799
Ah that starts to get a little complicated.
00:33:30.880 --> 00:33:34.079
So, but let's go into ideally who owns it.
00:33:34.240 --> 00:33:34.480
Yeah.
00:33:34.640 --> 00:33:40.240
So if you own a domain, you've curated the domain, guess what?
00:33:40.319 --> 00:33:41.359
You own the agent.
00:33:42.000 --> 00:33:46.240
It's it's it is for your analysts, for your services.
00:33:46.400 --> 00:33:47.920
You are the best person to train it.
00:33:48.000 --> 00:33:48.960
Ideally, you own it.
00:33:49.039 --> 00:33:57.759
Or if you don't own it, you're at least heavily directing and having influence on the engineering team that is doing it, would be my opinion on it.
00:33:58.079 --> 00:34:01.839
Um, however, we are not alone in this.
00:34:02.160 --> 00:34:05.279
Um, like for instance, we're working on this doorman concept.
00:34:05.359 --> 00:34:06.480
We don't own Lang Smith.
00:34:06.640 --> 00:34:08.800
That actually belongs to our AI platform team.
00:34:08.960 --> 00:34:17.039
They they have the AI platform and we work with them in concert to get features and functionality like this out the door.
00:34:17.519 --> 00:34:21.599
So at a really high level, that's that's the ownership.
00:34:21.679 --> 00:34:27.840
But we are still in such an experimental phase that we have not fully fleshed it out.
00:34:27.920 --> 00:34:29.360
We are still testing this.
00:34:29.519 --> 00:34:36.320
And currently we have released it to just internal product clients who are trying it out.
00:34:36.880 --> 00:34:41.360
Well, one thing is for sure, everyone is still experimenting and evolving.
00:34:41.440 --> 00:34:49.119
But at least you have some opinionated or prescriptive ways of what you've seen work and certainly what you have seen not work.
00:34:49.360 --> 00:35:10.559
Um, and in that spirit, I think you made you know a comment earlier that, you know, obviously it feels easy to build right when you're starting from scratch and it's green from but a lot of um application development leaders might be listening to this and they might be part of organizations that have been collecting data for years, decades, right?
00:35:11.039 --> 00:35:20.480
So I I'm curious if you have a perspective around what do they start implementing now to start gaining clarity?
00:35:20.639 --> 00:35:23.599
Like where would you recommend that they begin?
00:35:24.719 --> 00:35:29.760
This may seem like a really obvious spot, but do you know what you're asking for?
00:35:30.719 --> 00:35:32.719
Would be the first thing I would say.
00:35:32.960 --> 00:35:46.159
Do you have a good handle on what metrics you need, what those metrics actually represent, and what the metrics, the definition of those metrics are?
00:35:46.400 --> 00:35:49.840
Do you know what you should be tracking regularly?
00:35:50.079 --> 00:35:57.679
Do you know what you should be tracking as potentially new functionality comes out, or as functionality shifts?
00:35:57.920 --> 00:36:05.199
Um, so having a good handle on that is really helpful for just everybody involved.
00:36:05.360 --> 00:36:19.920
So as we're building out the data sets, as we're building out these layers, getting a good grasp on what actually we're supposed to be measuring, because you also have limited space and time for semantic layers, et cetera.
00:36:20.000 --> 00:36:21.599
Like so they can only be so large.
00:36:21.760 --> 00:36:26.400
They can only hold so much before you have to divide them up again, or they won't perform very well.
00:36:26.719 --> 00:36:34.559
Um so making certain you have a handle on that would actually be one of the first things I would say to do.
00:36:35.119 --> 00:36:48.400
Um and then outside of that, once you have a good handle on the metrics themselves, I would say having an understanding of a flow of how you want to interact with those metrics.
00:36:48.559 --> 00:36:54.880
So, like I'm starting on the front end, the front, the front end, um, like this would be an application.
00:36:55.599 --> 00:37:00.639
Do you see yourself interacting with an agent and asking these questions via MCP?
00:37:00.719 --> 00:37:02.880
Or actually, would you prefer a dashboard?
00:37:03.119 --> 00:37:10.559
Um, one of the things our doorman actually does is if metrics exist in a dashboard, we'll just send you to the dashboard.
00:37:10.639 --> 00:37:12.079
Why waste the tokens?
00:37:12.559 --> 00:37:14.639
Why, why consume more than we have to?
00:37:14.880 --> 00:37:23.360
We will send you to where those metrics are actually persisted, and you can just go and look at it rather than spending the extra money.
00:37:23.519 --> 00:37:30.719
Um, but even then, having an idea of that flow also figures okay, well, what kind of a skill am I building out?
00:37:30.880 --> 00:37:35.519
What kind of functionality am I building out for you to support you in doing this?
00:37:35.840 --> 00:37:38.400
Um that does help.
00:37:38.719 --> 00:37:53.679
And then if you haven't created a curated layer of data, if you threw it all in the data layer and hoped for the best, um, creating that layer once again, starting with, okay, well, here's my most important structures to have.
00:37:53.840 --> 00:37:56.159
Here's my most important things to surface.
00:37:56.320 --> 00:38:28.400
And even if you're manually putting it out there, if you don't have any kind of CICD process or you're triggering a some kind of cron job for DBT or whatever transformation you're doing, um just making certain that you are getting your monitoring and hands and your quality around that key layer, building it up, setting up the basic monitoring, even the basic testing around landing it in prod, as we discussed, is is helpful.
00:38:28.559 --> 00:38:31.840
And you'll slowly start building up the rest of it from there.
00:38:32.320 --> 00:38:32.960
Yeah.
00:38:33.360 --> 00:38:40.800
I think one of the key things there is like, you know, you can work with like a bounded set, right?
00:38:40.880 --> 00:38:42.719
It doesn't have to be everything all at once.
00:38:43.119 --> 00:38:44.159
Don't work with the ocean.
00:38:44.239 --> 00:38:45.920
It's I actually recommend not.
00:38:46.400 --> 00:38:48.159
Yes, don't don't work with the ocean.
00:38:48.239 --> 00:38:50.159
That's good right, that's a good recommendation.
00:38:50.400 --> 00:38:58.159
Um, and then you know, uh in that bounded set, I think you can do a lot of the recommendations that we've discussed here today.
00:38:58.480 --> 00:39:05.760
Make sure that that set is is governed, like that readiness that we were talking about, the governance that we were talking about.
00:39:05.920 --> 00:39:12.719
And then also on top of that, you you kind of started with this, but like what are you asking for, which bleeds into the definitions?
00:39:12.800 --> 00:39:16.880
Because if you're confused, I think the agent's gonna be confused, right?
00:39:17.280 --> 00:39:17.840
Yes.
00:39:18.079 --> 00:39:23.280
If you're confused, the agent is not really gonna necessarily make it any clearer.
00:39:23.440 --> 00:39:26.559
In fact, it it might lead you to further wrong answers.
00:39:26.719 --> 00:39:39.360
So the more pinpointed you can be in your requests, in your definitions, in what you are asking for, the more clarity we are going to be able to give back to you.
00:39:39.519 --> 00:39:52.719
And honestly, that even applies, it's more important for regenics, I think, and for these uh new processes as you have the business users themselves starting to interact with the data more.
00:39:52.960 --> 00:40:04.320
But even prior to that, it's still helpful for even an analytics team or folks who are just creating dashboards, having a really clear idea on what you're asking for, realizing that it's important.
00:40:04.400 --> 00:40:07.760
It's just not something you're gonna ask for once and toss away.
00:40:07.920 --> 00:40:19.280
It it is something important for you to track over time and having a good grasp on what those metrics are versus the things that I'm just gonna check once and then turn around and that's it.
00:40:19.599 --> 00:40:20.239
Yeah.
00:40:20.719 --> 00:40:40.400
Um, you know, as we start to wrap up, I'd love to hear a little bit more on what you are excited about in the landscape of data, in the landscape of analytics and, you know, data hygiene, especially when it comes to I actually think data, this is gonna sound weird.
00:40:40.559 --> 00:40:41.679
It's like having a renaissance.
00:40:41.760 --> 00:40:49.679
Like I think people are realizing the importance of this data hygiene, you know, at the foundation of what's going to power AI.
00:40:49.920 --> 00:40:55.519
I don't think I've ever had so many conversations about like foundationally your data has to be, right?
00:40:55.599 --> 00:40:56.400
What is that saying?
00:40:56.559 --> 00:40:58.719
Crap in, crap out type of thing, right?
00:40:59.039 --> 00:41:00.559
Garbage in, garbage out.
00:41:00.719 --> 00:41:01.360
Still applies.
00:41:01.679 --> 00:41:02.239
There it is.
00:41:02.400 --> 00:41:05.119
Because you live in this world, what are what are you looking forward to?
00:41:05.280 --> 00:41:06.159
What are you thinking about?
00:41:06.239 --> 00:41:07.760
What are you excited about?
00:41:08.320 --> 00:41:21.360
I think the thing that excites me most is just people getting great greater clarity on their data itself and really understanding their own needs and what they're asking for.
00:41:21.599 --> 00:41:34.079
Um it's it's really not easy, but it's really common to have organizations where data is very siloed, it's kind of all over the place.
00:41:34.239 --> 00:41:38.639
You have to really shift and sift through it to find its value.
00:41:38.800 --> 00:42:02.800
I'm really looking forward to people putting effort into their data, into their governance, um, and actually having data provide the clarity and insight that it is fully capable of without having to worry about whether or not that those are the right, right metrics, right information.
00:42:03.199 --> 00:42:08.960
Um, it's just I'm really looking and excited about that level of confidence and clarity.
00:42:09.440 --> 00:42:23.280
Um, it is oftentimes historically very difficult to get people excited about doing data definitions and doing um data cleanliness.
00:42:23.679 --> 00:42:34.480
I still don't think people are going to be absolutely excited about it, but I think the need for it will at least um convince people to do it.
00:42:35.039 --> 00:42:35.679
In essential.
00:42:36.559 --> 00:42:46.960
We've covered so much today, a lot of tactical advice around like how to get started with data cleanliness, especially if you're dealing with a world of a legacy data set.
00:42:47.360 --> 00:42:58.960
And I think that what it sounds like to me is that you have kind of a data governance council, maybe it's not necessarily called that, around what are the definitions?
00:42:59.280 --> 00:43:03.039
Where what the what role does the uh semantic layer play?
00:43:03.360 --> 00:43:11.760
This whole introduction of the doorman and where where that fits into the stack, that's probably like a cost savings uh you know initiative on its own.
00:43:12.079 --> 00:43:14.639
Where are those conversations being held?
00:43:14.800 --> 00:43:22.079
And is there a you know a group that you can recommend that other people can maybe adapt from what works well at DataSite?
00:43:22.480 --> 00:43:22.880
Sure.
00:43:23.280 --> 00:43:29.280
Um, so we do have a bit of data by committee, but it is a centralized committee.
00:43:29.360 --> 00:43:32.239
So we have a data governance department.
00:43:32.559 --> 00:43:40.239
Um, they are the ones who have counseled all of us in what data governance is, its importance.
00:43:40.480 --> 00:43:49.199
They actually have some of the best documentation on using their tools and finding the information in our in our data sets.
00:43:49.360 --> 00:43:52.159
We have in our metadata tool, for instance.
00:43:52.320 --> 00:43:55.920
Um, they are great at putting that information together.
00:43:56.079 --> 00:43:59.440
We do also currently have a data management office.
00:43:59.760 --> 00:44:08.320
They lead and have essentially regular meetings for data folks in the org to contribute to conversations.
00:44:09.039 --> 00:44:13.440
Um because I do work in product RD.
00:44:13.760 --> 00:44:25.280
A lot of the agenics for that we're working on came from our team, but there are other folks working in different areas of the org and doing other experiments.
00:44:25.519 --> 00:44:31.280
So we do have that centralized area to share that information.
00:44:31.679 --> 00:44:51.519
Um, in RD itself, we have a concept of communities of practice at DataSight, which has been absolutely invaluable for sharing the research and insights and experiments that folks throughout the product organization are doing.
00:44:51.760 --> 00:45:10.559
We have communities of practice on data, on AI, on Python, on back-end engineering, where we are doing demos, sharing thoughts, asking for feedback, and it is cross-team, oftentimes cross-org, where anybody is welcome to join.
00:45:10.800 --> 00:45:22.960
And it is for the purpose of sharing information, sharing philosophies, and sharing methods of practice and determining how we want to carry things out.
00:45:23.199 --> 00:45:32.079
So those are kind of the mechanisms that we have used for collaboration that have been pretty effective.
00:45:32.480 --> 00:45:33.840
I think that's extremely helpful.
00:45:34.000 --> 00:45:40.960
I mean, I think there is the the communities of practice model, which is like people that are actually building what's working, what's not working.
00:45:41.199 --> 00:45:51.360
And then when there's alignment uh and an agreed upon uh pond standard, that can then be, you know, bubbled up into like a more kind of formalized adoption within the organization.
00:45:51.760 --> 00:45:53.199
I don't care what it does.
00:45:53.599 --> 00:45:53.840
Yeah.
00:45:54.000 --> 00:45:57.199
Well, Heather, this has been a fascinating conversation for me.
00:45:57.360 --> 00:45:58.559
I've I've learned a lot.
00:45:58.719 --> 00:46:01.039
I know that others will take away a lot from it.
00:46:01.199 --> 00:46:09.360
If um there's one thing that you wanted to leave the audience or anything you wanted to share uh before we close, what would that be?
00:46:10.400 --> 00:46:17.519
I mean, it's definitely a period of experimentation and excitement.
00:46:17.760 --> 00:46:19.360
It can seem overwhelming.
00:46:19.519 --> 00:46:23.039
There's a lot happening all at the same time.
00:46:23.760 --> 00:46:29.760
Don't hesitate to experiment in a sandbox, not in prod, in a sandbox.
00:46:30.000 --> 00:46:43.920
Share the information, but don't forget about some of the core foundational data concepts, like your governance standards, like your cleanliness, like your data quality that are more important now than ever.
