WEBVTT
00:00:05.919 --> 00:00:08.080
Welcome to the Dashboard Effect Podcast.
00:00:08.160 --> 00:00:09.199
I'm Brick Thompson.
00:00:09.519 --> 00:00:10.800
And I'm Kate Eberley.
00:00:11.119 --> 00:00:17.359
Kate, we wanted to talk today about data governance in the context of BI and AI.
00:00:17.679 --> 00:00:21.199
Specifically AI sort of conversational analytics.
00:00:21.440 --> 00:00:26.000
So when you're plugging your LLM into your data lake house, that type of thing.
00:00:26.640 --> 00:00:32.159
You recently were part of a presentation to a group of lawyers talking about this.
00:00:32.240 --> 00:00:34.560
So I wanted to kind of pick your brain on that a little bit.
00:00:34.880 --> 00:00:39.679
Maybe you can just kind of lay the ground, lay the scene here for us to have a chat.
00:00:40.159 --> 00:00:40.399
Yeah.
00:00:40.479 --> 00:00:45.759
So this is a really great conversation with a group of really analytics practitioners in the legal space.
00:00:45.920 --> 00:00:49.119
So people who live and breathe working with data.
00:00:49.439 --> 00:00:53.840
For so many years, I feel like self-service analytics has been on the horizon.
00:00:53.920 --> 00:01:08.000
You know, there are ways that you can structure a semantic model to allow somebody to engage with the data, maybe in Excel or Power BI, but you still have to have a relatively high degree of competency in data in Power BI to access it that way.
00:01:08.319 --> 00:01:28.959
With AI and conversational analytics, things have changed so drastically in such a short period of time that while we thought that conversation was actually going to be very focused on semantic models and everything that you could do to combine data from multiple sources, shockingly, I'd say probably 20 minutes of that hour was actually just focused on how do you govern the data well?
00:01:29.120 --> 00:01:32.079
How do you make sure the right information is getting to the right people?
00:01:32.400 --> 00:01:46.560
And even with more access and ability to flex the information that you're getting, so you're not just relying on a report, how do you make sure people interpret that data correctly and aren't just making some of the wrong conclusions from data because they maybe weren't trained the same way?
00:01:46.879 --> 00:01:47.840
Yeah, interesting.
00:01:48.079 --> 00:01:56.640
So when you're connecting an LLM, say Claude or ChatGPT to a single transactional system, let's say you connect it to your HubSpot.
00:01:57.040 --> 00:02:08.159
Um obviously when you hit the or when you set up the MCP connector, you have to log in with your account, and then all of the permissions that you get normally when you're in HubSpot pass through.
00:02:08.400 --> 00:02:12.159
And so the LLM can't show you data that is not yours.
00:02:12.400 --> 00:02:14.000
In fact, I don't think it can even pull it.
00:02:14.319 --> 00:02:18.960
So another question about how to use that data well, but let's assume that the LLM knows how to do that well.
00:02:19.199 --> 00:02:23.439
It gets more complicated when you have a more complex data environment.
00:02:23.680 --> 00:02:33.599
Let's say you've got data from six or seven different transactional systems, um, you're pulling that into a data lake house, and then you're doing modeling to relate that data to each other.
00:02:33.759 --> 00:02:35.439
Um, gets a lot more complicated then.
00:02:35.520 --> 00:02:43.120
You still have to make sure people only see the data they're supposed to see, but you're not routing back to the transactional system to see what they're allowed to see.
00:02:43.199 --> 00:02:45.520
You have to do that in the lake house.
00:02:45.680 --> 00:02:53.360
And then as you said, you also need to help people make sure they're not pulling the data in some way that's telling them something that's not correct.
00:02:53.680 --> 00:02:55.520
So how do you go at that?
00:02:56.000 --> 00:02:58.639
Yeah, so I think there are really two lanes.
00:02:58.879 --> 00:03:02.800
One is more organizational, creating the right chains of approval.
00:03:03.039 --> 00:03:06.400
A lot of folks were talking about having actual data governance committees.
00:03:06.560 --> 00:03:12.400
And then I think there is actually that you know more technical component with security and just managing permissions.
00:03:12.639 --> 00:03:18.800
When it came to the organizational piece, what really stood out is that there are multiple layers of that governance and control.
00:03:19.360 --> 00:03:37.599
It was often the case that a lot of these organizations had data governance committees that were people who were engaged with the data, maybe not your highest level leadership, but folks who could dedicate an hour or two every couple of weeks to meet, review the reports that are out there with folks and really understand, okay, how are people consuming this?
00:03:37.759 --> 00:03:39.599
What are requests for data looking like?
00:03:39.840 --> 00:03:43.520
And what do we need to do to actually serve people the information that they need?
00:03:43.919 --> 00:03:51.360
From there down from the committee, there was a lot of feedback around actually training people who were interested in self-service analytics.
00:03:51.439 --> 00:04:15.759
So while you may have an analyst or um a department head who wants to be able to take those existing reports, manipulate them, answer their ad hoc questions, there was a lot of conversation around making sure they had really rigorous training on everything that the model was capable of, how to actually validate those results against existing reports, making sure that they had somebody on the data team that they could come ask questions to.
00:04:16.000 --> 00:04:18.319
Obviously, that takes a lot of care and feeding.
00:04:18.399 --> 00:04:21.759
So maybe you aren't doing that with everybody who's engaging with data.
00:04:21.839 --> 00:04:25.439
And in fact, that's why you'd probably keep that self-service group pretty small.
00:04:25.759 --> 00:04:32.399
But if you kept that group, you kept folks who were actually using BI more broadly, and those are your trusted reports.
00:04:32.639 --> 00:04:40.399
That was really the organizational framework to say we've got some champions for data, and then we're actually going to constrict most people into trusted BI.
00:04:40.800 --> 00:04:41.360
Okay, great.
00:04:41.519 --> 00:04:49.120
So when you say model, you're talking about the data model, the reporting data model in a medallion architecture, that would be your gold layer.
00:04:49.279 --> 00:04:55.680
Um, and then uh you can connect your BI tools to that, RBI, Tableau, whatever, what have you.
00:04:55.839 --> 00:05:01.120
Um, you can also connect LLM models to that, um, although there's more to do there.
00:05:01.279 --> 00:05:08.560
Um, did you guys talk about how to do that training uh for people so that they can connect to the data model and extract it well?
00:05:08.720 --> 00:05:11.680
And were there any great insights that came out of that?
00:05:12.319 --> 00:05:13.839
We did touch on it a little bit.
00:05:13.920 --> 00:05:19.199
You know, I think one of the takeaways was to get that one-on-one, you know, touch.
00:05:19.600 --> 00:05:28.079
Uh so a lot of times when we deploy reports, it may have a pre-recorded session on here's how you use these filters, here's how you navigate a Power BI report.
00:05:28.240 --> 00:05:36.319
But when you're thinking about self-serve, the margin for error or generating results that maybe you wouldn't have anticipated is a lot higher.
00:05:36.560 --> 00:05:41.439
So some of that initial walkthrough was maybe more technical than the typical user would get.
00:05:41.920 --> 00:05:57.040
Let's actually show you how a star schema works, how you're pulling data from each of these individual tables, and maybe actually how to produce maybe an exceptions report or another tool that would actually allow you to see and validate that you're getting the results that you would expect along the way.
00:05:57.439 --> 00:05:57.680
Okay.
00:05:57.759 --> 00:06:07.279
So this is for users who are actually building a Power BI report themselves, not just exporting data or uh connecting to a data source, that type of thing.
00:06:08.079 --> 00:06:10.879
So, all right, so you've got pretty technical users there.
00:06:11.040 --> 00:06:16.720
What if you have just business users that want to be able to do more ad hoc type of stuff with the data?
00:06:17.279 --> 00:06:24.480
I think that question was still very much out with this group, because you want to make sure that you're producing the same answers every time for folks.
00:06:24.800 --> 00:06:43.759
That's where I think, from a conversational analytics perspective, having that really well-defined and locked down MCP server in place with a lot of rich context with those adjusted platinum layer tables for specific individuals, maybe over others, I think would definitely come into play to making sure it is more democratic.
00:06:43.839 --> 00:06:46.800
You know, you're not just keeping data with a handful of folks.
00:06:46.879 --> 00:06:47.040
Yeah.
00:06:47.360 --> 00:06:49.759
But I'm also curious, you're talking to a lot of our clients.
00:06:49.920 --> 00:06:50.959
What are you seeing out there?
00:06:51.360 --> 00:06:51.600
Yeah.
00:06:51.839 --> 00:07:09.040
So by platinum layer, you're referring to sort of that extra layer on top of the gold layer in your medallion architecture that is for an LLM to be able to get the data that it wants, uh, or in a way that it can uh use it well and make sure it's delivering good results to you.
00:07:09.279 --> 00:07:17.199
Um, you know, uh what we're seeing there is very similar to with BI, except that we're not having to train users on data models.
00:07:17.279 --> 00:07:50.959
We're doing that in the platinum layer, and then we're adding a whole bunch of context in markdown files to be able to tell the MCP, well, transfer through the MCP server to the LLM how the model works, um, you know, any idiosyncrasies about the particular business, how they might refer to different uh measures and KPIs, metrics, that type of thing, um rules it should follow, what a good query against the data lake house would look like, what bad queries might look like, all of that stuff so that the user really doesn't have to be technical at all.
00:07:51.199 --> 00:08:01.839
They can just have uh conversational, you know, what we're calling conversational analytics, but asking for data and reports from their LLM, and it will do a good job returning the right stuff.
00:08:02.079 --> 00:08:10.879
You're still going to have problems sometimes because it's um non-deterministic, meaning that it, you know, it could, there's some probability that it'll give a wrong answer.
00:08:11.040 --> 00:08:16.800
But the more context you give it and the better you make that platinum layer model, the more you can lock that down.
00:08:16.959 --> 00:08:21.120
And you know, today's LLMs less and less are actually hallucinating.
00:08:21.279 --> 00:08:25.920
It's more a matter of setting things up so that the LLM actually understands the model well.
00:08:26.160 --> 00:08:31.040
Um and so that tends to be sort of an iterative process, especially at the beginning.
00:08:31.199 --> 00:08:38.320
We we typically, for our clients, have done BI work and so, or are doing BI work, so we can build on top of that.
00:08:38.480 --> 00:08:41.519
We have a lot of knowledge about the business and the data models.
00:08:41.600 --> 00:08:58.000
Um, but if we're coming in cold, um then we have to do a lot of interviewing of stakeholders and make sure we really understand it and then test it with a group of beta testers, business people though, who are giving feedback and who are being super critical about or you know, realizing it might give a wrong answer, that type of thing.
00:08:58.159 --> 00:09:00.159
So it still tends to be iterative.
00:09:00.320 --> 00:09:05.519
You can't you can't assume you set it up from the first second, it's going to be perfect right from the start.
00:09:05.679 --> 00:09:07.600
Although it's amazing how close we can get.
00:09:08.000 --> 00:09:08.240
Yeah.
00:09:08.399 --> 00:09:14.399
What stands out to me there is, and this came up in the conversation, was this idea of you're still curating the data.
00:09:14.559 --> 00:09:23.679
We're still really looking at what each of those systems is producing, looking at a well-documented data model that's connecting that data, validating all of those metrics.
00:09:23.759 --> 00:09:34.320
And then even as you get to the platinum layer, making sure that's designed to really answer very common questions well for those business users who may just not have as much data literacy.
00:09:34.480 --> 00:09:34.639
Yeah.
00:09:34.879 --> 00:09:42.000
So I do think that curation, circling back to that data committee is where you kind of see all of this start to connect.
00:09:42.159 --> 00:10:09.200
So you may have a committee who's out there figuring out what their roadmap for BI looks like, what reporting areas you're going to tackle next, aligning on documentation, your actual data team are the ones who are doing the hard work of preparing all of that and bringing it to that group for review, but then are also liaising with those end users, maybe starting with that locked-down initial group of folks who can give the critical feedback so that you can get to a point where we're finally at self-service data analytics for a broader population.
00:10:09.519 --> 00:10:10.399
Yeah, I like that.
00:10:10.639 --> 00:10:11.759
So it's really two parts.
00:10:11.840 --> 00:10:17.120
Um at the very beginning, I talked about determining what data people are allowed to get to.
00:10:17.360 --> 00:10:19.039
There's a whole other discussion there.
00:10:19.120 --> 00:10:31.039
And then um we've now been talking about how do you get them so that they can actually use the data well and not not pulling things in a way that's giving them false data or or a false impression about the data.
00:10:31.279 --> 00:10:41.440
Um, just to jump back to the who gets access to what, it's really important once you get to conversational analytics, because the LLMs are really good at digging into every nook and cranny.
00:10:41.519 --> 00:10:54.480
Um so you want to really think about, okay, how are we going to set up our user groups, um, our enter groups so so people have um uh membership in those, and that determines what data they're allowed to get to.
00:10:54.639 --> 00:10:56.320
That carries through to other systems too.
00:10:56.480 --> 00:11:00.639
I mean, you can set up uh the groups in Snowflake or BigQuery, that type of thing.
00:11:00.720 --> 00:11:02.480
You can map all of that as well.
00:11:02.720 --> 00:11:05.360
But you need to think about that even more, I think.
00:11:05.600 --> 00:11:14.080
I've I've had the experience of being surprised what data all of a sudden I've seen coming out of my LLM when I'm doing research, because it can get to everything.
00:11:14.320 --> 00:11:21.840
Whereas as an individual user um attaching to, say, a reporting data model, I might not get that deep.
00:11:22.000 --> 00:11:25.679
Um so that you need to think about that even more, I think, in this situation.
00:11:26.159 --> 00:11:26.320
Yeah.
00:11:26.559 --> 00:11:31.440
No, I think then you know we talked about the organizational piece, that is still looping back to the technical component.
00:11:31.759 --> 00:11:38.399
You need to understand what roles have access to data, in addition to maybe the curation for specific use cases or functions.
00:11:38.960 --> 00:11:41.360
So both of those things really do go hand in hand.
00:11:41.519 --> 00:11:54.879
And I think we're just seeing now more than ever the pressure to answer those questions really quickly and just accelerate how fast we're getting to that point of documentation, curation, and then good security rules to keep data under control.
00:11:55.200 --> 00:11:55.600
Yep.
00:11:55.919 --> 00:11:59.440
So ton of progress in the last, I would say, eight or nine months on this.
00:11:59.600 --> 00:12:14.639
The the um LLMs have gotten really good, which has caused us to have to think about okay, how do we surface this for conversational analytics, which has also led us to thinking about how do we service it just for um regular, sort of traditional BI users as well.
00:12:14.799 --> 00:12:17.840
And I think there's a just been a ton of progress there and a ton of value.
00:12:18.080 --> 00:12:26.320
I mean, we're seeing clients, you know, executives, senior executives that are pulling really cool analysis on questions they have that would have been hard before.
00:12:26.480 --> 00:12:29.519
You'd have to get a business analyst to spend a couple of weeks to get it for you.
00:12:29.759 --> 00:12:33.279
Now you can get it in a chat in 20 minutes, you know, that type of thing.
00:12:33.440 --> 00:12:34.320
So Yeah.
00:12:34.720 --> 00:12:39.840
I do think there's one outstanding item here because we've talked about governance for years.
00:12:39.919 --> 00:12:44.960
You know, ever since I started working in data 10 years ago, data governance was a huge topic.
00:12:45.200 --> 00:12:52.720
One of the things I still think remains to be seen is how we can navigate, really incentivizing people to participate in that process.
00:12:52.960 --> 00:13:02.720
A lot of times you the folks that you want on that data committee, maybe they're not your executives, but they're lower level VPs, you know, folks who are still very busy and stretched very thin.
00:13:02.960 --> 00:13:06.639
So I do think for anyone out there who's thinking through, well, man, that sounds great.
00:13:06.879 --> 00:13:08.559
How do I set up my data committee?
00:13:08.720 --> 00:13:11.200
How do I translate that to my data team?
00:13:11.519 --> 00:13:16.240
Would definitely underscore making sure that you have visibility into adoption as it stands today.
00:13:16.320 --> 00:13:21.519
So making sure you kind of have that pre-test and post-test around who's using data and how much.
00:13:21.840 --> 00:13:25.360
Because then I think you can start to compare, wow, we've really seen the uptick.
00:13:25.519 --> 00:13:28.320
People just want to engage in this data intensely.
00:13:28.799 --> 00:13:40.960
And you're starting to build some of those ROI cases to attract people to actually want to be part of that team because they're materially driving forward the business if they can equip people with the data they need to do their jobs more effectively.
00:13:41.200 --> 00:13:42.399
Yeah, I love that.
00:13:42.720 --> 00:13:48.080
Okay, well, uh, we're running over on time, so we'll go ahead and sign off, but we can come back and talk about this further.
00:13:48.240 --> 00:13:50.399
I I know there's a lot of interesting ground to cover.
00:13:50.639 --> 00:13:50.960
Awesome.
00:13:51.039 --> 00:13:51.519
Thanks very much.
00:13:51.840 --> 00:13:52.960
All right, thank you.
00:00:05.919 --> 00:00:08.080
Welcome to the Dashboard Effect Podcast.
00:00:08.160 --> 00:00:09.199
I'm Brick Thompson.
00:00:09.519 --> 00:00:10.800
And I'm Kate Eberley.
00:00:11.119 --> 00:00:17.359
Kate, we wanted to talk today about data governance in the context of BI and AI.
00:00:17.679 --> 00:00:21.199
Specifically AI sort of conversational analytics.
00:00:21.440 --> 00:00:26.000
So when you're plugging your LLM into your data lake house, that type of thing.
00:00:26.640 --> 00:00:32.159
You recently were part of a presentation to a group of lawyers talking about this.
00:00:32.240 --> 00:00:34.560
So I wanted to kind of pick your brain on that a little bit.
00:00:34.880 --> 00:00:39.679
Maybe you can just kind of lay the ground, lay the scene here for us to have a chat.
00:00:40.159 --> 00:00:40.399
Yeah.
00:00:40.479 --> 00:00:45.759
So this is a really great conversation with a group of really analytics practitioners in the legal space.
00:00:45.920 --> 00:00:49.119
So people who live and breathe working with data.
00:00:49.439 --> 00:00:53.840
For so many years, I feel like self-service analytics has been on the horizon.
00:00:53.920 --> 00:01:08.000
You know, there are ways that you can structure a semantic model to allow somebody to engage with the data, maybe in Excel or Power BI, but you still have to have a relatively high degree of competency in data in Power BI to access it that way.
00:01:08.319 --> 00:01:28.959
With AI and conversational analytics, things have changed so drastically in such a short period of time that while we thought that conversation was actually going to be very focused on semantic models and everything that you could do to combine data from multiple sources, shockingly, I'd say probably 20 minutes of that hour was actually just focused on how do you govern the data well?
00:01:29.120 --> 00:01:32.079
How do you make sure the right information is getting to the right people?
00:01:32.400 --> 00:01:46.560
And even with more access and ability to flex the information that you're getting, so you're not just relying on a report, how do you make sure people interpret that data correctly and aren't just making some of the wrong conclusions from data because they maybe weren't trained the same way?
00:01:46.879 --> 00:01:47.840
Yeah, interesting.
00:01:48.079 --> 00:01:56.640
So when you're connecting an LLM, say Claude or ChatGPT to a single transactional system, let's say you connect it to your HubSpot.
00:01:57.040 --> 00:02:08.159
Um obviously when you hit the or when you set up the MCP connector, you have to log in with your account, and then all of the permissions that you get normally when you're in HubSpot pass through.
00:02:08.400 --> 00:02:12.159
And so the LLM can't show you data that is not yours.
00:02:12.400 --> 00:02:14.000
In fact, I don't think it can even pull it.
00:02:14.319 --> 00:02:18.960
So another question about how to use that data well, but let's assume that the LLM knows how to do that well.
00:02:19.199 --> 00:02:23.439
It gets more complicated when you have a more complex data environment.
00:02:23.680 --> 00:02:33.599
Let's say you've got data from six or seven different transactional systems, um, you're pulling that into a data lake house, and then you're doing modeling to relate that data to each other.
00:02:33.759 --> 00:02:35.439
Um, gets a lot more complicated then.
00:02:35.520 --> 00:02:43.120
You still have to make sure people only see the data they're supposed to see, but you're not routing back to the transactional system to see what they're allowed to see.
00:02:43.199 --> 00:02:45.520
You have to do that in the lake house.
00:02:45.680 --> 00:02:53.360
And then as you said, you also need to help people make sure they're not pulling the data in some way that's telling them something that's not correct.
00:02:53.680 --> 00:02:55.520
So how do you go at that?
00:02:56.000 --> 00:02:58.639
Yeah, so I think there are really two lanes.
00:02:58.879 --> 00:03:02.800
One is more organizational, creating the right chains of approval.
00:03:03.039 --> 00:03:06.400
A lot of folks were talking about having actual data governance committees.
00:03:06.560 --> 00:03:12.400
And then I think there is actually that you know more technical component with security and just managing permissions.
00:03:12.639 --> 00:03:18.800
When it came to the organizational piece, what really stood out is that there are multiple layers of that governance and control.
00:03:19.360 --> 00:03:37.599
It was often the case that a lot of these organizations had data governance committees that were people who were engaged with the data, maybe not your highest level leadership, but folks who could dedicate an hour or two every couple of weeks to meet, review the reports that are out there with folks and really understand, okay, how are people consuming this?
00:03:37.759 --> 00:03:39.599
What are requests for data looking like?
00:03:39.840 --> 00:03:43.520
And what do we need to do to actually serve people the information that they need?
00:03:43.919 --> 00:03:51.360
From there down from the committee, there was a lot of feedback around actually training people who were interested in self-service analytics.
00:03:51.439 --> 00:04:15.759
So while you may have an analyst or um a department head who wants to be able to take those existing reports, manipulate them, answer their ad hoc questions, there was a lot of conversation around making sure they had really rigorous training on everything that the model was capable of, how to actually validate those results against existing reports, making sure that they had somebody on the data team that they could come ask questions to.
00:04:16.000 --> 00:04:18.319
Obviously, that takes a lot of care and feeding.
00:04:18.399 --> 00:04:21.759
So maybe you aren't doing that with everybody who's engaging with data.
00:04:21.839 --> 00:04:25.439
And in fact, that's why you'd probably keep that self-service group pretty small.
00:04:25.759 --> 00:04:32.399
But if you kept that group, you kept folks who were actually using BI more broadly, and those are your trusted reports.
00:04:32.639 --> 00:04:40.399
That was really the organizational framework to say we've got some champions for data, and then we're actually going to constrict most people into trusted BI.
00:04:40.800 --> 00:04:41.360
Okay, great.
00:04:41.519 --> 00:04:49.120
So when you say model, you're talking about the data model, the reporting data model in a medallion architecture, that would be your gold layer.
00:04:49.279 --> 00:04:55.680
Um, and then uh you can connect your BI tools to that, RBI, Tableau, whatever, what have you.
00:04:55.839 --> 00:05:01.120
Um, you can also connect LLM models to that, um, although there's more to do there.
00:05:01.279 --> 00:05:08.560
Um, did you guys talk about how to do that training uh for people so that they can connect to the data model and extract it well?
00:05:08.720 --> 00:05:11.680
And were there any great insights that came out of that?
00:05:12.319 --> 00:05:13.839
We did touch on it a little bit.
00:05:13.920 --> 00:05:19.199
You know, I think one of the takeaways was to get that one-on-one, you know, touch.
00:05:19.600 --> 00:05:28.079
Uh so a lot of times when we deploy reports, it may have a pre-recorded session on here's how you use these filters, here's how you navigate a Power BI report.
00:05:28.240 --> 00:05:36.319
But when you're thinking about self-serve, the margin for error or generating results that maybe you wouldn't have anticipated is a lot higher.
00:05:36.560 --> 00:05:41.439
So some of that initial walkthrough was maybe more technical than the typical user would get.
00:05:41.920 --> 00:05:57.040
Let's actually show you how a star schema works, how you're pulling data from each of these individual tables, and maybe actually how to produce maybe an exceptions report or another tool that would actually allow you to see and validate that you're getting the results that you would expect along the way.
00:05:57.439 --> 00:05:57.680
Okay.
00:05:57.759 --> 00:06:07.279
So this is for users who are actually building a Power BI report themselves, not just exporting data or uh connecting to a data source, that type of thing.
00:06:08.079 --> 00:06:10.879
So, all right, so you've got pretty technical users there.
00:06:11.040 --> 00:06:16.720
What if you have just business users that want to be able to do more ad hoc type of stuff with the data?
00:06:17.279 --> 00:06:24.480
I think that question was still very much out with this group, because you want to make sure that you're producing the same answers every time for folks.
00:06:24.800 --> 00:06:43.759
That's where I think, from a conversational analytics perspective, having that really well-defined and locked down MCP server in place with a lot of rich context with those adjusted platinum layer tables for specific individuals, maybe over others, I think would definitely come into play to making sure it is more democratic.
00:06:43.839 --> 00:06:46.800
You know, you're not just keeping data with a handful of folks.
00:06:46.879 --> 00:06:47.040
Yeah.
00:06:47.360 --> 00:06:49.759
But I'm also curious, you're talking to a lot of our clients.
00:06:49.920 --> 00:06:50.959
What are you seeing out there?
00:06:51.360 --> 00:06:51.600
Yeah.
00:06:51.839 --> 00:07:09.040
So by platinum layer, you're referring to sort of that extra layer on top of the gold layer in your medallion architecture that is for an LLM to be able to get the data that it wants, uh, or in a way that it can uh use it well and make sure it's delivering good results to you.
00:07:09.279 --> 00:07:17.199
Um, you know, uh what we're seeing there is very similar to with BI, except that we're not having to train users on data models.
00:07:17.279 --> 00:07:50.959
We're doing that in the platinum layer, and then we're adding a whole bunch of context in markdown files to be able to tell the MCP, well, transfer through the MCP server to the LLM how the model works, um, you know, any idiosyncrasies about the particular business, how they might refer to different uh measures and KPIs, metrics, that type of thing, um rules it should follow, what a good query against the data lake house would look like, what bad queries might look like, all of that stuff so that the user really doesn't have to be technical at all.
00:07:51.199 --> 00:08:01.839
They can just have uh conversational, you know, what we're calling conversational analytics, but asking for data and reports from their LLM, and it will do a good job returning the right stuff.
00:08:02.079 --> 00:08:10.879
You're still going to have problems sometimes because it's um non-deterministic, meaning that it, you know, it could, there's some probability that it'll give a wrong answer.
00:08:11.040 --> 00:08:16.800
But the more context you give it and the better you make that platinum layer model, the more you can lock that down.
00:08:16.959 --> 00:08:21.120
And you know, today's LLMs less and less are actually hallucinating.
00:08:21.279 --> 00:08:25.920
It's more a matter of setting things up so that the LLM actually understands the model well.
00:08:26.160 --> 00:08:31.040
Um and so that tends to be sort of an iterative process, especially at the beginning.
00:08:31.199 --> 00:08:38.320
We we typically, for our clients, have done BI work and so, or are doing BI work, so we can build on top of that.
00:08:38.480 --> 00:08:41.519
We have a lot of knowledge about the business and the data models.
00:08:41.600 --> 00:08:58.000
Um, but if we're coming in cold, um then we have to do a lot of interviewing of stakeholders and make sure we really understand it and then test it with a group of beta testers, business people though, who are giving feedback and who are being super critical about or you know, realizing it might give a wrong answer, that type of thing.
00:08:58.159 --> 00:09:00.159
So it still tends to be iterative.
00:09:00.320 --> 00:09:05.519
You can't you can't assume you set it up from the first second, it's going to be perfect right from the start.
00:09:05.679 --> 00:09:07.600
Although it's amazing how close we can get.
00:09:08.000 --> 00:09:08.240
Yeah.
00:09:08.399 --> 00:09:14.399
What stands out to me there is, and this came up in the conversation, was this idea of you're still curating the data.
00:09:14.559 --> 00:09:23.679
We're still really looking at what each of those systems is producing, looking at a well-documented data model that's connecting that data, validating all of those metrics.
00:09:23.759 --> 00:09:34.320
And then even as you get to the platinum layer, making sure that's designed to really answer very common questions well for those business users who may just not have as much data literacy.
00:09:34.480 --> 00:09:34.639
Yeah.
00:09:34.879 --> 00:09:42.000
So I do think that curation, circling back to that data committee is where you kind of see all of this start to connect.
00:09:42.159 --> 00:10:09.200
So you may have a committee who's out there figuring out what their roadmap for BI looks like, what reporting areas you're going to tackle next, aligning on documentation, your actual data team are the ones who are doing the hard work of preparing all of that and bringing it to that group for review, but then are also liaising with those end users, maybe starting with that locked-down initial group of folks who can give the critical feedback so that you can get to a point where we're finally at self-service data analytics for a broader population.
00:10:09.519 --> 00:10:10.399
Yeah, I like that.
00:10:10.639 --> 00:10:11.759
So it's really two parts.
00:10:11.840 --> 00:10:17.120
Um at the very beginning, I talked about determining what data people are allowed to get to.
00:10:17.360 --> 00:10:19.039
There's a whole other discussion there.
00:10:19.120 --> 00:10:31.039
And then um we've now been talking about how do you get them so that they can actually use the data well and not not pulling things in a way that's giving them false data or or a false impression about the data.
00:10:31.279 --> 00:10:41.440
Um, just to jump back to the who gets access to what, it's really important once you get to conversational analytics, because the LLMs are really good at digging into every nook and cranny.
00:10:41.519 --> 00:10:54.480
Um so you want to really think about, okay, how are we going to set up our user groups, um, our enter groups so so people have um uh membership in those, and that determines what data they're allowed to get to.
00:10:54.639 --> 00:10:56.320
That carries through to other systems too.
00:10:56.480 --> 00:11:00.639
I mean, you can set up uh the groups in Snowflake or BigQuery, that type of thing.
00:11:00.720 --> 00:11:02.480
You can map all of that as well.
00:11:02.720 --> 00:11:05.360
But you need to think about that even more, I think.
00:11:05.600 --> 00:11:14.080
I've I've had the experience of being surprised what data all of a sudden I've seen coming out of my LLM when I'm doing research, because it can get to everything.
00:11:14.320 --> 00:11:21.840
Whereas as an individual user um attaching to, say, a reporting data model, I might not get that deep.
00:11:22.000 --> 00:11:25.679
Um so that you need to think about that even more, I think, in this situation.
00:11:26.159 --> 00:11:26.320
Yeah.
00:11:26.559 --> 00:11:31.440
No, I think then you know we talked about the organizational piece, that is still looping back to the technical component.
00:11:31.759 --> 00:11:38.399
You need to understand what roles have access to data, in addition to maybe the curation for specific use cases or functions.
00:11:38.960 --> 00:11:41.360
So both of those things really do go hand in hand.
00:11:41.519 --> 00:11:54.879
And I think we're just seeing now more than ever the pressure to answer those questions really quickly and just accelerate how fast we're getting to that point of documentation, curation, and then good security rules to keep data under control.
00:11:55.200 --> 00:11:55.600
Yep.
00:11:55.919 --> 00:11:59.440
So ton of progress in the last, I would say, eight or nine months on this.
00:11:59.600 --> 00:12:14.639
The the um LLMs have gotten really good, which has caused us to have to think about okay, how do we surface this for conversational analytics, which has also led us to thinking about how do we service it just for um regular, sort of traditional BI users as well.
00:12:14.799 --> 00:12:17.840
And I think there's a just been a ton of progress there and a ton of value.
00:12:18.080 --> 00:12:26.320
I mean, we're seeing clients, you know, executives, senior executives that are pulling really cool analysis on questions they have that would have been hard before.
00:12:26.480 --> 00:12:29.519
You'd have to get a business analyst to spend a couple of weeks to get it for you.
00:12:29.759 --> 00:12:33.279
Now you can get it in a chat in 20 minutes, you know, that type of thing.
00:12:33.440 --> 00:12:34.320
So Yeah.
00:12:34.720 --> 00:12:39.840
I do think there's one outstanding item here because we've talked about governance for years.
00:12:39.919 --> 00:12:44.960
You know, ever since I started working in data 10 years ago, data governance was a huge topic.
00:12:45.200 --> 00:12:52.720
One of the things I still think remains to be seen is how we can navigate, really incentivizing people to participate in that process.
00:12:52.960 --> 00:13:02.720
A lot of times you the folks that you want on that data committee, maybe they're not your executives, but they're lower level VPs, you know, folks who are still very busy and stretched very thin.
00:13:02.960 --> 00:13:06.639
So I do think for anyone out there who's thinking through, well, man, that sounds great.
00:13:06.879 --> 00:13:08.559
How do I set up my data committee?
00:13:08.720 --> 00:13:11.200
How do I translate that to my data team?
00:13:11.519 --> 00:13:16.240
Would definitely underscore making sure that you have visibility into adoption as it stands today.
00:13:16.320 --> 00:13:21.519
So making sure you kind of have that pre-test and post-test around who's using data and how much.
00:13:21.840 --> 00:13:25.360
Because then I think you can start to compare, wow, we've really seen the uptick.
00:13:25.519 --> 00:13:28.320
People just want to engage in this data intensely.
00:13:28.799 --> 00:13:40.960
And you're starting to build some of those ROI cases to attract people to actually want to be part of that team because they're materially driving forward the business if they can equip people with the data they need to do their jobs more effectively.
00:13:41.200 --> 00:13:42.399
Yeah, I love that.
00:13:42.720 --> 00:13:48.080
Okay, well, uh, we're running over on time, so we'll go ahead and sign off, but we can come back and talk about this further.
00:13:48.240 --> 00:13:50.399
I I know there's a lot of interesting ground to cover.
00:13:50.639 --> 00:13:50.960
Awesome.
00:13:51.039 --> 00:13:51.519
Thanks very much.
00:13:51.840 --> 00:13:52.960
All right, thank you.
