Stitching Together Enterprise Analytics With Microsoft Fabric

WEBVTT
NOTE
Transcription provided by Podhome.fm
Created: 7/6/2024 3:08:56 PM
Duration: 3202.630
Channels: 1
1
00:00:11.264 --> 00:00:15.525
Hello, and welcome to the Data Engineering podcast, the show about modern data management.
2
00:00:16.145 --> 00:00:18.085
Data lakes are notoriously complex.
3
00:00:18.500 --> 00:00:29.075
For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst is an end to end data lake has platform built on Trino, the query engine Apache Iceberg was designed for.
4
00:00:29.775 --> 00:00:34.995
Starburst has complete support for all table formats, including Apache Iceberg, Hive, and Delta Lake.
5
00:00:35.350 --> 00:00:39.530
And Starburst is trusted by teams of all sizes, including Comcast and DoorDash.
6
00:00:40.070 --> 00:00:43.290
Want to see Starburst in action? Go to data engineering podcast.com/starburst
7
00:00:44.835 --> 00:00:50.934
today and get $500 in credits to try Starburst Galaxy, the easiest and fastest way to get started using Trino.
8
00:00:51.930 --> 00:01:07.235
Your host is Tobias Macy, and today, I'd like to welcome back Dipti Borkar to talk about her work on Microsoft Fabric and performing analytics on data without having to move it to various locations. So, Deepi, can you start by introducing yourself for anybody who hasn't heard you in previous episodes?
9
00:01:07.710 --> 00:01:11.170
Yeah. Absolutely. Well, great to be back. I was remembering,
10
00:01:11.549 --> 00:01:14.450
the previous, 2 times, and so 3rd time's the charm.
11
00:01:15.310 --> 00:01:16.850
So excited to be here.
12
00:01:17.335 --> 00:01:21.034
I'm Dipdi Borkar. I'm part of the Azure data leadership team,
13
00:01:21.814 --> 00:01:24.634
VP and GM at Microsoft and responsible
14
00:01:24.935 --> 00:01:25.435
for
15
00:01:25.830 --> 00:01:27.210
our strategic ISVs,
16
00:01:27.510 --> 00:01:28.970
including Azure Databricks
17
00:01:29.510 --> 00:01:30.810
as well as our fabric,
18
00:01:31.190 --> 00:01:32.250
developer experiences.
19
00:01:32.605 --> 00:01:34.065
So data beyond Toby.
20
00:01:34.685 --> 00:01:40.770
And for folks who haven't heard you previously, if you can just remind us how you first get started working in data.
21
00:01:41.409 --> 00:01:43.429
Yeah. It's been a while now.
22
00:01:44.130 --> 00:01:47.990
20 20 plus years started off in the database lab at UC San Diego
23
00:01:48.450 --> 00:01:49.909
doing research on,
24
00:01:50.325 --> 00:01:54.985
at that point, semi structured data. So the world was moving from SQL to,
25
00:01:55.365 --> 00:01:58.985
semi structured and, yeah, you know, the the interactive
26
00:01:59.285 --> 00:02:01.460
applications were starting to go big.
27
00:02:01.760 --> 00:02:05.380
So that's kind of where I started off, started my journey in
28
00:02:05.760 --> 00:02:11.045
in d b 2 distributed on the Linux side of things in the core engine. So storage and indexing kernel.
29
00:02:12.084 --> 00:02:16.265
And I've been pretty lucky to be involved in pretty major step functions and data.
30
00:02:16.830 --> 00:02:20.770
As we were talking earlier, it hasn't been a dull day since since I started.
31
00:02:21.470 --> 00:02:23.170
And so relational to NoSQL
32
00:02:23.550 --> 00:02:24.050
with
33
00:02:24.525 --> 00:02:29.585
with JSON was a pretty big switch. So was, the forefront of that, disaggregated
34
00:02:30.045 --> 00:02:33.665
stack with a lot of open source tech, Spark, Presto,
35
00:02:34.370 --> 00:02:37.110
founded a company in that space, to drive simplicity.
36
00:02:37.890 --> 00:02:41.110
And now it's, Gen AI needs, needs data.
37
00:02:41.695 --> 00:02:42.195
And,
38
00:02:42.735 --> 00:02:44.015
and it's the next next,
39
00:02:44.655 --> 00:02:47.235
5 to 10 years of innovation. So,
40
00:02:47.535 --> 00:02:50.415
it's been a fun fun journey in data management, and,
41
00:02:50.950 --> 00:02:54.010
honestly, it's still evolving. I feel like I'm constantly learning,
42
00:02:54.629 --> 00:02:54.950
and,
43
00:02:55.590 --> 00:03:00.250
transforming myself over every time there's a new technology and trying to stay relevant.
44
00:03:01.815 --> 00:03:12.220
So recently, you moved to the team that supports the Microsoft Fabric product. I'm wondering if you can just give some sense about what it is, some of the story behind,
45
00:03:12.680 --> 00:03:17.660
how you came to be involved in that, and some of the ways that people should be thinking about Microsoft
46
00:03:17.975 --> 00:03:21.435
Fabric as it relates to their own work in their data systems?
47
00:03:21.895 --> 00:03:28.220
Yeah. Absolutely. Would, love to share. There's a lot of people working on fabric. Right? It's a pretty major investment,
48
00:03:28.920 --> 00:03:31.740
at Microsoft within, within Azure data itself.
49
00:03:32.280 --> 00:03:41.694
And, you know, if you think about if you look away from fabric for 1 minute and just look at data platform teams and what they've had to deal with, there's a lot of complexity.
50
00:03:41.995 --> 00:03:42.495
Right?
51
00:03:43.090 --> 00:03:48.870
Over the years, there have been many, many products and tools that have services that have emerged
52
00:03:49.170 --> 00:03:56.034
with the goal of solving a problem, but that has has added even more complexity and fragmentation. Right?
53
00:03:56.334 --> 00:04:01.090
When you look at, data platform teams and all the services and products that they use,
54
00:04:01.430 --> 00:04:07.890
you know, you you think of Mad Turks, mad landscape, right, machine learning AI and data landscape.
55
00:04:08.405 --> 00:04:12.905
And, in fact, this year was the 10th year of, publishing that landscape,
56
00:04:13.364 --> 00:04:26.539
and it's getting more and more and more complex. So if you the logos keep shrinking every year in size. Right? And you can barely recognize them at this point. And so if if data platform teams are dealing with that level of
57
00:04:27.625 --> 00:04:32.205
it essentially just takes a lot more time to innovate, time to get insights, time to market.
58
00:04:32.825 --> 00:04:36.580
And as we've talked with it and as the the fabric teams talk with customers,
59
00:04:37.440 --> 00:04:44.500
whether it's a CIO or a data platform deep leader, you know, organizations want to simplify. They wanna simplify and reduce costs,
60
00:04:45.025 --> 00:04:54.210
at at at at the, you know, at the basis of it. And then, you know, obviously, be faster on innovating, differentiating on top of that. Right? And so the complex architectures
61
00:04:55.230 --> 00:05:03.225
have that have been driven out of this so many products in the space, it's gotten much harder. And in some ways, that's sort of what
62
00:05:03.685 --> 00:05:05.945
fabric envisions and promises to simplify
63
00:05:06.565 --> 00:05:06.965
is,
64
00:05:07.919 --> 00:05:09.460
that, you know, you can,
65
00:05:10.000 --> 00:05:18.305
there there are many platforms that call themselves unified or, you know, that with this promise of having a simplicity across all of these components.
66
00:05:19.005 --> 00:05:26.840
But as a cloud, we do have all the pieces. We have many, many pieces that are involved, in a data in a data platform,
67
00:05:27.620 --> 00:05:30.555
from business intelligence with Power BI to
68
00:05:31.035 --> 00:05:34.175
pipelines data flows with Azure Data Factory to
69
00:05:34.475 --> 00:05:38.175
our warehouse, right, lake house, real time, intelligence.
70
00:05:38.555 --> 00:05:41.199
You know, we have event streams, event hubs, etcetera.
71
00:05:41.900 --> 00:05:46.879
All of this really can, you know, can come together, and that's what fabric is.
72
00:05:47.245 --> 00:05:50.865
It's it's, combines the best of all of these
73
00:05:51.245 --> 00:05:53.345
core products and services
74
00:05:53.724 --> 00:05:55.104
into a single unified,
75
00:05:55.645 --> 00:05:56.145
platform,
76
00:05:56.789 --> 00:06:04.090
which is a SaaS platform. It's a software as a service platform so that it it's a really seamless experience, low code, no code.
77
00:06:04.545 --> 00:06:13.765
And what we've envisioned to do is have as minimal knobs as possible so that everything is sort of auto managed for you, AI powered underneath the covers,
78
00:06:14.170 --> 00:06:16.830
and and that's the that's the unified platform,
79
00:06:17.690 --> 00:06:19.550
that customers can use. So, essentially,
80
00:06:19.930 --> 00:06:21.630
you know, taking away that complexity,
81
00:06:22.090 --> 00:06:22.410
giving
82
00:06:23.104 --> 00:06:25.845
try try to get to market faster for our customers
83
00:06:26.384 --> 00:06:29.285
is the eventual promise that that Fabric's making.
84
00:06:29.665 --> 00:06:32.085
And, you know, with the name like Fabric,
85
00:06:32.819 --> 00:06:38.120
it it kind of, encompasses the the vision is to encompass everything, and, it's been phenomenal,
86
00:06:38.580 --> 00:06:40.120
since the GA last
87
00:06:41.104 --> 00:06:41.604
November.
88
00:06:41.905 --> 00:06:43.604
At Ignite, we announced GA,
89
00:06:43.905 --> 00:06:44.705
and we have,
90
00:06:45.585 --> 00:06:49.205
we just announced a couple weeks ago where we have about 11, 000 customers
91
00:06:49.790 --> 00:06:53.490
on fabric already. And so very, very, great to see the momentum.
92
00:06:54.030 --> 00:07:01.345
And, I think it's just the beginning. You know, we'll evolve over time. There's lots to build. There's lots of adoption that will happen as well.
93
00:07:02.044 --> 00:07:17.035
With the fact that fabric is focused on being this unified location for data across an organization, particularly for larger businesses that have multiple different business units that each have their own little silos of data.
94
00:07:17.815 --> 00:07:20.155
What is your heuristic for
95
00:07:20.535 --> 00:07:27.970
when somebody is asking, is Fabric the tool for me? And what are the cases where you can say, Fabric is
96
00:07:28.430 --> 00:07:34.255
a bazooka to your fly and you're actually better suited using some other tool?
97
00:07:34.875 --> 00:07:36.794
Yeah. It's a good question. Right? And,
98
00:07:37.595 --> 00:07:43.069
1 that we get in a slightly different topic is where do I start? Right? Where what's the best place to start off?
99
00:07:43.449 --> 00:07:51.069
And what we've seen with fabric, number 1, is the the business model is very different from a range of other
100
00:07:51.535 --> 00:08:06.700
services and and so on. If you take away the, you know, the Microsoft stack for a minute and you think about a stack that, user might, data engineer might leverage in their data platform. You might have Snowflake. You might have Fivetran to do, you know, some of the
101
00:08:07.240 --> 00:08:09.080
ETL, ELT. You might have,
102
00:08:09.604 --> 00:08:13.465
Tableau to do the BI. Right? You might have DBT and other products,
103
00:08:14.085 --> 00:08:16.505
Spark and and other, other technologies.
104
00:08:17.289 --> 00:08:22.110
Each of these ends up being provisioned and oftentimes the users overprovision.
105
00:08:22.490 --> 00:08:22.990
Right?
106
00:08:23.530 --> 00:08:45.824
And each of these need to be bought separately, managed separately to the separate line item in the budget. And so when you look at the complexity across these, some of it is actually the business model and the cost. Right? With fabric, you buy 1 product. So you you buy 1 thing. Right? And you can use all of these experiences within that product. So the the business model itself is pretty transformational
107
00:08:46.445 --> 00:08:58.720
where you once you start off with fabric, you can start off with data factory where you might pull in data using a data pipeline or data flows, which have, you know, lots of 200, 300 connectors.
108
00:08:59.085 --> 00:09:07.185
And then what we've seen is once you get started with that, you then end up using more and more workloads because it's a pretty seamless experience.
109
00:09:07.500 --> 00:09:13.680
You don't have to go and ask someone for, you know, permission to buy either from a marketplace or,
110
00:09:14.220 --> 00:09:16.959
go back to your admin and say, hey. Can you, you know,
111
00:09:17.355 --> 00:09:20.415
enable this or, you know, we are running out. Can you overprovision?
112
00:09:21.195 --> 00:09:34.160
It's all of it is serverless. Right? And so first thing, you only pay for what you use within each of these, and it's all at your fingertips. So, typically, we see users start off with 1 and then use 3 3 workloads, 4 workloads,
113
00:09:34.705 --> 00:09:40.005
right, across fabric. And over time, more and more get used. And so the simplicity
114
00:09:40.385 --> 00:09:41.605
of usage
115
00:09:42.220 --> 00:09:47.440
is is what we're, you know, what we're seeing. So going back to the question of where do people get started,
116
00:09:47.740 --> 00:09:52.625
what is it good for, what what it may not be good for, What we're seeing is,
117
00:09:53.245 --> 00:09:53.745
pipelines,
118
00:09:54.205 --> 00:09:57.585
with data factory get it's used quite a bit, lake houses,
119
00:09:58.445 --> 00:09:58.945
and
120
00:09:59.700 --> 00:10:08.279
and data engineering workload gets used quite a bit, and then the warehouse. Right? So you sort of, for your medallion architecture, you typically have,
121
00:10:08.775 --> 00:10:11.275
your, you know, your raw data and then your,
122
00:10:12.055 --> 00:10:27.515
silver gold broad silver gold, etcetera. And so they're starting off with 1, and they're starting to leverage others. So depending on your need, first thing, you could you should only you should only buy the SKU that makes sense for you, and it starts we it's 2 months free trial,
123
00:10:28.295 --> 00:10:38.260
each month at $17, 000, I believe. And so that's pretty pretty great to get started off. And and then you can start off with really small SKUs. Right? The, the SKUs
124
00:10:38.639 --> 00:10:39.360
start at,
125
00:10:40.320 --> 00:10:54.780
f, f 8 and and so on, and so you can you can pay very little to begin with and then grow over time, to more to larger skews if that's if that's where you, you know, the the product takes you and you start using more workloads.
126
00:10:55.240 --> 00:11:00.060
Now in terms of, so I would say starting points tend to be pipelines, right, usage,
127
00:11:00.885 --> 00:11:05.545
ingestion. So you start with ingesting into 1 lake, which is the foundation of fabric,
128
00:11:06.085 --> 00:11:07.785
and then using other engines,
129
00:11:08.245 --> 00:11:09.305
on top of it.
130
00:11:10.310 --> 00:11:13.290
On the point of data lakes and lake houses,
131
00:11:14.150 --> 00:11:21.265
there has definitely been a significant trend towards that as the default location for data for an organization.
132
00:11:22.525 --> 00:11:25.025
Data lakes in general have come to be
133
00:11:25.339 --> 00:11:29.600
more well understood from their first introduction where there was all the talk about data swamps,
134
00:11:29.980 --> 00:11:33.279
and the lakehouse architecture in particular has helped to
135
00:11:34.665 --> 00:11:39.005
give people a common architectural perspective on how to approach that.
136
00:11:40.265 --> 00:11:50.325
And I've seen in recent years less emphasis on the data warehouse, more on the data lake or data lake house. I'm wondering what are some of the motivating factors that you see for that overall trend
137
00:11:51.025 --> 00:11:58.645
towards lakes being a more widely accepted and more widely adopted approach to how to manage data at the organizational scale?
138
00:11:59.670 --> 00:12:08.965
Yeah. Absolutely. I mean, you know, I'm I feel like I'm still learning. Like, I've been in this space for many, many years. Like, 2018, 2019 is when I got involved,
139
00:12:09.345 --> 00:12:14.485
working with a lot of the Internet companies at that point. And since then, a lot has changed. Right? So,
140
00:12:14.865 --> 00:12:17.685
in terms of motivation factors, the early ones,
141
00:12:18.100 --> 00:12:25.960
it was, this the disaggregation of the stack. Right? It was, you know, separation of storage and compute was a big deal because they could scale at different
142
00:12:26.345 --> 00:12:27.385
different levels. They,
143
00:12:27.865 --> 00:12:33.405
and cost is different for each 1 too. Right? Compute in general was a little bit more expensive. Storage,
144
00:12:33.790 --> 00:12:34.450
with object stores,
145
00:12:34.830 --> 00:12:36.029
etcetera, it it,
146
00:12:36.910 --> 00:12:38.050
it got significantly,
147
00:12:38.670 --> 00:12:45.505
you know, cheaper to store large and large amounts of data with ADLS and s 3 and others. And so I would say the,
148
00:12:45.885 --> 00:12:46.785
that that,
149
00:12:47.565 --> 00:12:49.665
the starting point, which was the disaggregated
150
00:12:50.045 --> 00:12:59.149
stack at this this, separation of storage and compute is is proven. Right? It's absolutely proven, and that continues to be 1 of the motivating factors.
151
00:12:59.605 --> 00:13:04.904
The other 1, that emerged over time is, just no lock in. Right? Open formats.
152
00:13:05.285 --> 00:13:11.540
And it started off with the file formats, which you are, you know, at that point, ORC, Avro, Parquet.
153
00:13:12.079 --> 00:13:13.600
Parquet kind of emerged as,
154
00:13:14.319 --> 00:13:19.735
the more popular 1, and then table formats layer on on top of it to make it more structured,
155
00:13:20.115 --> 00:13:31.380
to simulate in some ways a a warehouse. All the metadata, the table schema got layered on top. And, of course, you know, Delta Lake, Iceberg, and Hudi and others emerge. But foundationally,
156
00:13:32.080 --> 00:13:36.065
the motivation is, it's open formats, which means that customers are not
157
00:13:36.865 --> 00:13:39.685
in. And this has allowed for a lot more interop
158
00:13:40.065 --> 00:13:56.395
across platforms. So you can store data in 1, but you could, you know, you could may have your own engine, internal engine. A lot of enterprises, you know, innovate internally. They have they might have their own, either Python libraries or, or their own engines, whether it's a query engine and a batch engine,
159
00:13:56.775 --> 00:14:13.225
they could run on top of it. And, of course, just the interop across even the the commercialized software, right, on top of these open formats is pretty significant now. And so that continues to be a huge driver is no more walled gardens. Right? Truly open data.
160
00:14:13.605 --> 00:14:16.890
And, and this is 1 that Fabric has adopted where
161
00:14:23.110 --> 00:14:33.415
storage costs are significantly lower now. Warehouses are, where where storage costs are significantly lower now. Warehouses are, you know your more traditional warehouses are still monoliths. Right? So you you still have that
162
00:14:33.850 --> 00:14:37.310
grouping of, storage and and compute, which becomes expensive.
163
00:14:38.090 --> 00:14:51.530
Here, you pay for it in different tiers, and so costing becomes it it will become, you know, significantly cheaper over some of the other even on prem data warehouses that, customers still have. Right? A large amount of.
164
00:14:51.850 --> 00:14:54.010
And with the innovative models like,
165
00:14:54.730 --> 00:14:55.230
fabrics,
166
00:14:55.930 --> 00:15:02.295
capacity unit model where, you know, you have virtual currency that you could use and spend across all of these different,
167
00:15:02.695 --> 00:15:03.195
workloads,
168
00:15:03.975 --> 00:15:06.715
that will help reduce cost. So you're not over provisioning
169
00:15:07.095 --> 00:15:11.115
for everything. You know, together, you you have 1 product that you could use,
170
00:15:11.490 --> 00:15:12.930
with the same bucket of,
171
00:15:13.490 --> 00:15:14.870
of virtual currency.
172
00:15:15.410 --> 00:15:21.135
Now the other things that are emerging when we look at verticals. Right? So you have retail financial services,
173
00:15:22.075 --> 00:15:23.295
travel, telco,
174
00:15:24.235 --> 00:15:37.375
media, like, lot lots and lots of, you know, verticals. Each of them in some ways has become a technology company now. Right? They have some part of tech that is driving their innovation and their new products.
175
00:15:37.755 --> 00:15:56.215
But even then, it is it's very hard to, you know, keep up with new skills with, you know, new products that come up with maybe it's a a warehouse or maybe it's a new tool here and there. Data lake houses essentially create that foundation of 1 lake, which gives you visibility across, you know, all the different
176
00:15:56.595 --> 00:16:00.855
aspects of data you might have. And so from a team skill perspective,
177
00:16:01.330 --> 00:16:07.330
if you at least have some base level of skill that you can build on top of. So you're not, you know, you're not,
178
00:16:07.970 --> 00:16:25.700
every 3 years, you're not bringing in new skill new tech, which means new skills. Hopefully, this stays for a long period of time. It's lasted over you know, I would say it started off with Hadoop, which was extremely complex. Right? And now people didn't see much value of it to a point where it's completely SaaSified now
179
00:16:26.080 --> 00:16:30.745
and seamless where you don't need a lot of you don't need a PhD in data engineering
180
00:16:31.285 --> 00:16:53.785
to start using this. So, skills have, you know, the the level of skill you can start off with base level of skill now that, data lakes have become pretty, they're they're everywhere. Right? Every everyone sort of has a common understanding of a of a data lake a lakehouse and what that means. And then and finally, I would say just Gen AI and, innovation that's ongoing,
181
00:16:54.740 --> 00:16:59.399
you do need your models are only as good as your data. Right? And so increasingly,
182
00:16:59.940 --> 00:17:01.380
whether you are using,
183
00:17:01.860 --> 00:17:03.399
the core foundational models,
184
00:17:03.955 --> 00:17:04.195
you know,
185
00:17:05.075 --> 00:17:05.395
the,
186
00:17:06.275 --> 00:17:14.730
with with the OpenAI, Azure OpenAI, or other open source ones, or you're training your own, a lot of this is gonna happen on open lakes.
187
00:17:15.030 --> 00:17:18.230
And, and so that's the other reason to, you know,
188
00:17:18.790 --> 00:17:21.050
increasingly, we're seeing a lot more
189
00:17:21.615 --> 00:17:28.915
users, enterprise customers coming into us saying, hey. We, you know, we we this combination of GenAI and and OpenDataLakes
190
00:17:29.455 --> 00:17:32.409
and, and and 1 lake for us is,
191
00:17:32.789 --> 00:17:36.809
you know, is tell us more. Like, we we believe in this foundational architecture
192
00:17:37.269 --> 00:17:41.115
because that's going to be the next next set of innovation that continues,
193
00:17:41.415 --> 00:17:44.155
and we want to make sure we are starting off at the right point,
194
00:17:44.695 --> 00:17:48.450
with with open open data leak. So, you know, looking forward,
195
00:17:48.910 --> 00:17:50.850
it's going to be about Jenny and,
196
00:17:51.630 --> 00:17:52.690
and custom models,
197
00:17:53.559 --> 00:17:54.059
enterprises
198
00:17:54.405 --> 00:17:58.105
training their creating their own on top of their own proprietary data.
199
00:17:58.645 --> 00:18:03.305
All of these will, you know, will be sitting in the data will be sitting in open formats in data lakes.
200
00:18:03.920 --> 00:18:08.580
On that note of open and open formats, open lakes, Microsoft
201
00:18:08.880 --> 00:18:13.220
has been investing substantially into the open source ecosystem in recent years.
202
00:18:13.905 --> 00:18:15.525
And Fabric, I noted,
203
00:18:15.905 --> 00:18:27.720
relies at least in part on various open source components. I'm wondering if you can talk to some of the benefits that the product team has seen on layering on top of those existing open source solutions,
204
00:18:28.395 --> 00:18:35.455
and what are the elements that were built specifically for fabric that are proprietary and built in house?
205
00:18:36.020 --> 00:18:36.760
Yeah. Absolutely.
206
00:18:37.540 --> 00:18:43.080
You know, personally, it's it's been fantastic for me to see and be involved continue to be involved,
207
00:18:43.575 --> 00:18:53.900
in open source. So I've been involved in many projects in the past, over the past 12 to 15 years. And, and it's it's great to see the Microsoft data stack also,
208
00:18:54.360 --> 00:19:00.460
not only adopting open source, which is the beginning, but also giving back, being more involved in the community. Right?
209
00:19:00.935 --> 00:19:01.335
Which,
210
00:19:02.295 --> 00:19:04.935
which which not necessarily a lot of,
211
00:19:05.335 --> 00:19:05.835
companies
212
00:19:06.215 --> 00:19:12.340
do. It started off with Delta Lake. Right? So that was the the foundation. That was the primary format that we adopted.
213
00:19:12.720 --> 00:19:19.275
And, and and 1 of the big reasons was it's it's the, we have a lot of customers that are already on,
214
00:19:20.055 --> 00:19:21.015
Delta given our,
215
00:19:21.815 --> 00:19:26.475
Azure Databricks service. Right? It's a, a mission critical service for us and,
216
00:19:27.130 --> 00:19:28.110
and our customers.
217
00:19:28.410 --> 00:19:36.645
You know, we have a lot of data in ADLS, Gen 2, which is Delta. So that's kind of where we started off. In fact, to embrace Delta,
218
00:19:37.025 --> 00:19:46.390
all our engines were, you know, redid it in large parts because they needed to natively support Delta. Right? It's a pretty big change moving for from a proprietary
219
00:19:46.690 --> 00:19:51.945
format, especially for Microsoft, right, to a totally open format. And then rewriting
220
00:19:52.325 --> 00:19:53.465
SQL, rewriting
221
00:19:54.005 --> 00:19:54.745
the pipelines,
222
00:19:55.205 --> 00:20:00.184
rewriting Power BI even. Right? So that it can directly access Delta,
223
00:20:00.900 --> 00:20:09.160
even without an engine in the middle, without a SQL engine. So Power BI now supports direct lake where you could just go and directly read delta
224
00:20:09.925 --> 00:20:12.025
lake files that are up on on parquet,
225
00:20:12.565 --> 00:20:20.020
and it saves a lot of cost, and it's it's faster because we've we've optimized on top of that. We we use our
226
00:20:20.500 --> 00:20:36.960
there's a sorting order that we add on top for compression as well, which is called VertiPaq. And so we've we've taken the base open source, but we've optimized on top of it. Right? We continue to, add in our own innovation and IT, of of course, on, you know, on top of the the core open source. And so it started off with,
227
00:20:37.440 --> 00:20:38.180
with Delta.
228
00:20:38.640 --> 00:20:41.760
Few weeks ago, we announced iceberg support. Right? So it is
229
00:20:42.320 --> 00:21:01.240
iceberg is coming to 1 lake as well, and so customers now have an option of picking which format they want, which means that not only we do we have full interrupt with Azure Databricks, which is our our service, but also with Snowflake, which we announced. I will was at, the Snowflake Summit last week. And, and so that will that interrupt between
230
00:21:01.715 --> 00:21:04.515
snowflake iceberg, where it might be might be sitting in,
231
00:21:05.075 --> 00:21:08.375
s 3, for example, can now be shortcutted into 1 lake,
232
00:21:08.675 --> 00:21:14.410
and used as well. And so, you know, we truly believe that storage should be open, format should be open.
233
00:21:15.030 --> 00:21:17.850
This allows for massive interrupt across platforms.
234
00:21:18.390 --> 00:21:19.450
And with fabric,
235
00:21:20.265 --> 00:21:22.285
and 1 lake, there are capabilities
236
00:21:22.825 --> 00:21:26.765
like, you know, shortcutting and mirroring that really gets you
237
00:21:27.200 --> 00:21:35.220
the single view of your data estate. Wherever your data might be, it might be in a different cloud. It might be in a different format. All of that accessible,
238
00:21:36.065 --> 00:21:50.000
through fabric. So formats is is sort of the foundation, right, as as we all know and you're very familiar with. But on top of that, of course, the engines are now also embracing open source. And so, last week, we announced our native Spark engine,
239
00:21:50.380 --> 00:21:54.055
which is now built in with Gluten and Velox
240
00:21:54.595 --> 00:21:55.075
as,
241
00:21:55.875 --> 00:21:57.655
the it's kind of a composable
242
00:21:58.035 --> 00:22:02.230
stack down, right, for for Spark. And we've seen significant
243
00:22:02.610 --> 00:22:11.305
improvements in performance as well as reducing the amount of CPU used. Right? So you're moving from Java to to c plus plus. It's a a pretty big, optimization,
244
00:22:12.325 --> 00:22:15.465
and we've seen this with, you know, essentially the standard benchmarks, TPCBAs,
245
00:22:16.165 --> 00:22:16.665
TPC,
246
00:22:17.380 --> 00:22:18.120
HR, TPCBS
247
00:22:18.580 --> 00:22:19.220
in terms of,
248
00:22:20.020 --> 00:22:22.040
how much it benefits. Now
249
00:22:22.500 --> 00:22:24.040
there are lots, for example, is,
250
00:22:24.605 --> 00:22:27.184
essentially, it's a set of operators. Right? And
251
00:22:27.565 --> 00:22:36.019
they can it can be used within any engine. Right? In fact, it it was it's fantastic to see my my old team's innovation sort of being used,
252
00:22:36.320 --> 00:22:42.975
within within fabric, and, and we're building on on top of it and and collaborating with Intel and Meta and, and
253
00:22:43.615 --> 00:22:48.975
others. And so that has been, you know, fantastic to see. The other project that, we are,
254
00:22:49.375 --> 00:22:56.710
extremely involved within our founding members of is, the Apache x table. So, you know, all of this all of this interop,
255
00:22:57.090 --> 00:23:00.870
you know, needs to be seamless. Right? Otherwise, you're adding yet another
256
00:23:01.345 --> 00:23:12.470
layer of complexity for our users, and it you know, we don't wanna leave it to them to to deal with this. And so that's where X table comes in. We are, partnering with, 1 House and GCP and others.
257
00:23:12.850 --> 00:23:18.390
And our GSL Labs, base systems lab that's part of the the Azure data product team
258
00:23:18.770 --> 00:23:25.875
is extremely involved with this effort, and that's going to be the core to interrupt between the formats. Right? So in,
259
00:23:26.275 --> 00:23:27.415
iceberg, delta,
260
00:23:28.035 --> 00:23:36.730
and others, essentially, the metadata layer is separate. Like, it's different how they handle metadata. And so x table is used at runtime to essentially create the metadata
261
00:23:37.190 --> 00:23:50.720
layer for both formats, 1 to the other if it does not exist so that over time, they it'll be bidirectional access. You could read, write to both delta and 1 lake. Yeah. So exciting to see all of this open source. Honestly, it's
262
00:23:51.260 --> 00:23:51.760
very
263
00:23:52.620 --> 00:23:53.120
very,
264
00:23:53.660 --> 00:23:55.120
good to see
265
00:23:55.580 --> 00:23:56.080
both
266
00:23:56.525 --> 00:24:12.929
projects that I've been involved with to be leveraged, but also that I continue to get to be involved. And at the end of the day, customers benefit. Right? Customers win with when we use more open source. Of course, we have to be thoughtful and careful and make sure the enterprise grade security is added on top, etcetera.
267
00:24:13.455 --> 00:24:25.110
And going back to your point on layering, you know, with the the core open source is great. We need to, innovate on top of that, making sure that our customers are well protected as well. And in that stack
268
00:24:25.490 --> 00:24:36.405
of open source proprietary, particularly when you get into the security layering and making something enterprise accessible and enterprise ready, what are some of the most interesting or complicated
269
00:24:36.785 --> 00:24:39.605
engineering and integration challenges that you're running into?
270
00:24:40.210 --> 00:24:41.350
Yeah. You know,
271
00:24:41.650 --> 00:24:47.590
I'm I'm not involved with, every every aspect of fabric, of course. Right? But in terms of integration
272
00:24:48.365 --> 00:24:54.145
I mean, honestly, the the the first big jump to rewriting the engines was pretty transformational.
273
00:24:54.525 --> 00:25:02.700
Right? It, took some time to to get there and not just to support it, but actually get it performant, right, and optimize
274
00:25:03.080 --> 00:25:04.780
a stack with various things.
275
00:25:05.424 --> 00:25:11.205
We now have caching built in on top of 1 date, for example, to to optimize that as well.
276
00:25:11.505 --> 00:25:13.284
Security layering is,
277
00:25:13.664 --> 00:25:14.540
you know, is,
278
00:25:15.180 --> 00:25:17.360
and and making sure that it's
279
00:25:17.740 --> 00:25:21.120
every workload that we have in fabric because the the
280
00:25:21.715 --> 00:25:31.810
the scope and the the breadth is pretty wide. Right? It's it's it's it's pretty wide surface area. And so making sure that everything is is protected, there's consistency
281
00:25:32.430 --> 00:25:38.130
across each of these workloads from a security perspective. So VNet support, for example,
282
00:25:38.775 --> 00:25:52.419
and and we are building in 1 security on top, which continues to be you know, it's a journey. Right? You don't get it takes time. So it's not that you start off with 1 thing and and and, you know, you can get it done. We're adding 1 security as a layer on top where,
283
00:25:52.799 --> 00:25:57.684
on top of 1 link where it you you can you have extremely fine grained access control,
284
00:25:58.264 --> 00:25:59.684
for every workload.
285
00:26:00.225 --> 00:26:11.450
The way that it's you know, the way that it emerges, every workload currently, like SQL obviously is extremely mature and, you know, we have a fine grained access control within it. But to have this consistency and,
286
00:26:11.830 --> 00:26:14.090
security across all the workloads consistently,
287
00:26:14.665 --> 00:26:14.985
it's,
288
00:26:15.545 --> 00:26:19.725
it it it's taken time. So it it's been interesting to see, you know,
289
00:26:20.025 --> 00:26:20.765
that unification
290
00:26:21.385 --> 00:26:23.005
on principle. It is
291
00:26:23.840 --> 00:26:43.250
it it is really great for customers, but it is actually very hard to develop and get there. Right? The SaaS experiences as well. You know, it's it's simple to say, okay. No knobs, minimal knobs. Right? But what that means underneath the covers is you really need to ensure that you're optimizing for a range of use cases.
292
00:26:43.710 --> 00:26:45.710
Typically, you might say, oh, you know, with,
293
00:26:46.350 --> 00:26:59.790
straight out of the box open source, hey. Set this config to x and set that config to y and that to z. And, you know, for this use case, that's the best config. Well, if there are no knobs, you have to do everything underneath the covers. And so,
294
00:27:00.430 --> 00:27:04.690
that's the other area that is challenging is how do you, you know, auto optimize
295
00:27:04.990 --> 00:27:06.370
for a range of workloads
296
00:27:06.995 --> 00:27:12.935
as well as, use AI and, you know, ML techniques underneath the covers to actually make it happen.
297
00:27:13.635 --> 00:27:15.095
On that note of
298
00:27:15.480 --> 00:27:21.580
AI and embedding generative AI and other forms of AI into the data layer,
299
00:27:22.745 --> 00:27:28.925
I'm wondering if you can talk to some of the benefits that you see of having the Microsoft Copilot functionality
300
00:27:29.865 --> 00:27:30.925
as an integrated
301
00:27:31.410 --> 00:27:31.910
piece
302
00:27:32.370 --> 00:27:33.990
of the fabric product
303
00:27:34.370 --> 00:27:42.815
versus something that people can use out of band and then apply those changes. Just some of the ways that that changes the thinking around
304
00:27:43.514 --> 00:27:50.530
how the data platform is supposed to work, how people are supposed to interact with these data engineering, data analytics use cases,
305
00:27:51.230 --> 00:27:53.730
and also some of the potential risks
306
00:27:54.350 --> 00:27:58.930
and problems around reliability given the fact that generative AI is nondeterminative.
307
00:28:00.655 --> 00:28:02.655
Yes. Yeah. I mean, we're still all,
308
00:28:03.135 --> 00:28:05.775
you know, Microsoft is obviously leading that that,
309
00:28:06.815 --> 00:28:07.635
the the
310
00:28:08.009 --> 00:28:09.370
revolution or, you know, this,
311
00:28:09.850 --> 00:28:20.665
this era in some ways, and it's so great to be part of that. If you just if you if you leave data out for a minute and just talk about, you know, the promise of Jenny and what it, you know, what it allows us to do,
312
00:28:21.045 --> 00:28:23.225
productivity gain is a big 1. Right?
313
00:28:23.525 --> 00:28:25.945
Whether it's data engineering, whether it's,
314
00:28:26.700 --> 00:28:28.000
you know, a doctor
315
00:28:28.539 --> 00:28:30.780
typing in, their notes, whether it's a,
316
00:28:31.900 --> 00:28:34.220
a financial analyst trying to figure out,
317
00:28:35.020 --> 00:28:35.679
the market,
318
00:28:36.145 --> 00:28:48.260
productivity and, you know, reducing the amount of effort that anyone does is really 1 of the biggest promises of Gen AI. Right? That's why you call it a copilot. You're not it's not the pilot yet. It is a copilot, and,
319
00:28:48.980 --> 00:28:49.720
it assists,
320
00:28:50.595 --> 00:28:54.215
you know, humans in some ways and data engineers in our world,
321
00:28:54.595 --> 00:28:55.235
to do,
322
00:28:55.715 --> 00:28:56.215
more
323
00:28:56.515 --> 00:28:58.375
with less, to do things faster,
324
00:28:59.090 --> 00:29:05.110
to do things with less errors if you know, and in some ways and so that because you have more data at hand,
325
00:29:05.570 --> 00:29:08.230
because it's trained on a lot more, information.
326
00:29:08.695 --> 00:29:20.450
And so productivity, I think, is the number 1 thing that most enterprises are trying to begin with. Right? This means that it needs to be a part of the experience. Right? Because,
327
00:29:20.910 --> 00:29:35.320
it's a behavioral change otherwise where if it's not a part of that product and experience, you have to kind of switch, go somewhere else, try something else out, and then come back. Right? Versus if if if Copilot sort of, you know, in any AI experiences natively
328
00:29:35.779 --> 00:29:37.320
within the product experience,
329
00:29:37.620 --> 00:29:47.705
it becomes as it becomes seamless, becomes easier to use, and it becomes a habit. It needs to become a behave it's a it's a behavioral change. Right? So that's where Copilot
330
00:29:48.085 --> 00:29:50.904
is part of various other Microsoft products.
331
00:29:51.365 --> 00:29:53.225
GitHub for 1 has, you know,
332
00:29:53.669 --> 00:29:54.809
its own copilot.
333
00:29:55.350 --> 00:29:57.049
Versus Code has its copilot.
334
00:29:57.669 --> 00:30:01.909
Fabric has its copilot, and the this is now, generally available as well,
335
00:30:02.390 --> 00:30:04.215
but with our Power BI,
336
00:30:04.915 --> 00:30:06.055
Copilot experience.
337
00:30:06.515 --> 00:30:06.915
And,
338
00:30:07.395 --> 00:30:10.375
essentially, what it allows people to do is
339
00:30:10.755 --> 00:30:11.895
have a lot more
340
00:30:12.600 --> 00:30:17.900
a broader view of their data. You could ask about, you know, any dataset even if you don't have that information
341
00:30:18.360 --> 00:30:23.475
or maybe the tribal knowledge doesn't exist within your team. The copilot will know it. Right?
342
00:30:23.855 --> 00:30:29.955
And allow you to, you know, create complex dashboards, create, you know, ML to SQL essentially,
343
00:30:30.600 --> 00:30:32.539
Anything with, you know, natural language
344
00:30:32.840 --> 00:30:41.075
to the output format, right, is taken care of. And it we've we've seen that especially in you know, for nontechnical users,
345
00:30:41.375 --> 00:30:44.674
this becomes a a a big driver because, you know, they,
346
00:30:44.975 --> 00:30:48.179
they may not know SQL. They may not know how to create complex dashboards,
347
00:30:49.120 --> 00:31:00.375
but they now have the power to do a lot more with the data that they have access to. And so Copilot, you know, comes in very useful in that in that way. So every experience within fabric, whether it's,
348
00:31:00.675 --> 00:31:03.415
whether it's a warehouse experience, a data science experience,
349
00:31:03.799 --> 00:31:05.580
or BI, as I mentioned, etcetera,
350
00:31:06.039 --> 00:31:12.299
has a copilot and will have more, as we as we continue to innovate. Now your question around
351
00:31:12.845 --> 00:31:30.900
safety, reliability, and, you know, also, you know, doesn't have all the information. You might still have a lot of tribal information knowledge within the within the company that needs to be taught to the copilot in some ways. And so we have the notion of AI skills within fabric, which means that you can actually update the responses
352
00:31:31.585 --> 00:31:32.645
for certain prompts
353
00:31:33.105 --> 00:31:35.665
based on what you know as as,
354
00:31:36.065 --> 00:31:39.275
accurate. Right? A good 1 is, you know, we had our
355
00:31:40.190 --> 00:31:44.050
the project name for fabric in before it was announced was Trident.
356
00:31:44.430 --> 00:32:08.205
And there's a lot of information on Trident everywhere, and that correlation may or may not exist between what Trident is and what fabric is. And so prompt you know, updating prompt responses and making sure you get the you give correct 1 to the users is something that, you know, is is part of is is part of the product, so AI skills is a is a good 1. But in terms of, you know, you you might still have, you know,
357
00:32:08.825 --> 00:32:11.165
hallucinations are in some ways real. We,
358
00:32:11.545 --> 00:32:16.860
with with these LLMs, and so there are there's still a lot more work in general to, to do with,
359
00:32:17.340 --> 00:32:24.720
safety and reliability, but we take it you know, it's we we take it extremely seriously. A lot of these are you know, there's limits and
360
00:32:25.174 --> 00:32:33.195
throttling, etcetera, built in and, and hope, you know, over time, it'll keep getting better and better. Now we have multimodal responses, right, that are possible.
361
00:32:33.580 --> 00:32:36.240
So it's not just text. There there will be, you know, visuals,
362
00:32:36.700 --> 00:32:37.679
audio, etcetera.
363
00:32:38.380 --> 00:32:42.000
So with with every step, we have to build in that,
364
00:32:42.965 --> 00:32:43.625
the responsibility
365
00:32:44.325 --> 00:32:47.865
that, you know, we have built a responsible AI, to be part of it.
366
00:32:48.325 --> 00:32:50.025
On that note of modality
367
00:32:50.405 --> 00:32:53.090
and also on the subject of AI,
368
00:32:53.790 --> 00:32:59.330
that brings us back around to the idea of the data lake being a de facto and default
369
00:32:59.755 --> 00:33:01.135
storage interface where
370
00:33:01.595 --> 00:33:02.255
up until
371
00:33:02.555 --> 00:33:04.335
recent memory, the primary
372
00:33:04.635 --> 00:33:08.335
type of data that everybody was interacting with was textual or tabular.
373
00:33:09.140 --> 00:33:13.080
Now with AI and the idea of vector embeddings, multimodality,
374
00:33:13.460 --> 00:33:22.424
we're bringing in more of this unstructured and semi structured data, whether it's in the context of LLMs and reattrieval augmented generation or in the context
375
00:33:23.365 --> 00:33:23.945
of more
376
00:33:24.539 --> 00:33:28.480
detailed search functionality around being able to search for audio and images.
377
00:33:29.020 --> 00:33:29.760
And I'm wondering
378
00:33:30.220 --> 00:33:30.720
how
379
00:33:31.395 --> 00:33:32.695
that element of multimodality,
380
00:33:33.155 --> 00:33:35.335
AI, retrieval augmented generation
381
00:33:35.795 --> 00:33:43.640
is being factored into the ways that the fabric team is thinking about the role that they play in this ever expanding data ecosystem?
382
00:33:44.420 --> 00:33:47.935
Yeah. Yeah. It's a great question. And, you know, we're
383
00:33:48.275 --> 00:33:48.775
seeing,
384
00:33:49.355 --> 00:33:53.035
both on the analytics side, but also on the operational side,
385
00:33:53.835 --> 00:33:58.240
a lot of innovation happening. Right? So if you start off with the Gen AI applications,
386
00:33:59.260 --> 00:34:10.515
more the simple ones, right, the easiest way to get going is in in embed a a bot into the website, which does a lot of various variety of things. Right? Gives buying recommendations,
387
00:34:12.049 --> 00:34:13.109
customer support,
388
00:34:13.809 --> 00:34:17.329
handling a range of things. And for those, we have,
389
00:34:17.730 --> 00:34:19.349
you know, the vector index obviously,
390
00:34:19.809 --> 00:34:21.029
comes in very handy.
391
00:34:21.365 --> 00:34:27.925
Cosmos DB, which is extremely widely used and and is actually used for the the,
392
00:34:28.940 --> 00:34:31.599
used by chat gbt as 1 of the back end,
393
00:34:31.900 --> 00:34:33.119
databases offers,
394
00:34:34.059 --> 00:34:38.484
vector indexing. Our post offers vector indexing. And so there's,
395
00:34:38.964 --> 00:34:41.464
on all our databases essentially offer
396
00:34:41.765 --> 00:34:45.320
support for, a vector index so that that context is,
397
00:34:45.800 --> 00:34:52.040
is is stored, and you can build these very, very rich gen AI applications on top. Now on the other hand, you have,
398
00:34:52.565 --> 00:35:02.950
your lakes, right, as we were talking about. We've been talking for some time. That is, you obviously have structured data on top, but now you have there's a lot of semi structured data as well. You might have images,
399
00:35:03.670 --> 00:35:06.410
logos for companies that you wanna analyze, etcetera.
400
00:35:06.870 --> 00:35:07.355
And
401
00:35:07.675 --> 00:35:11.535
currently, there's, you know, there's a deep deep integration between
402
00:35:11.835 --> 00:35:16.015
fabric as well as, like, AI studio and our Azure AI, side
403
00:35:16.430 --> 00:35:22.530
services. Right? So that's 1. But over time and and vector search is part of our Azure AI
404
00:35:23.525 --> 00:35:26.105
experience. Right? So you can you can have vector search,
405
00:35:26.405 --> 00:35:27.225
that you can,
406
00:35:27.845 --> 00:35:29.305
support on top of the lake.
407
00:35:29.605 --> 00:35:30.105
But,
408
00:35:30.405 --> 00:35:39.680
yeah. I mean, stay tuned. I mean, these experiences are getting more and more integrated. And so, you know, today, you might have to use 2 or 3 services to to pull it together.
409
00:35:40.175 --> 00:35:40.675
Tomorrow,
410
00:35:40.975 --> 00:35:44.915
you it's all built in, and it's it's a seamless experience. So
411
00:35:45.215 --> 00:35:49.395
1 aspect that we're seeing is, it's it's also new that customers
412
00:35:49.970 --> 00:36:03.555
just want to have it available as easy as possible. Right? So that they're not stuck with doing all this integration work across these services together. And so while the services exist today with, whether it's vector indexes as part of our databases,
413
00:36:04.015 --> 00:36:06.640
vector search, which is part of Azure
414
00:36:07.020 --> 00:36:11.900
AI, on top of, 1 Lake and so on. It will be even more seamless over time as,
415
00:36:12.780 --> 00:36:14.720
as more, you know, as more satisfaction,
416
00:36:15.775 --> 00:36:18.194
in some ways happens on the Jenny I stack.
417
00:36:18.655 --> 00:36:20.835
And then bringing it around to
418
00:36:21.135 --> 00:36:23.075
your experience in this space,
419
00:36:23.760 --> 00:36:31.125
As we've noted, you previously worked at Ahana, which was very focused on this lake house paradigm. You have a lot of context
420
00:36:31.445 --> 00:36:33.465
in data lakes in general. I'm wondering
421
00:36:34.485 --> 00:36:39.385
what are the ways that your prior experience at Ahana working in the Presto ecosystem,
422
00:36:41.050 --> 00:36:43.790
leveraging, cloud storage, data lakehouse architectures
423
00:36:44.250 --> 00:36:51.525
has informed the work that you're doing at Fabric and some of the ways that it's a continuity and some of the ways that it is a new experience.
424
00:36:52.305 --> 00:36:52.625
Yeah.
425
00:36:53.505 --> 00:36:55.980
It's it's been, yeah, it's been quite some
426
00:36:56.540 --> 00:37:00.080
time in the lakehouse space. Founding a company on lakehouses itself
427
00:37:00.619 --> 00:37:04.720
shows how how how bought in into the tech I was.
428
00:37:05.305 --> 00:37:05.945
And so,
429
00:37:06.265 --> 00:37:09.484
you know, there's a few different aspects. Right? I I get to,
430
00:37:09.945 --> 00:37:13.565
influence and help the teams on an open source front for sure.
431
00:37:13.980 --> 00:37:15.020
Still actively participating,
432
00:37:16.860 --> 00:37:22.800
with, through a lot of the projects that we talked about. But the other aspect is actually just the entrepreneurial,
433
00:37:23.205 --> 00:37:25.945
you know, nature and, attributes that
434
00:37:26.325 --> 00:37:31.465
are core to what I have been doing, right, in the past. And then in some ways, bringing in new ideas,
435
00:37:32.420 --> 00:37:33.300
taking taking
436
00:37:34.260 --> 00:37:38.360
starting new things, like, you know, 0 to ones are always hard,
437
00:37:38.900 --> 00:37:43.555
whether it's as a startup or whether you're doing it within a large corporate
438
00:37:44.575 --> 00:37:49.700
company with with with deep pockets. Right? Everyone resources are scarce everywhere,
439
00:37:50.320 --> 00:37:56.075
in some ways. You always wanna do a 100 things and you can do 5. And so, in some ways, that entrepreneurial,
440
00:37:56.935 --> 00:37:59.755
characteristic has helped me drive new areas.
441
00:38:00.135 --> 00:38:00.954
For example,
442
00:38:01.494 --> 00:38:02.234
with fabric,
443
00:38:02.640 --> 00:38:09.700
we are now opening up the platform. So we obviously have our 6 workloads that are core and, you know, that are Microsoft
444
00:38:10.215 --> 00:38:13.675
workloads in some ways, Power BI and others within fabric. Right?
445
00:38:14.055 --> 00:38:19.755
But there are a lot of areas there which are, you know, might be narrower gaps that we may not invest in.
446
00:38:20.130 --> 00:38:27.510
And it it it makes sense for our partners to come in and build on top of fabric and integrate into it natively,
447
00:38:28.055 --> 00:38:37.140
allowing them to have these experiences as well. And so we have opened up the the platform. So not only is it open data, you know, it's open APIs, but the platform itself
448
00:38:39.140 --> 00:38:44.599
platform development kit that I announced, at Microsoft Build, which was our big developer conference
449
00:38:44.980 --> 00:38:46.119
a few weeks ago
450
00:38:46.515 --> 00:38:54.695
where any ISV, any startup can come in and build a native experience on top of fabric. Right? And, and so in some ways, fabric
451
00:38:55.100 --> 00:38:59.040
is, as an OS, you know, in some ways, a data operating system.
452
00:38:59.580 --> 00:39:09.335
Microsoft has its set of apps. Right? And now we are, we are integrating in ISPs to come in and pull in their own native apps in terms in into fabric,
453
00:39:09.799 --> 00:39:11.980
and really make it a strong ecosystem
454
00:39:12.599 --> 00:39:16.059
overall. At the end of the day, customers win. They have a lot more experiences,
455
00:39:16.760 --> 00:39:18.380
capabilities to choose from.
456
00:39:18.775 --> 00:39:21.975
And so a couple of examples that we announced were,
457
00:39:22.455 --> 00:39:23.355
you know, SAS,
458
00:39:23.655 --> 00:39:24.155
Informatica,
459
00:39:24.935 --> 00:39:26.075
ASG for geospatial,
460
00:39:26.859 --> 00:39:27.680
Neo 4J,
461
00:39:28.220 --> 00:39:29.119
for graph,
462
00:39:29.500 --> 00:39:33.119
Teradata. Yeah. We have a warehouse, but, you know, Teradata engine
463
00:39:33.500 --> 00:39:40.565
on 1 link, and, also, prophecy support, you know, master data management. So this is just the beginning. Right? We will have more and more,
464
00:39:41.185 --> 00:39:41.685
ISVs,
465
00:39:42.225 --> 00:39:45.450
get pulled in. And then the other areas where,
466
00:39:45.829 --> 00:39:50.890
I've been able to sort of get more 0 to 1 new experiences, built new experiences on fabric
467
00:39:51.315 --> 00:39:54.055
is bring technologies like GraphQL, which is
468
00:39:54.355 --> 00:39:58.055
essentially an API on top of data data, services
469
00:39:58.690 --> 00:40:00.869
into fabric. So we added GraphQL support.
470
00:40:01.410 --> 00:40:10.345
We added user data functions natively within fabric. So these are all kind of 0 to 1 experiences that have it's been great to work with the teams
471
00:40:10.805 --> 00:40:13.545
to drive and kind of deliver out. So,
472
00:40:13.845 --> 00:40:17.285
they're now publicly announced, so I can finally talk about it. But,
473
00:40:17.605 --> 00:40:19.990
yeah, just, you know, being involved in new
474
00:40:20.470 --> 00:40:21.849
completely new areas,
475
00:40:22.390 --> 00:40:23.930
driving the ecosystem.
476
00:40:24.309 --> 00:40:25.905
Alright? Because, you know,
477
00:40:26.525 --> 00:40:27.665
fairly, connected
478
00:40:28.045 --> 00:40:40.090
with the the data space and then the products and the leaders within it. And then I would I would still say, you know, having that customer first mindset, you know, as a founder, you really still you know, you you're you're constantly thinking about,
479
00:40:40.470 --> 00:40:43.755
the customer base. You talk to hundreds of customers and,
480
00:40:44.555 --> 00:40:46.734
what prospects, I would say, you know, they're,
481
00:40:47.115 --> 00:40:53.214
with the, with the hope of being customers. But here, I can, you know, I can leverage a lot of that background,
482
00:40:54.340 --> 00:40:55.880
and, the the rich
483
00:40:56.180 --> 00:40:57.320
solution architectures,
484
00:40:58.100 --> 00:41:10.605
that we've, that I've been, you know, exposed to in the past. Now with fabric and really kind of innovate at a massive scale. Right? So it's been it's been really fun, whether it's health care, whether it's financial services,
485
00:41:11.930 --> 00:41:14.430
telco, lot of adoption that's happening,
486
00:41:14.890 --> 00:41:15.289
with,
487
00:41:15.690 --> 00:41:16.509
very large
488
00:41:16.809 --> 00:41:18.829
ISVs as well as enterprises.
489
00:41:19.245 --> 00:41:21.265
And so it's been great to influence,
490
00:41:21.965 --> 00:41:23.025
all of these areas.
491
00:41:23.725 --> 00:41:32.890
And in your experience of working in the space, working on fabric, what are some of the most interesting or innovative or unexpected ways that you've seen that platform applied?
492
00:41:33.190 --> 00:41:36.170
Yeah. It's it's interesting. I mean, there's always great stories.
493
00:41:36.815 --> 00:41:37.455
The most,
494
00:41:38.015 --> 00:41:44.515
recent 1, you know, I I like, I like speed. I like driving fast, and I had follow Formula 1 and all this.
495
00:41:45.109 --> 00:41:50.410
And there was a McLaren is 1 of, you know, that we highlighted, 1 of the users and,
496
00:41:51.430 --> 00:41:55.495
that is using our real time intelligence stack we recently announced,
497
00:41:55.935 --> 00:41:58.255
to store a lot of that their data. Right?
498
00:41:58.815 --> 00:42:03.234
And to be able to because, obviously, very high speeds of in you know, ingestion needed,
499
00:42:03.600 --> 00:42:04.760
the ability to,
500
00:42:05.200 --> 00:42:06.800
analyze this at great speed.
501
00:42:07.200 --> 00:42:17.435
So that's a that's a really good 1 and, you know, personal favorite for me as well. But I would say it's been it's been interesting to see, how creative our customers are getting,
502
00:42:17.895 --> 00:42:19.275
with with using fabric
503
00:42:19.660 --> 00:42:25.520
in, you know, where it's it's across verticals. It's really a horizontal platform, so,
504
00:42:25.980 --> 00:42:26.480
helping,
505
00:42:26.860 --> 00:42:27.360
with
506
00:42:27.885 --> 00:42:28.385
medical
507
00:42:29.005 --> 00:42:31.745
researchers, given electronic medical records, for example,
508
00:42:32.525 --> 00:42:33.425
getting in
509
00:42:33.725 --> 00:42:34.225
financial
510
00:42:34.605 --> 00:42:35.105
data,
511
00:42:35.805 --> 00:42:39.859
into so benchmark data, composite data, things like
512
00:42:40.160 --> 00:42:42.420
that, through different various different sources
513
00:42:42.799 --> 00:42:52.405
into into 1 lake so that they have a single view of that data. You know, that's another. And so I think the multi cloud aspect of the shortcutting aspect of, 1 lake,
514
00:42:52.945 --> 00:42:54.085
has truly been
515
00:42:54.900 --> 00:42:58.839
why it it it's something that a lot of customers are very interested in,
516
00:42:59.299 --> 00:43:03.880
because it you know, we've had virtualization for a long period of time. It's not new. That virtualization
517
00:43:04.785 --> 00:43:18.790
concept itself isn't new, but the ease of which we enable it and the fact that we, you know, write everything into, by default into into, Delta Lake, and you immediately can then access this data through various other workloads,
518
00:43:19.250 --> 00:43:21.510
that part of it is quite revolutionary.
519
00:43:21.890 --> 00:43:26.645
So I think that is going to drive a little a lot more data gravity into 1 week.
520
00:43:27.425 --> 00:43:32.630
And, yeah, you know, it's it'll be interesting to see more and more, use cases,
521
00:43:33.090 --> 00:43:33.830
on top.
522
00:43:34.610 --> 00:43:35.590
In your experience
523
00:43:35.890 --> 00:43:39.955
of working in this space, transitioning into the fabric team,
524
00:43:40.255 --> 00:43:40.755
understanding
525
00:43:41.055 --> 00:43:47.395
the use cases, the customer base? What are some of the most interesting or unexpected or challenging lessons that you've learned in the process?
526
00:43:48.110 --> 00:43:49.170
Yeah. I I think,
527
00:43:50.190 --> 00:43:55.230
a lot of lot of good lessons learned. In some ways, I see you know, as we've evolved over the last,
528
00:43:55.550 --> 00:43:57.330
5 years in the data lakes,
529
00:43:57.785 --> 00:44:06.765
it's good to see people taking the the the mistakes made in the past and using you know, fixing them in some ways and and leveraging them. In terms of
530
00:44:07.530 --> 00:44:12.510
lessons learned, I would say, you know, that last mile of making it enterprise grade,
531
00:44:13.610 --> 00:44:23.445
where, you know, we we have many, many fundamentals that we need to get to for GA, whether it's it's a a security is about big 1, scale testing, etcetera.
532
00:44:24.020 --> 00:44:27.640
That continues to be hard and, you know, takes time.
533
00:44:28.020 --> 00:44:31.160
So there's no, there's no free way to get there faster.
534
00:44:31.535 --> 00:44:33.635
You still have to go through that process
535
00:44:34.255 --> 00:44:34.755
and,
536
00:44:35.135 --> 00:44:48.980
in building a good enterprise software. Right? There's, there's no way out for it. And, and, I mean, honestly, it's and it it's very important. Right? The mission critical applications that we're involved with, it's absolutely critical we go through that process,
537
00:44:49.535 --> 00:44:54.755
and make sure that we've not only the capabilities are there, but it's, reliable. It's stable.
538
00:44:55.135 --> 00:44:57.955
The high availability is built in. Right? So many
539
00:44:58.460 --> 00:44:59.520
aspects of the foundations,
540
00:44:59.900 --> 00:45:11.935
that you just expect out of a Microsoft service. It's it takes time, and it's, it's still it's still hard. Right? Because everything is so tightly integrated, you have to make sure that the solution architecture is right.
541
00:45:12.655 --> 00:45:17.280
Internally, the design is correct. There's no holes. So that process continues
542
00:45:17.660 --> 00:45:24.815
to, be hard and, you know, I I always feel like when you build the next set of software, some people get easier. At least with that,
543
00:45:25.215 --> 00:45:26.915
it it it still takes time.
544
00:45:27.455 --> 00:45:28.355
Let me think.
545
00:45:28.895 --> 00:45:29.855
In terms of,
546
00:45:30.415 --> 00:45:32.675
you know, adoption, whenever there's a new
547
00:45:33.060 --> 00:45:44.415
new product, and I've gone through this process many times in the past, the adoption itself, you know, it it takes some time to educate. Education is a big part of the learning experience for customers.
548
00:45:45.195 --> 00:45:50.240
And, of course, with Microsoft, it you know, it's it's a extremely well known brand. There's a lot of,
549
00:45:50.800 --> 00:45:51.300
expectations
550
00:45:51.600 --> 00:45:58.865
out of that brand as well. And so, you know, the the adoption phase, it's been phenomenal to see how fast fabric has,
551
00:45:59.805 --> 00:46:01.345
has grown and been adopted.
552
00:46:01.645 --> 00:46:05.185
But even then, right, it's it is a process. Adoption takes time.
553
00:46:05.520 --> 00:46:10.099
Education, community building, all of these things remain critical in terms of
554
00:46:10.400 --> 00:46:12.180
getting customers, end users,
555
00:46:12.825 --> 00:46:18.045
comfortable with the tech. Training is a big part of it. So we're doing putting a lot of these things in place,
556
00:46:18.425 --> 00:46:25.010
to make sure that at the end of the day, it's a new product. And so we need to make sure that our customers have all the tools available
557
00:46:25.390 --> 00:46:27.010
to make them successful. Otherwise,
558
00:46:27.630 --> 00:46:29.330
it it, you know, it it,
559
00:46:29.845 --> 00:46:32.505
in in some cases, it may not may may not,
560
00:46:33.205 --> 00:46:35.785
leave a good taste in their, in their mouths. Yeah.
561
00:46:36.405 --> 00:46:38.105
As you continue to
562
00:46:38.560 --> 00:46:41.700
invest in fabric, continue to explore the ecosystem
563
00:46:42.400 --> 00:46:52.515
of data lakes, data lake houses, and see the ways that people are applying them, what are some of the projections that you have for the future of data lake analytics, data lake house architectures,
564
00:46:52.895 --> 00:46:57.310
the role of AI in the data engineering and data analytics ecosystem.
565
00:46:58.170 --> 00:47:00.590
Yeah. It's it's gonna be a fun next 5 years,
566
00:47:01.290 --> 00:47:03.130
Tobias. It'll be interesting to see,
567
00:47:03.845 --> 00:47:06.905
you know, how it all kind of continues to come together.
568
00:47:07.445 --> 00:47:10.585
I'm seeing a lot of consolidation. Right? Essentially, we've seen consolidation
569
00:47:11.525 --> 00:47:14.789
now first in the analytics stack, right, where, with fabric,
570
00:47:15.569 --> 00:47:25.115
if a lot of these pieces are sort of consolidated in, Gen AI is is part of it. Yeah. Right? It's being consolidated in, where we'll see increasingly even applications
571
00:47:25.415 --> 00:47:44.435
have obviously their their operational apps or RTP apps. There's a real time interface to it. There's an analytics interface and a Gen AI interface to all of these apps, which means that the platforms behind it will also the the more consolidated they are, the easier it will be for customers to to build not only the back end,
572
00:47:44.815 --> 00:47:51.470
analytics up, services, but also the front end customer facing services. Right? So I do see,
573
00:47:51.770 --> 00:47:52.510
in general,
574
00:47:52.810 --> 00:47:54.190
more consolidation happening
575
00:47:54.490 --> 00:47:58.205
in in the space. Now Gen AI itself will evolve into,
576
00:47:58.745 --> 00:48:02.925
becoming a lot more robust. I mean, it's been, you know, mind blowing already,
577
00:48:03.290 --> 00:48:06.349
but you can only imagine over the next 5 years as,
578
00:48:06.890 --> 00:48:11.150
more GPUs are available, as more training happens, as more datasets,
579
00:48:12.355 --> 00:48:26.850
as as more, you know, custom models start evolving on proprietary datasets. There's still a lot of proprietary data that's, that cost enterprise companies have. Right? And so as as these evolve, there will be, interesting co pilots that emerge
580
00:48:27.255 --> 00:48:30.234
over time that will also get, you know, built in.
581
00:48:30.615 --> 00:48:31.994
So Jenny I itself,
582
00:48:32.535 --> 00:48:34.454
will emerge. And and so,
583
00:48:34.775 --> 00:48:43.680
hope the hope is at the end of the day that, you know, our job as, you know, the way I've seen it is as as people building these platforms,
584
00:48:44.665 --> 00:48:47.244
we need to keep this promise of simplification
585
00:48:47.785 --> 00:48:48.185
and,
586
00:48:48.825 --> 00:48:49.325
unification
587
00:48:49.865 --> 00:48:52.125
and, and and help customers
588
00:48:52.744 --> 00:48:53.565
sort of
589
00:48:53.980 --> 00:48:55.840
reduce confusion in some ways,
590
00:48:56.220 --> 00:48:57.120
and help customers,
591
00:48:57.580 --> 00:48:58.960
you know, be successful
592
00:48:59.260 --> 00:48:59.760
because
593
00:49:00.060 --> 00:49:00.960
as we've seen,
594
00:49:01.340 --> 00:49:02.080
many times,
595
00:49:02.725 --> 00:49:04.425
you know, people adopt technologies,
596
00:49:04.885 --> 00:49:05.205
and,
597
00:49:05.685 --> 00:49:10.825
they they may not see the benefits of it. So we really want with this with this phase
598
00:49:11.210 --> 00:49:13.609
of innovation as every everyone invests in,
599
00:49:14.250 --> 00:49:15.869
Gen AI and open data.
600
00:49:16.330 --> 00:49:19.164
We really want them to get the most out of it. Right?
601
00:49:19.724 --> 00:49:21.825
So anything that will speed that process,
602
00:49:22.125 --> 00:49:25.585
I think will over time emerge into our products, and,
603
00:49:26.444 --> 00:49:29.670
and it's great to be part of it, part of the journey.
604
00:49:30.450 --> 00:49:40.855
Are there any other aspects of Microsoft Fabric, data lakes and lake houses, your experience working in this ecosystem that we didn't discuss yet that you'd like to cover before we close out the show?
605
00:49:41.155 --> 00:49:43.755
I think we've talked about a lot of different aspects.
606
00:49:44.195 --> 00:49:44.695
So,
607
00:49:44.995 --> 00:49:45.540
you you
608
00:49:46.740 --> 00:49:53.720
Tobay. So thank you so much for all the the interesting questions and and, you know, the in some ways, the the ordering that that
609
00:49:54.165 --> 00:49:57.365
you had as well. You know, honestly, I think that we're still,
610
00:49:58.165 --> 00:50:01.340
it's a great platform, but we're, you know, we're still starting and,
611
00:50:01.820 --> 00:50:03.760
and, looking forward to,
612
00:50:04.380 --> 00:50:06.220
seeing where, you know, where it goes and,
613
00:50:06.780 --> 00:50:08.160
hopefully, in this process,
614
00:50:08.895 --> 00:50:09.935
make a lot of customers,
615
00:50:10.335 --> 00:50:13.395
successful and get them to innovate on top of fabric.
616
00:50:14.015 --> 00:50:30.165
Alright. Well, for anybody who wants to get in touch with you and follow along with the work that you and your team are doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gap in the tooling or technology that's available for data management today.
617
00:50:30.625 --> 00:50:37.140
Yeah. Good question. Yeah. Absolutely. Everyone's free to, you know, reach out to me, LinkedIn, various other channels.
618
00:50:38.240 --> 00:50:45.540
Gaps in, I think there are there is a lot more technology now. I think the the gap is probably in adopting that technology
619
00:50:45.840 --> 00:50:46.340
and
620
00:50:46.705 --> 00:50:47.525
really understanding
621
00:50:47.985 --> 00:50:48.885
that tech tech,
622
00:50:49.265 --> 00:50:50.325
embracing it.
623
00:50:51.185 --> 00:51:03.130
We still see so many enterprises or, you know, that are maybe behind the curve. Right? And, I think it's really time now where data lakes are mainstream. So if if you're not
624
00:51:03.589 --> 00:51:04.089
using
625
00:51:05.305 --> 00:51:12.925
this stack, the you know, any in any form, like, some form, then you are falling behind. Right? And so it it really is,
626
00:51:13.600 --> 00:51:20.340
that there's a lot of great tech that's available today, whether it's clouds, whether it's, you know, other other companies. And
627
00:51:20.954 --> 00:51:29.295
and so, you know, as as CIOs, as CDOs, it's really important to, you know, embrace and try in some in some cases,
628
00:51:29.950 --> 00:51:34.690
you know, maybe stop investing in other old technologies and and look forward
629
00:51:35.310 --> 00:51:36.590
and innovate because,
630
00:51:37.015 --> 00:51:39.914
in some cases, it's, existential now. Right?
631
00:51:40.375 --> 00:51:44.635
There will be companies that, out innovate others with with JNII,
632
00:51:45.640 --> 00:51:47.340
whether it's LLMs, SLMs,
633
00:51:47.640 --> 00:51:49.260
you know, whatever that might be.
634
00:51:49.720 --> 00:51:57.045
It is really important to have that stack. Right? So, you know, open data with gen AI on top is that next gen stack.
635
00:51:57.505 --> 00:52:01.820
It's proven. It's extremely proven now, and so it's it's time to start,
636
00:52:02.380 --> 00:52:03.200
getting there,
637
00:52:03.660 --> 00:52:05.599
soon if you haven't done so already.
638
00:52:06.300 --> 00:52:12.115
Well, thank you very much for taking the time today to join me, share your experience working in this ecosystem,
639
00:52:12.415 --> 00:52:16.675
your insight into the Microsoft Fabric product. It's definitely a very
640
00:52:17.135 --> 00:52:21.859
interesting platform. It's great to see more investment in open data lake infrastructure,
641
00:52:22.319 --> 00:52:24.900
the entree into the enterprise.
642
00:52:25.200 --> 00:52:29.515
So appreciate the time and energy you're putting into this space, and I hope you enjoy the rest of your day.
643
00:52:29.994 --> 00:52:32.414
Absolutely. Thank you so much for having me again.
644
00:52:32.954 --> 00:52:35.214
Take care. Thanks, everyone. Bye bye.
645
00:52:41.510 --> 00:52:51.775
Thank you for listening. Don't forget to check out our other shows, podcast dot in it, which covers the Python language, its community, and the innovative ways it is being used, and the Machine Learning Podcast,
646
00:52:52.155 --> 00:52:54.910
which helps you go from idea to production with machine learning.
647
00:52:55.390 --> 00:52:56.690
Visit the site at dataengineeringpodcast.com.
648
00:52:58.190 --> 00:53:04.115
Subscribe to the show, sign up for the mailing list, and read the show notes. And if you've learned something or tried out a project
649
00:53:06.035 --> 00:53:08.215
from the show, then tell us about it. Email hosts at data engineering podcast.com
650
00:53:08.755 --> 00:53:09.575
with your story.
651
00:53:09.875 --> 00:53:14.990
And to help other people find the show, please leave a review on Apple Podcasts, and tell your friends and coworkers.