Confidently Incorrect — with Caitlin Colgrove

WEBVTT
00:00:02.640 --> 00:00:04.080
Welcome to the Test Set.
00:00:04.080 --> 00:00:10.720
Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning.
00:00:10.720 --> 00:00:16.745
Dig into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.
00:00:17.905 --> 00:00:22.304
Michael's traveling at the moment, so you're stuck with the producer's voice over this week.
00:00:22.304 --> 00:00:33.920
On this episode, the Test Set crew jams with Caitlin Colgrove, CTO and co founder of Hex, the data platform that somehow counts a Sweetgreen head chef among its power users.
00:00:33.920 --> 00:00:39.640
Caitlin joins Michael, Hadley, and Isabelle to make the case that data teams aren't going anywhere.
00:00:39.664 --> 00:00:44.945
But right now, their job is changing from writing SQL to vouching for the answers.
00:00:44.945 --> 00:00:46.744
We hope you enjoy.
00:00:49.585 --> 00:00:51.905
Caitlin, welcome to the Test Set.
00:00:51.905 --> 00:00:52.465
Thank you.
00:00:52.465 --> 00:00:54.090
Thanks so much for coming on.
00:00:54.090 --> 00:01:00.490
Just by way of introduction, you're Caitlin Colgrove, the CTO of Hex.
00:01:00.490 --> 00:01:04.730
And I'm Michael Chow. I'm the host of the Test Set.
00:01:04.730 --> 00:01:13.334
And I'm joined by Hadley Wickham, chief scientist at Posit, and Isabelle, software engineer extraordinaire at Posit.
00:01:13.454 --> 00:01:15.694
And we're so excited to have you on.
00:01:15.694 --> 00:01:23.019
I feel like Hex is everywhere today, and I'm constantly trying to explain to people what Hex is and why they should use it.
00:01:23.019 --> 00:01:28.579
I know Hex is included in Claude.ai now.
00:01:28.780 --> 00:01:34.379
Maybe do you mind just explaining a little bit about what Hex is and what it does?
00:01:34.379 --> 00:01:35.819
Yeah. Totally.
00:01:35.819 --> 00:01:46.105
Hex started when we when we started as as founders, we were basically we'd all left Palantir and we were kind of out in the world looking at how people were working with data.
00:01:46.385 --> 00:01:52.950
And what we discovered was that there are a lot of new patterns for working with data that were emerging.
00:01:52.950 --> 00:01:54.150
People were a lot more technical.
00:01:54.150 --> 00:01:59.629
There was a lot less clicking of buttons and a lot more writing of code, SQL, Python, all of that.
00:01:59.909 --> 00:02:07.754
And the tools out there were Jupyter Notebooks, those are very popular, this is twenty nineteen or so.
00:02:07.754 --> 00:02:24.419
But in order to actually sort of string together a full enterprise data workflow, like to actually do something useful with your data, people would be doing these ad hoc analyses in Python, SQL, do SQL in a notebook, still had to do the backticks, you know, it was like really not native.
00:02:24.419 --> 00:02:30.104
And then, you know, that was in a notebook, and it was a locally running notebook with your own Python environment.
00:02:30.104 --> 00:02:33.384
And, okay, so maybe you do this whole analysis and, like, what are you what are you gonna do?
00:02:33.384 --> 00:02:36.025
Are you gonna email that notebook to the CFO?
00:02:36.025 --> 00:02:37.625
Like, that that, you can't do that.
00:02:37.625 --> 00:02:39.944
So they would take these notebooks and they would okay.
00:02:39.944 --> 00:02:40.504
They're like, okay.
00:02:40.504 --> 00:02:54.590
We're gonna turn it into a Python script, and then we're gonna run the Python script on a schedule on AWS, and then we're gonna dump the output into Looker so that the CFO could look at it in Looker so that they could take a screenshot of the chart and put it in the board deck.
00:02:54.590 --> 00:02:56.349
And that was the workflow.
00:02:56.349 --> 00:03:05.365
And so Hex was really born of this idea that there just has to be a better way, that there needs to be first class tooling for this new era of of data work.
00:03:05.365 --> 00:03:07.125
And that was the original product that we built.
00:03:07.125 --> 00:03:10.884
And then obviously like, you know, in the past couple years AI has become a huge part of that.
00:03:10.884 --> 00:03:15.409
And I think even though some of the technology has changed, the underlying story is still true.
00:03:15.409 --> 00:03:18.770
People who are working with day in and day out, what are their workflows?
00:03:18.770 --> 00:03:21.969
What are the best tools for them? And Hex should be that.
00:03:21.969 --> 00:03:27.064
I've seen so many engineers in Python use like pandas read SQL, that Yeah.
00:03:27.064 --> 00:03:31.865
Classic like pandas.read_sql with a string of SQL in it Yeah.
00:03:31.865 --> 00:03:34.824
Which feels like maybe a little bit unnatural.
00:03:34.824 --> 00:03:36.344
And then More than a little bit unnatural.
00:03:36.344 --> 00:03:36.585
Yeah.
00:03:36.585 --> 00:03:37.664
Yeah. Yeah.
00:03:37.669 --> 00:03:43.789
In that scheduling dynamic, just all these bits of duct tape kind of like in the process.
00:03:44.710 --> 00:03:45.590
I'm really curious.
00:03:45.590 --> 00:03:53.735
I know, like, with the Anthropic integration, Like, what what does that look like for a person to use Hex?
00:03:53.735 --> 00:03:54.055
Yeah.
00:03:54.055 --> 00:03:58.414
So I think this is one of the big ways that work is changing.
00:03:58.854 --> 00:04:19.834
Pre AI, you basically had these big heavyweight vertical SaaS tools, and you depending on what you wanted to do, you would navigate to HubSpot, or you would navigate to Salesforce, or you would navigate to Hex because you're like, I wanna do something with my marketing data, I wanna do data science, data analysis, and that would be where you would start.
00:04:19.834 --> 00:04:24.274
And so you'd have these big full featured front doors to these vertical applications.
00:04:24.475 --> 00:04:28.995
Today everybody just starts by like asking Claude or ChatGPT a question.
00:04:29.889 --> 00:04:41.009
And and we've actually had some really, really interesting conversations with the Anthropic team and the and the OpenAI teams about how they're seeing the, like, the changing and and sort of the emergence of these of these new workflows.
00:04:41.009 --> 00:04:45.305
And and they really view a lot of these chat applications as a front door.
00:04:45.305 --> 00:04:52.985
They're not gonna go and build, you know, a first class data science agent or a first class, you know, customer service agent or all of these things.
00:04:52.985 --> 00:04:55.464
They wanna be the platforms on which those things run.
00:04:55.464 --> 00:05:00.720
But still today, like, it's very sticky, these habits of just ask your question to Claude.
00:05:00.720 --> 00:05:06.959
And so what Anthropic released, and ChatGPT has sort of its own version as well, is this plugin architecture.
00:05:06.959 --> 00:05:13.120
And it's based on MCP, as you might imagine, but they've enhanced that with, you know, sort of more UX around it.
00:05:13.120 --> 00:05:21.245
So now if you hook up your MCP to Claude, you actually get a interactive Hex experience in Claude.
00:05:21.245 --> 00:05:25.165
And Claude, you know, the models are actually smart enough to know how to route.
00:05:25.165 --> 00:05:35.750
So if you go and you have Hex wired up as your sort of data science back end, and you ask a question, Claude knows that the Hex plugin is the plugin to use to answer that.
00:05:35.750 --> 00:05:45.144
And so it'll actually go and it'll go kick off sort of a sub agent architecture where it kicks off an agent in Hex, which will go look at all your data, it'll go look at all your semantic models, it'll do all this stuff, and it'll come back.
00:05:45.144 --> 00:05:46.345
And it's actually really nice.
00:05:46.345 --> 00:05:50.584
You get to, like, sort of see the thinking tokens, you can get an interactive set of charts back.
00:05:50.584 --> 00:05:59.504
And then if you if you really wanna go deep or you wanna go build a dashboard or you wanna do something further, you can kind of click through into the underlying Hex project that it's been building up behind the scenes.
00:05:59.670 --> 00:06:03.829
What's a question I could ask that might produce like a Hex widget?
00:06:03.829 --> 00:06:13.670
Like is it like if I wanted to check the weather in different cities, or like, let's say we just had yeah, like Like what like what's something you have used, Hex, to answer about like, Hex?
00:06:13.670 --> 00:06:21.764
Well, today so so Hex is typically most most useful for the types of enterprise data workflows where it's all governed in the data warehouse.
00:06:21.764 --> 00:06:23.285
You've probably done some modeling on top of it.
00:06:23.285 --> 00:06:27.045
You probably have some dashboards, you know, maybe have your KPI dashboards or whatever.
00:06:27.045 --> 00:06:33.060
Those it's like sort of classic data workflows, and then you can start to pull in other sources obviously via MCP and other things.
00:06:33.060 --> 00:06:40.259
But it's very similar to, you know, like pre AI, the types of things that you'd be managing with dbt and Snowflake and all those things.
00:06:40.259 --> 00:06:42.180
And so it's usually business data.
00:06:42.180 --> 00:06:47.514
And if you're syncing other sources, those are those are to enhance the questions that you're asking about the business data.
00:06:47.514 --> 00:06:54.634
So these days I'm asking a lot of questions about how much we're spending on Anthropic.
00:06:54.634 --> 00:06:58.740
Where are those tokens coming How are people using these these products?
00:06:58.740 --> 00:06:58.899
Products?
00:06:58.899 --> 00:07:05.060
And so that's a very it's a sort of com I would call it combination kind of finance, product analytics type workflows.
00:07:05.060 --> 00:07:18.685
And so basically what we're doing is we're piping all of our logging around the usage of Hex's AI into into Snowflake and all of those, like, fine grained token token counting and, you know, attribution and where those are coming from.
00:07:18.685 --> 00:07:25.285
And so I'm spending a lot of time looking at, you know, where are we spending the most tokens, are there places where we could be being more efficient.
00:07:25.610 --> 00:07:34.410
We also have a lot of data around, you know, our internal evals and performance, and so are there areas where we, like, should be using Opus and we can't actually drop down to something like Sonnet or Haiku.
00:07:34.410 --> 00:07:37.545
So I spent a lot of time looking at that in Hex.
00:07:37.545 --> 00:07:40.745
We know that, like, models can be, like, pretty sycophantic.
00:07:40.745 --> 00:07:55.800
Like, how much do you worry about, like, when you come in with a question and there's, like, an sort of an answer that, like, you wanna hear, and then, like, the you know, do you worry about you kind of accidentally steering the model in that direction that just reinforces, like, what you wanna hear?
00:07:55.800 --> 00:07:56.279
Yeah.
00:07:56.279 --> 00:08:07.055
I think I think that's actually one of the bigger prob like, model behavior is one of the bigger problems in sort of agentic AI for data science today.
00:08:07.055 --> 00:08:08.735
And it's a it's a combination of that.
00:08:08.735 --> 00:08:17.134
It's it's I think it's less of the sycophancy side of things that we see, but we see a lot of confidently incorrect.
00:08:17.134 --> 00:08:28.430
So the models will, instead of doing what even a relatively junior data scientist would do, which is you start looking at the data, you start exploring, you get an answer, and then you check that answer.
00:08:28.430 --> 00:08:33.870
Maybe you look at some other, you know, sort of other sources to see if you can corroborate what you're seeing.
00:08:33.870 --> 00:08:34.965
Models don't do that at all.
00:08:34.965 --> 00:08:39.924
They find a table, they're like, ah, this table has a column that looks like it should be useful, and then they just go.
00:08:39.924 --> 00:08:55.940
And so that's one thing that we're thinking a lot about is like how do we give the models how do we push them to explore the space more and how do we push them to sort of self validate in a way that more likely will because often, and you see this even with humans, right?
00:08:55.940 --> 00:09:04.995
If you start at different points in the data and there's different ways and different assumptions that you might make about how to answer a question, you can end up with just wildly different answers.
00:09:04.995 --> 00:09:09.634
And you kind of want to look at all of those and understand, okay, what's what's the actual sort of truth here?
00:09:09.634 --> 00:09:17.789
There's a lot of knobs to turn when making sure that the model is, you know, being a little bit more creative, maybe thinking a little bit harder about the answers.
00:09:17.789 --> 00:09:23.110
What are maybe the knobs that you're thinking about in Hex to make sure that this is being solved?
00:09:23.629 --> 00:09:24.029
Yeah.
00:09:24.029 --> 00:09:30.350
I think the models these days are very there are the the reasoning steps are actually quite good and quite solid.
00:09:30.350 --> 00:09:41.995
And it you know, if you rewind a couple years and hallucinations and just like them not paying attention to the things that you want them to pay attention to are huge, huge problems and even just keeping them on task.
00:09:41.995 --> 00:09:50.840
These days I actually find that they're quite creative in taking the pieces of information that they have and putting them together into an answer.
00:09:50.840 --> 00:10:00.815
Like that's act it's actually like really impressive when you read their thinking traces, how much they're able to like hit a roadblock and overcome it and like cycle through that in order to get to an answer.
00:10:00.815 --> 00:10:09.894
Where I usually see the problems happening is, as I was saying before, in them having not sufficiently explored the space.
00:10:09.934 --> 00:10:21.610
And so they're very, very good at coming up with an answer based on what they have, but if they don't have all of the pieces that you need them to have, then that's where they'll, you know, that's where they'll kind of fall on their face.
00:10:21.610 --> 00:10:28.014
They may even have an answer that looks plausible but it's actually not what you want, or maybe they'll, you know, they won't even get you an answer.
00:10:28.014 --> 00:10:33.215
So what we're thinking about is that exploration phase.
00:10:33.215 --> 00:10:40.169
And see this in other modes too, like in coding you have like a, can have like a plan and design phase and then an implementation phase.
00:10:40.169 --> 00:10:46.809
That's something that we're thinking about how we bake into our models, where you don't necessarily just like start from the question and have them go at it.
00:10:46.809 --> 00:10:54.365
You actually kind of push them to look harder for things before they come back and start to actually then go and do the analysis.
00:10:54.365 --> 00:11:03.325
Just to be sure, I'm I'm gonna try to paint in exquisite detail what it looks like to use Hex just because it's it's such an interesting cool experience.
00:11:03.325 --> 00:11:08.870
But I also think some people when I talk to people, some people listening I think might have trouble.
00:11:08.870 --> 00:11:25.085
If you've never used Hex, like, conveying it that and and correct me for anything I get wrong, but if I understand it's kind of like you have often like data in the warehouse and say, like, an analytics engineer or some people have tried to transform and kinda curate what you need.
00:11:25.085 --> 00:11:28.884
And and you might even be a person who doesn't do a ton of data analysis.
00:11:29.485 --> 00:11:36.929
You know, you might be working in, like, finance or not no shade on finance or, like, another area.
00:11:36.929 --> 00:11:42.330
And essentially, you go into this notebook that's wired up to your warehouse.
00:11:42.370 --> 00:11:46.409
And right now, you're able to ask an agent essentially a question.
00:11:46.450 --> 00:11:52.365
And then what happens is it starts populating cells that run and produce outputs.
00:11:52.365 --> 00:11:55.404
And some of these outputs even are like filters.
00:11:55.404 --> 00:12:07.029
So it might be like a drop down to select like something you're interested in or to filter a company that you might want data on.
00:12:07.149 --> 00:12:13.549
And then essentially, like, you can adjust the filter and everything else in the notebook will kinda update.
00:12:13.549 --> 00:12:18.470
So you can sort of like build this notebook and explore your data at the same time.
00:12:18.565 --> 00:12:25.044
But then what I I think is also interesting that I just wanna make sure to mention is, if I understand, you also have an app mode.
00:12:25.044 --> 00:12:25.605
Is that right?
00:12:25.605 --> 00:12:35.750
Where you kinda shift from seeing the code in the notebook view to the app mode, where the code sort of, like, goes away or is more hidden?
00:12:35.750 --> 00:12:36.870
Is that Yeah.
00:12:36.870 --> 00:12:38.629
Yeah. That's that's all totally correct.
00:12:38.629 --> 00:12:43.429
I think you can think about it in Hex as the backbone of the product is this notebook.
00:12:43.429 --> 00:12:45.509
And and we were just chatting about this earlier.
00:12:45.509 --> 00:12:52.695
That notebook actually under the hood is essentially a graph of transformation nodes that we can run behind the scenes.
00:12:52.695 --> 00:12:57.254
And then you can take that notebook, which we call project in Hex, and you can do a bunch of different things with it.
00:12:57.254 --> 00:13:05.830
So you can manipulate it with the agent, so the agent will actually edit that graph, it will build up that graph, and you can do that just to do sort of some exploratory analysis.
00:13:05.830 --> 00:13:15.590
But often what you want, and often, and this is actually one of the huge pain points that Hex was solving in the early days, is, you know, data workflows don't end with getting a number.
00:13:15.590 --> 00:13:17.825
They end with making a business decision.
00:13:17.825 --> 00:13:27.904
And so the way you get from I have an answer from the data to we've actually decided to do something, there has to be sort of a collaboration and presentation and sharing step.
00:13:27.904 --> 00:13:36.110
And so what we've done with Hex, it used to be excruciatingly difficult to go from those analyses to that, know, screenshotting charts in a PowerPoint and things like that.
00:13:36.110 --> 00:13:39.950
And so what we've done is we've actually layered a presentation layer directly on top of the notebook.
00:13:39.950 --> 00:13:48.294
So it's exactly the same and, you know, that, you know, it's a typical more dashboard like experience I would say.
00:13:48.294 --> 00:13:54.215
We call them apps because they're more, they can be more powerful under the hood than what you would expect from a traditional dashboard.
00:13:54.215 --> 00:13:55.654
And then you can also really extend this.
00:13:55.654 --> 00:14:00.340
So this has actually like been such a versatile paradigm for us.
00:14:00.340 --> 00:14:05.139
So Hex has the notebook and the notebook agent, which is really geared towards technical people.
00:14:05.139 --> 00:14:06.899
It's people who want to be in the code.
00:14:06.899 --> 00:14:10.794
Even if the agent's writing it, they want to be looking at it and and and working with it.
00:14:10.794 --> 00:14:20.315
But there's a lot of people in the organization who don't want to look at the code and but also still want to be able to take advantage of the power of the the Hex agent.
00:14:20.315 --> 00:14:25.634
And so what we have is we have our our conversational agent, which we call threats.
00:14:25.674 --> 00:14:28.879
But under the hood, it's actually the same as the Notebook agent.
00:14:28.879 --> 00:14:31.600
And so what it's doing I love it puts a different hat on.
00:14:31.600 --> 00:14:33.919
Yeah. It's a skin. Yep. Yep.
00:14:33.919 --> 00:14:36.320
But it's actually building up a Notebook behind the scenes.
00:14:36.320 --> 00:14:44.684
And this is really really powerful because it means that if you get to the end of this conversation and then you wanna do something with it, maybe that thing is share it.
00:14:44.684 --> 00:14:46.605
Well, great. You're in a notebook.
00:14:46.605 --> 00:14:51.164
You can like make the same types of dashboards, things like that, that you could before.
00:14:51.164 --> 00:14:54.710
Or maybe you're like, hey, I don't know if this is right.
00:14:54.710 --> 00:14:57.029
I actually wanna send it to the data team to check my work.
00:14:57.029 --> 00:15:00.190
Well, the work's all there and then I'll put behind the project.
00:15:00.470 --> 00:15:04.070
Yeah. Hadley, I know you in R, it's super different.
00:15:04.070 --> 00:15:04.309
Right?
00:15:04.309 --> 00:15:10.955
Like, the analysis the kind of workflow, like, versus a notebook where you're sort of like, this workflow is so great.
00:15:10.955 --> 00:15:13.195
If I'm gonna say, it's kinda like, whoops, I made a dashboard.
00:15:13.195 --> 00:15:17.034
Like, you do your notebook stuff, but then you have a dashboard at the end.
00:15:17.034 --> 00:15:18.715
Yeah. I think it's yeah.
00:15:18.715 --> 00:15:33.379
I think it's really interesting because the sort of heritage of R is this, like, REPL, this read eval print loop where you're in the console trying stuff out, and that's kind of your primary, like, means of exploration and experimentation.
00:15:33.379 --> 00:15:39.335
And then as you discover things you kinda believe to be true, like, that's when you record them, like, into a script.
00:15:39.335 --> 00:15:52.029
And that's kind of what led to R Markdown and Quarto, which is, like, you've written this mingling of prose and code, like, much like a notebook, but you then you run the whole thing, and that produces your output.
00:15:52.029 --> 00:15:56.190
And that that has some, like, big advantages, like, kind of prereactive notebooks.
00:15:56.190 --> 00:16:03.715
Very easy in a Jupyter notebook to kinda get into this state where you've, like, run some cells in, like, different orders, and there's no way to reproduce what you've done.
00:16:03.715 --> 00:16:08.274
Like, no way to have that problem with R Markdown because you always run it from scratch.
00:16:08.274 --> 00:16:14.034
Like, there's a different set of problems, obviously, and a different set of trade offs, but, like, that's the one problem you never experience.
00:16:14.034 --> 00:16:14.355
Yes.
00:16:14.355 --> 00:16:17.700
And then I guess there's often a dashboard phase after Yeah.
00:16:17.700 --> 00:16:20.100
And so the the thing that you produce might be a document.
00:16:20.100 --> 00:16:23.620
It might be a dashboard.
00:16:20.100 --> 00:16:23.620
It might be a presentation.
00:16:23.620 --> 00:16:33.044
Like, I think that that same, like, underlying thing, and you're putting on, like, different hats for different people, that's just that's, like, such a valuable metaphor.
00:16:33.044 --> 00:16:36.804
I mean, we we kind of often joke in engineering, I guess.
00:16:36.804 --> 00:16:38.404
I'm like, oh, that's a DAG.
00:16:38.404 --> 00:16:39.924
That's a directed acyclic graph.
00:16:39.924 --> 00:16:41.764
Like, everything's, oh, that's an immutable DAG.
00:16:41.764 --> 00:16:44.820
That's a reactive DAG. Like, oh, wow. What a surprise.
00:16:44.820 --> 00:16:48.100
Like, we've created a DAG, and it turns out to be the right model.
00:16:48.100 --> 00:16:54.259
Like, it's just And sometimes at obviously, the Hex notebook, as we've been talking about behind the scenes, is DAG.
00:16:54.259 --> 00:17:04.474
And as the as the cells within the notebook got more and more complicated, we started to realize that the cells themselves needed to be DAG. And so then you have DAGs and DAGs and DAGs.
00:17:04.474 --> 00:17:05.355
It's DAGs all the way down.
00:17:05.355 --> 00:17:06.954
And, actually, you know, what is a DAG of DAGs?
00:17:06.954 --> 00:17:08.954
It's just A DAG.
00:17:08.954 --> 00:17:13.529
It's kinda like every it feels like every problem with an LLM, you solve with, like, another LLM.
00:17:13.529 --> 00:17:15.690
Like, you don't know if your LLM is doing a good job or not.
00:17:15.690 --> 00:17:19.450
Well, let's have an LLM a different LLM judge, but it's doing a good job.
00:17:19.450 --> 00:17:20.569
It's yeah.
00:17:20.569 --> 00:17:22.650
It's like turtles all the way down.
00:17:22.650 --> 00:17:34.805
And another thing just just to not to keep buttering up Hex, but OpenAI, they released their one of their reports was using Hex.
00:17:34.805 --> 00:17:35.285
Is that right?
00:17:35.285 --> 00:17:38.644
Like a state of data Probably Anthropic.
00:17:38.644 --> 00:17:41.779
Anthropic. Yes. Okay. Yes. I gotta be careful about my Yeah.
00:17:41.779 --> 00:17:42.980
Do you recall what that was?
00:17:42.980 --> 00:17:45.779
There was like a I don't know specifically what you're referring to.
00:17:45.779 --> 00:17:50.099
There's a there's a few times where we've seen Anthropic kind of use Hex in Pops up.
00:17:50.099 --> 00:17:52.339
Yeah. Because, you know, they're they're Hex customers.
00:17:52.339 --> 00:17:53.299
Yeah.
00:17:53.299 --> 00:17:56.785
And sometimes it's fun to like watch their live streams and like spot the Hex chart.
00:17:56.785 --> 00:17:59.505
Yeah. Yeah. For Hex.
00:17:56.785 --> 00:17:59.505
Yeah. Yeah. That's cool.
00:17:59.505 --> 00:18:03.825
And I know I know you mentioned Sweetgreen as an example.
00:18:03.825 --> 00:18:05.744
Oh, yeah. This is a recent one.
00:18:05.744 --> 00:18:08.944
Hex Hex is like the whole spectrum of customers.
00:18:08.944 --> 00:18:16.579
You it's like very traditional data science all the way to some of these use cases that you would just never imagine.
00:18:16.859 --> 00:18:22.619
So Sweetgreen just signed on as a Hex customer and this is the wildest one that I've I've heard in a while.
00:18:22.619 --> 00:18:36.255
They use Hex to monitor the temperature of their ovens so that they can correlate customer complaints about the temperature of their food to, you know, what's happening on the ground in their restaurants.
00:18:36.255 --> 00:18:38.089
It was wild.
00:18:38.089 --> 00:18:40.970
Every now and then you come up with you can't you hear some of these things from customers.
00:18:40.970 --> 00:18:44.890
It's actually one of the really fun things about building a product that's so flexible and so horizontal.
00:18:44.890 --> 00:18:47.450
Did the head chef have, like, a lot of data experience?
00:18:47.450 --> 00:18:50.049
Like, what's You know, I'm not I'm not sure.
00:18:50.125 --> 00:18:55.884
I I, you know, I I actually haven't met him him personally, but this is one that we heard kind of from our field teams.
00:18:55.884 --> 00:18:56.365
Yeah.
00:18:56.365 --> 00:18:56.525
Yeah.
00:18:56.525 --> 00:19:10.980
Like, one of the things I've sort of been, you know, thinking about, like, how like, as these, you know, people who never would have, you know, worked with data in this way before because they couldn't program, like, and now they can because they can write in English.
00:19:10.980 --> 00:19:19.045
But, you know, there's a lot of, like, vocab and, like, ways of thinking about data that you don't you you don't have and you can't express in everyday English?
00:19:19.045 --> 00:19:35.759
Like, how do we start to, like, train those people up to give them, like, the basic kind of like, that kinda gut feel for, like, I should be a little bit skeptical here or, oh, what I what I need right now is, like, a smooth or, like, I need tidy data? Or how do you all think about that?
00:19:35.759 --> 00:19:36.079
Yeah.
00:19:36.079 --> 00:19:50.035
So I think this is an unsolved problem broadly, in that I think building these agents to behave like a data scientist, truly, I think we haven't gotten there yet, and there's a lot more work to be done there.
00:19:50.035 --> 00:20:01.700
But I actually think that this is one of the big breakthroughs broadly in terms of, you know, AI for data and really enabling a much broader spectrum of people to work with data.
00:20:01.700 --> 00:20:06.819
Because, yes, there's, you know, the actual can you write code aspect of it.
00:20:06.819 --> 00:20:16.025
But if you look at sort of the prior generation of data tools, even ones where you don't have to write any code, people still don't use them.
00:20:16.025 --> 00:20:18.505
Like, the laypeople in the organization do not use them.
00:20:18.505 --> 00:20:24.904
Like, we talked to a Looker, an ex Looker customer now, they're an ex customer, and they said, you know, they bought Looker for everybody.
00:20:24.904 --> 00:20:33.430
And in practice, it was like they looked at the last ninety days and only two people in the company had made like new dashboards in Looker.
00:20:33.630 --> 00:20:36.750
And when you like unpack that a little bit, it's it's just buttons.
00:20:36.750 --> 00:20:38.910
Right? You don't have to write code. How hard can it be?
00:20:38.910 --> 00:20:43.454
It's actually that doing data work is just inherently very complex.
00:20:43.815 --> 00:20:48.775
And so none of these tools, code or otherwise, were walk up usable.
00:20:48.775 --> 00:20:53.990
There was just too much foundational knowledge that you needed in order to be able to be effective in the tool.
00:20:53.990 --> 00:21:02.349
And so one of the big breakthroughs besides the, you know, LLM writing code bit is actually imbuing the agents with that knowledge.
00:21:02.470 --> 00:21:21.184
And today, that's already a lot of what we're doing with the Hex agent, is like teaching it not just like, you know, how to write code, but you know, how to work with the data, you know, your tribal knowledge around the organization of what data is where, all of those things you bake into the agent.
00:21:21.464 --> 00:21:28.579
And so you're starting to see this pattern across, you know, across agents and general purpose models with skills and things like that.
00:21:28.579 --> 00:21:42.494
Again, I don't think we've totally nailed it because I think even with a skill, some of these more specialized domains like data and data science, the models haven't been post trained on it as, you know, as rigorously as they have been with coding.
00:21:42.494 --> 00:21:45.454
I think there's like, you know, still a fair amount of headroom.
00:21:45.454 --> 00:21:52.529
But it's like, even if it's, you know, sixty, seventy percent, it's still massively better than someone who's never really thought about how to work with data before.
00:21:52.529 --> 00:21:59.490
So our internally at Hex, some of our we have we have this feature that we call Guides, which is basically basically Skills.
00:21:59.490 --> 00:22:07.035
And some of them are like, you know, this is how this dataset is shaped and things like And some of them are, you know, this is how we do data analysis at Hex.
00:22:07.035 --> 00:22:09.434
These are the types of things that you should do.
00:22:09.434 --> 00:22:12.795
And the models are pretty good at adhering to it today.
00:22:12.795 --> 00:22:19.009
But again, as we've been talking about, if there's any kind of like judgment or anything that they would need to off road on, that that they're like much, much less good at.
00:22:19.009 --> 00:22:31.690
It is and, like, I think one of the things a good data scientist has always done, like, you come with a question, and then the data scientist, like, helps you turn that into a question that you can actually answer, like, with the data you have.
00:22:31.714 --> 00:22:43.555
And, like, you know, if you just go into the data warehouse and you just start asking questions, like, you you just don't even know, like, the right questions to ask, and that's such a valuable kind of and we're starting to get that a little bit with LLMs, I think.
00:22:43.555 --> 00:22:46.509
They can at least help you brainstorm and kind of say, oh, hey.
00:22:46.509 --> 00:22:47.950
You could ask this instead.
00:22:47.950 --> 00:22:53.150
But, yeah, it's it's it's interesting interesting time to be a data scientist.
00:22:53.150 --> 00:23:05.045
The other thing that we do in Hex that I think is is is very powerful is the agents have access to not just the underlying data warehouse, but the other things that people are doing with the data.
00:23:05.325 --> 00:23:09.484
And so some of this is gonna be garbage, just like, you know, people not knowing what they're doing.
00:23:09.484 --> 00:23:15.670
But, you know, you may have that company dashboard that gets looked at thousands of times a day or whatever.
00:23:15.670 --> 00:23:20.630
And that is actually a very rich source of context for the agent for how you should do something.
00:23:20.630 --> 00:23:33.414
And so when a user comes in is like, I wanna know something about revenue, they may not know anything about how we actually calculate revenue, but that's actually encoded in various places in your data workspace that the agent Which is yeah.
00:23:33.414 --> 00:23:35.815
Which is because that I mean, that's also how humans work.
00:23:35.815 --> 00:23:37.335
Right? Often that's not documented.
00:23:37.335 --> 00:23:41.855
You have to go to another successful project and, like, copy and paste that code. Yeah.
00:23:42.670 --> 00:23:42.910
Yeah.
00:23:42.910 --> 00:23:52.990
That was an interesting point I think I saw posted recently was, like, this idea of maybe, like, some of your best documentation is just the other projects around.
00:23:52.990 --> 00:24:14.220
And the the agents it's if I understand, y'all made it so the agent can essentially, like, look across projects, so that if it doesn't exist, like, in the documentation or in the models, it can sort of figure out how someone else maybe, like, went off road or use some kind of, like, knowledge in their head to do something?
00:24:14.220 --> 00:24:15.019
Yeah. Exactly.
00:24:15.019 --> 00:24:28.194
I mean, the the idea here is it's always been extremely difficult and basically impossible for humans to exhaustively and thoroughly document their entire data ecosystem and then maintain that and keep that keep them up to date.
00:24:28.194 --> 00:24:37.075
So what are the other places that you can go to understand, you know, how to use particular parts of data or how how to do particular types of analyses?
00:24:37.075 --> 00:24:48.029
And, you know, it's it's a range because when you have when you have the the documented context, for the most part, that's like highly trusted and that's something that the agents can just rely on.
00:24:48.029 --> 00:25:03.555
But then everything else is this range from you could go all the way down from like, oh, I'm just gonna like look at the SQL queries that were executed against this table, all the way up to, you know, you have a project that, you know, is highly used and maintained by the data team.
00:25:03.555 --> 00:25:08.500
And in Hex, you We have this mode called endorsements where basically the data team can put a tag on it that's like, this is Oh, funny.
00:25:08.500 --> 00:25:10.579
This is like a less Gold gold star.
00:25:10.579 --> 00:25:11.779
Yeah. Yeah. Essentially, yes.
00:25:11.779 --> 00:25:24.964
And and all of this is to help guide the agent towards you want it you want it to be able to take advantage of all the context that it's able to, but you also want it to be able to distinguish between the highly trusted parts and then the the less trusted parts.
00:25:24.964 --> 00:25:29.765
Because I think that that is, like, one of the risks of agents because they can be so much more productive.
00:25:29.765 --> 00:25:42.049
You just get stuck in this loop where they kinda get worse and worse and worse, and a greater and greater proportion of your internal dashboards and stuff are written by agents, and they're just learning from other agents who made mistakes, like, fifteen months ago.
00:25:42.049 --> 00:25:46.210
And, yeah, like that that kind of idea of saying, like, we also need to give feedback.
00:25:46.210 --> 00:25:47.890
Like, these are the really good things.
00:25:47.890 --> 00:25:49.775
Like, like, copy from that, not from.
00:25:49.775 --> 00:25:57.055
And and this, I think, is one of the key sort of new jobs to be done for the data teams, if that makes sense.
00:25:57.055 --> 00:26:09.430
And a big part of of Hex that we haven't really talked about yet, but is relatively new post agent, is what we call the context studio, which is basically about observability and management of all of the context for all of the agents.
00:26:09.430 --> 00:26:13.430
And so you can see what people, you know, what the agents are actually doing.
00:26:13.430 --> 00:26:17.190
For each thread, you can actually see what context the agent is using.
00:26:17.190 --> 00:26:26.505
And then we have a second sort of like a separate set of agentic workflows around managing the context, around extracting the pieces out of these things.
00:26:26.505 --> 00:26:40.690
And then putting that, feeding that back into the like highly trusted, highly governed context to kind of avoid this like spiraling problem of, you know, the agent just kind of deciding that that's the way to do something and then not producing sort of being a self fulfilling prophecy over there.
00:26:40.690 --> 00:26:42.210
You can actually like break that.
00:26:42.210 --> 00:26:45.529
You can see that it's actually doing something that's that's incorrect.
00:26:45.569 --> 00:26:55.095
And then update your human managed context, which is prioritized, you know, more highly above the kind of ambient context for the agents.
00:26:55.095 --> 00:26:56.295
I was gonna say, I'm really curious.
00:26:56.295 --> 00:27:01.494
I read recently that now more agents are deploying code on Hex than humans.
00:27:01.494 --> 00:27:07.630
How do you manage as someone who obviously cares so much about their users and, like, is giving people such a beautiful suite of tools?
00:27:07.630 --> 00:27:11.869
Like, what are you making for agents versus what are you making for people nowadays?
00:27:11.869 --> 00:27:13.230
Yeah. It's a really good question.
00:27:13.230 --> 00:27:22.265
I think in the early days of building the Hex Agent, it was very clear that the tools that we'd built for humans were actually also very good for agents.
00:27:22.265 --> 00:27:24.025
And so there was a lot of overlap.
00:27:24.025 --> 00:27:33.585
You know, we had this modular notebook structure, looks a lot like tools that you can give to the agent, and the agents just took to that very naturally.
00:27:33.710 --> 00:27:38.990
And that was like the first generation, I would say, where it was pretty pretty one to one.
00:27:38.990 --> 00:27:49.684
But now that we've had this sort of crossover moment of more code is being written in Hex by agents than by than by people, you start to think about what that implies for the rest of the system.
00:27:49.684 --> 00:27:55.285
And you still have, you still want the human interfaces because there are still times where people are gonna come in and tweak things.
00:27:55.285 --> 00:27:58.964
And you know, it's it's not it's not fully fully be a natural language.
00:27:58.964 --> 00:28:04.819
Like sometimes typing a whole sentence is just a very inefficient way to do something that's, you know, as opposed to clicking a button.
00:28:04.819 --> 00:28:18.605
But at the same time, knowing that over time more and more of this is going to get produced by agents, it actually gives you a lot of opportunities to build things that are useful for agents but less useful for humans but are very powerful for the agents.
00:28:18.644 --> 00:28:33.109
And so, you know, one thing that we've thought about in Hex is could could we give the agents like slightly different all all of our cells are built to be, you know, they're they're very sort of like easy to inspect.
00:28:33.109 --> 00:28:36.789
You get a little preview. You get all these nice UI features.
00:28:36.789 --> 00:28:38.150
Agents don't need that.
00:28:38.150 --> 00:28:39.750
And actually they're quite expensive.
00:28:39.750 --> 00:28:45.444
So if an agent is kind of building up a project, could you give it a slightly different set of tools that would have different characteristics?
00:28:45.444 --> 00:28:48.565
Maybe, you know, less data warehouse load, maybe it runs faster.
00:28:48.565 --> 00:28:53.924
Maybe it's just honestly, sometimes agents like to work in a different way than humans.
00:28:53.924 --> 00:28:58.924
They've been trained on a certain distribution of data, and it's not exactly what our human brains have been trained on.
00:28:59.049 --> 00:29:02.490
And so I do think that there will be some some divergence there.
00:29:02.490 --> 00:29:09.210
But the, you know, the power of Hex is that these things are, you know, shared across like, they're they're interoperable.
00:29:09.210 --> 00:29:12.125
And that way, I think think that will continue to be true for a while.
00:29:12.125 --> 00:29:23.244
One of the things I really like about the kind of context studio is it feels like Hex is sort of thinking, well, like, the job of, like, what a data team or data scientists do is changing.
00:29:23.244 --> 00:29:40.045
This is gonna be one of the things where you give value to your organization as opposed to, like, there's also a lot of startups out there today, which their pitch is basically, like, buy us and fire your data team, which, you know, as a data scientist doesn't feel very nice.
00:29:40.045 --> 00:29:43.964
Well, we were talking to a head of of data, and he said something that was really interesting.
00:29:43.964 --> 00:29:52.960
And I think when you frame it in this way, it it made it really clear to me, you know, why data teams are not going away.
00:29:52.960 --> 00:29:57.839
He said, my job is to vouch for the correctness of the data and the answer that you get.
00:29:57.839 --> 00:30:03.015
Like he was a he was sort of, you know, was the head of the data team and obviously like that's slightly slightly different role than ICs on the data team.
00:30:03.015 --> 00:30:11.015
But when you frame it like that, you're like, oh yeah, actually like in the abstract that is when you when you don't have people clicking buttons, that is the job of the data team.
00:30:11.015 --> 00:30:15.535
The job of the data team is is to make a system that produces correct answers.
00:30:16.349 --> 00:30:28.069
And today, a lot of that has to do with instead of, you know, being the person who writes the SQL that is correct, it doesn't have bugs in it, how do you set up the context layer such that the agents can arrive there themselves?
00:30:28.345 --> 00:30:29.305
I mean yeah.
00:30:29.305 --> 00:30:39.065
I mean, like, the depressing thing is, like, definitely in corporate America, there is, like like, having the person to fire when something goes wrong is, like, important.
00:30:39.065 --> 00:30:40.825
Well, I mean, even more positively.
00:30:40.825 --> 00:30:45.470
Like, someone has to own the stuff and be like, yeah, it's my reputation on the line that this is correct.
00:30:45.470 --> 00:30:50.990
Like, you can't just distribute that to a bunch of, like, nebulous agents that no one really understands.
00:30:50.990 --> 00:30:51.150
Yeah.
00:30:51.150 --> 00:30:58.174
And I think I think ownership is not just like, it's my reputation on the line, but it's like, you know, my my job is to make sure that this functions.
00:30:58.174 --> 00:31:15.779
And because it's, you know, in in organizations, like this data is being used for, you know, existential, answering existential questions, like, the business needs someone to care that that is correct and to, you know, build the system that is correct rather than just kind of assuming that everything's gonna be fine and figure it out.
00:31:15.779 --> 00:31:16.500
But yeah.
00:31:16.500 --> 00:31:24.164
But it does feel like data the data teams are gonna get more, like, unhinged analysis from execs.
00:31:24.164 --> 00:31:26.164
Like, that's also gonna have to be part of their job.
00:31:26.164 --> 00:31:28.724
Like, someone has done something that they think is really cool.
00:31:28.724 --> 00:31:31.684
Like, you know, that and and it's cool it is really cool Yeah.
00:31:31.684 --> 00:31:35.125
That more people in the org are, like, interacting directly with the data.
00:31:35.125 --> 00:31:41.640
But the consequence of that is, like, more people are gonna be making mistakes, and the data team is gonna have to, like, walk them back.
00:31:41.640 --> 00:31:41.960
Yeah.
00:31:41.960 --> 00:31:45.240
You already see this you see this in in software engineering.
00:31:45.240 --> 00:31:58.714
Like, you know, a CEO will come in and, like, you know, use Cursor or whatever to, like, vibe code some insane thing and then, like, make a pull request because, like, all the, like, traditional barriers to that happening and then the the engineers are kept to code and like review this thing.
00:31:58.714 --> 00:32:07.275
I I do think you'll start to see a similar pattern like that in in data where all of a sudden people have all these really powerful tools at their fingertips and they don't know how to use them.
00:32:07.275 --> 00:32:08.910
And I I think there's a couple things there.
00:32:08.910 --> 00:32:18.430
One is, you know, in Hex, we're thinking a lot about building review tools, both, you know, manual and also agent assisted review tools so that that process is as painless as possible.
00:32:18.430 --> 00:32:22.255
But I also think it's just another indication of, you know, where there will still be value.
00:32:22.255 --> 00:32:26.654
Like, there are there are times where you actually want an expert to do the analysis.
00:32:26.654 --> 00:32:31.255
There's many, many ways to do the wrong things with statistics.
00:32:31.295 --> 00:32:43.099
And if you if you have a very important decision, there will still be a a time and a place for for people to kind of do those things and not just have someone, you know, fire off an agent query and and, you know, not not be able to really think critically about the the result they're getting.
00:32:43.099 --> 00:32:49.940
It's it's funny you say that because our CEO got in trouble with our IT department for an app he vibe coded with HR.
00:32:50.005 --> 00:33:01.525
But I think it's so, like, maybe empowering and lovely to think about, you know, data scientists are coming with all of the the muscles for, you know, really productive AI usage.
00:33:01.525 --> 00:33:03.880
Like, data scientists do a lot of experiments.
00:33:03.880 --> 00:33:07.480
Data scientists know you look at something, say, wow, this is bad.
00:33:07.480 --> 00:33:09.640
Like, let's scrap it all and start again.
00:33:09.640 --> 00:33:11.559
So I think there's something really interesting.
00:33:11.559 --> 00:33:20.775
And especially with the aid of things like review tools built in the loop, you know, no better time, question mark, to be someone who's writing code to do data science.
00:33:20.775 --> 00:33:21.174
Yeah.
00:33:21.174 --> 00:33:25.734
And I think there's a lot of hype around AI engineering and and things like that.
00:33:25.734 --> 00:33:28.454
And we internally at Hex have an AI engineering team.
00:33:28.454 --> 00:33:37.259
But one thing that I've observed is sort of the lines between our AI engineering team and our data team really blurring and starting to overlap.
00:33:37.380 --> 00:33:48.164
And so, for example, a lot of things that we'll do internally for our ourselves to build our own product and build our own agent, We'll build, know, we have evals, we have various other things that we've built.
00:33:48.444 --> 00:34:02.599
You know, we'll we'll look at those and be like, oh, we should actually take those and put those into the product for our customers because these workflows of, you know, building a highly trusted agent actually, you know, converge over time.
00:34:02.599 --> 00:34:03.000
Yeah.
00:34:03.000 --> 00:34:07.480
Who runs Hex's internal context studio?
00:34:07.480 --> 00:34:10.119
Who's the person curating that?
00:34:10.119 --> 00:34:11.315
Yeah. That's that's our data team.
00:34:11.315 --> 00:34:12.195
Nice. Yeah. Okay.
00:34:12.195 --> 00:34:15.474
And they are, like, our number one dog fooders for obvious reasons.
00:34:15.474 --> 00:34:15.954
Yeah.
00:34:15.954 --> 00:34:18.514
And do you see, like, the lines blurring elsewhere?
00:34:18.514 --> 00:34:28.570
Like, it you know, not just because you know, I I guess, like, in our like, we are starting to see, like, designers who previously would create, you know, like, a beautiful thing and turn it over to engineers to implement.
00:34:28.570 --> 00:34:34.730
Now they can give a work deliver a working prototype or, like, product people who previously were like, hey, engineer.
00:34:34.730 --> 00:34:37.369
Can you implement this? They just go away and do that.
00:34:37.369 --> 00:34:41.224
Like, are you seeing that blurring of lines in your engineering org too?
00:34:41.224 --> 00:34:43.585
I would say absolutely.
00:34:44.184 --> 00:34:52.585
Our but I think it's a little bit less dramatic at Hex than maybe at other places because our org has always kind of been designed like this, if that makes sense.
00:34:52.585 --> 00:34:59.269
Most of our PMs used to be technical, like, actually hands on technical in some some capacity.
00:34:59.269 --> 00:35:05.750
All of our designers, we, like, give them an interview that's designed to be like, can you do light, you know, CSS and React work?
00:35:05.750 --> 00:35:11.105
Like, we don't want people to just be completely boxed out of a core part of the of the technology.
00:35:11.304 --> 00:35:15.224
You know, our front end engineers and our back end engineers kind of flow back and forth.
00:35:15.224 --> 00:35:17.065
Everybody has their spikes.
00:35:17.065 --> 00:35:21.199
You know, it's not like we're expecting, you know, a back end engineer to be, you know, deep in React.
00:35:21.199 --> 00:35:28.840
But we don't want people to have these really hard boundaries because that can often create just like a lot of really annoying inefficiencies in the organization.
00:35:29.280 --> 00:35:31.039
And so, yes, we are seeing that blur.
00:35:31.039 --> 00:35:32.960
We're sort of seeing that intensify, I would say.
00:35:32.960 --> 00:35:40.164
Like, you know, designers and product managers are like can go much deeper of in what they're able to contribute than than they used to be.
00:35:40.164 --> 00:35:42.644
But, you know, we hired a lot of people who are excited about that.
00:35:42.644 --> 00:35:44.804
So I think that's like just strictly a good thing.
00:35:44.804 --> 00:35:45.525
Yeah. Yeah.
00:35:45.525 --> 00:35:57.170
It's it's interesting and maybe this is a nice segue into to a less kinda AI more style topic, but Hex has incredible style.
00:35:57.170 --> 00:35:58.610
Hex is just cool.
00:35:58.610 --> 00:35:59.889
I don't know how to explain.
00:35:59.889 --> 00:36:03.570
Like you go to the Hex website and it's been like this for years and years and years.
00:36:03.570 --> 00:36:08.155
I feel like the website, cool.
00:36:08.155 --> 00:36:13.034
The the booths at conferences, unhinged.
00:36:13.034 --> 00:36:14.554
It's like In a good way.
00:36:14.554 --> 00:36:18.994
In a good way. I'm sorry.
00:36:14.554 --> 00:36:18.994
Like a a retro dining booth.
00:36:19.019 --> 00:36:21.420
Hex is throwing like magic shows.
00:36:21.420 --> 00:36:23.820
What I I don't know how to ask this.
00:36:23.820 --> 00:36:27.900
How did Hex acquire such immaculate style?
00:36:27.900 --> 00:36:31.974
Well, I'm gonna have to give some credit to some other people in the organization for that one.
00:36:31.974 --> 00:36:37.175
My cofounder Barry, the second person we ever hired was Adam, our head of design.
00:36:37.175 --> 00:36:41.094
And I think, you know, those two have been kind of the masterminds.
00:36:41.094 --> 00:36:45.215
Also Annie, our our head of brand design, have been the masterminds of a lot of these things.
00:36:45.250 --> 00:36:52.849
In some ways it's down to specific individuals and having particular individuals who have these kind of off the wall takes.
00:36:52.849 --> 00:37:02.945
But I think also it's about the culture that you build as an organization and and who you find and and us trying to bring in people who are creative, who don't take themselves too seriously.
00:37:02.945 --> 00:37:10.184
And when you have you have some of those things, it creates an environment where you can be generative and think think differently like this.
00:37:10.980 --> 00:37:11.860
And and yeah.
00:37:11.860 --> 00:37:19.019
So it's it's both there's like some very, very talented people that we have and also, you know, the team that we've built around them.
00:37:19.059 --> 00:37:20.579
Yeah. I really like the Hex.
00:37:20.579 --> 00:37:25.215
Like, the Hex color scheme is like so like, it really stands out in, like, tech.
00:37:25.215 --> 00:37:27.375
No one is, like, pink and cream and brown.
00:37:27.375 --> 00:37:31.534
Like, that's I mean, probably now that's the next next trend.
00:37:31.534 --> 00:37:43.059
But, like, you know, I I kinda love like, we we spend a lot of effort at Posit coming up with our logo and stuff, but just, like, blue and orange is just, like, such a, like, traditional kinda tech color scheme.
00:37:43.059 --> 00:38:01.755
It's, yeah, it's it's just fascinating to see, like, as we grow as a company, how these kind of forces of, like, unification and kind of corporatization, like, pull on you towards these just, like this is, like, the safe safe choice.
00:38:01.755 --> 00:38:02.635
And Yeah.
00:38:02.635 --> 00:38:08.230
And some of that just has to do with you have to scale your content and branding engine.
00:38:08.230 --> 00:38:15.389
And so early on, a lot of creativity can actually just come from particular individuals.
00:38:15.670 --> 00:38:22.894
And you almost always get certainly a more, like, unique result when you're when you're doing that, you're doing that in small teams.
00:38:22.894 --> 00:38:33.454
And then you get to a point where it's like, oh, you have to write all of this content and you have to put out things on Twitter and LinkedIn to get the, you know, SEO or AEO or whatever we're doing now.
00:38:33.454 --> 00:38:40.530
And you, the just the amount of stuff that you have to produce can't bottleneck through your main creative engines.
00:38:41.130 --> 00:38:43.690
And sometimes you can scale.
00:38:43.690 --> 00:38:49.730
Like, you you obviously scale your brand voice and and and your brand styling and things like that, but you can't do it perfectly.
00:38:50.324 --> 00:38:55.045
And so I do think that, you know, we're we're trying to fight really hard against this and trying to, like, balance the two.
00:38:55.045 --> 00:39:03.980
But you do sometimes sort of inevitably have feel this, like, regression towards the mean pull, just like the more and more stuff you have to do on the branding side.
00:39:03.980 --> 00:39:04.460
Yeah.
00:39:04.460 --> 00:39:12.340
And it's sort of also interesting just as the company grows, like, the the amount of visibility you have within all of the parts of the organization.
00:39:12.619 --> 00:39:20.815
And and this, like, sounds weird coming from a data person, but, like, you are forced to start developing, like, metrics so that you can get some oversight.
00:39:20.815 --> 00:39:36.340
And then, like, metrics are good, obviously, but they're also bad because they kinda, like, trap you into these, like, predefined like, how like, how do you, like, wrestle with that as, like, clearly a data driven company, but you don't wanna be, like, too data driven at the same time.
00:39:36.460 --> 00:39:36.940
Yeah.
00:39:36.940 --> 00:39:52.355
I think for me, a lot of it a lot of it has come down to being able to sort of shift the altitude at which you operate and then knowing when is the right time to be at what altitude.
00:39:52.355 --> 00:40:02.199
So obviously, it's extremely important, you know, as a senior leader of a company to be thinking about strategy and long term time horizons, and I spend a lot of time, obviously we spend a lot of time talking about AI.
00:40:02.199 --> 00:40:06.079
That's where I spend our AI strategy is where I spend a lot of my time thinking.
00:40:06.760 --> 00:40:11.119
And then, you know, you're intermediated through layers and layers of organization.
00:40:11.800 --> 00:40:16.864
And sometimes there's something down at the bottom that's like not going well.
00:40:16.864 --> 00:40:20.704
And if you've built a a good organization, you'll know about that most of the time.
00:40:20.704 --> 00:40:22.864
But sometimes you don't.
00:40:22.864 --> 00:40:28.425
And sometimes you just, you see things in the data or you see things, you know, that that don't feel right.
00:40:29.000 --> 00:40:39.320
And I think for me, cultivating that kind of like instinct of, hey, this section feels like it feels like there's something not working here.
00:40:39.320 --> 00:40:44.320
The vibes are off and being able to just do a deep dive on that.
00:40:44.385 --> 00:40:50.864
And in order to be able to do that, you have to know enough about the things that are like happening on the ground.
00:40:50.864 --> 00:41:10.355
And so for me, you know, when I first started really building up a bunch of our AI programs, spent, and I still do actually, like, I read the papers, like I spend a bunch of time in the technology and that's to both to like be able to see the big picture, but also when something's not right on the ground to be able to kind of go all the way down and like help debug, you know, where something is breaking down.
00:41:10.355 --> 00:41:12.835
Yeah. As CTO of Hex, I'm so curious.
00:41:12.835 --> 00:41:15.394
What is your what does that look like?
00:41:15.394 --> 00:41:18.474
Like, what's your day to day look like?
00:41:18.514 --> 00:41:21.355
It's a pretty big range of things.
00:41:21.394 --> 00:41:31.059
I think one of the funny things about being CTO is sometimes I feel like my job is more about humans than it is about technology, even though technology is in the title.
00:41:31.059 --> 00:41:43.335
And so a lot of it is about when I'm not sort of thinking about the AI side of the AI technology side of things, a lot of it is about like how how are we running a functional technology organization?
00:41:43.335 --> 00:41:49.375
And there's engineering, there's data, there's there's our AI research side, there's security, there's all of these things involved in that.
00:41:49.894 --> 00:41:55.289
And a lot of that day to day is actually not necessarily like making technology decisions.
00:41:55.289 --> 00:41:56.409
Like, I've hired smart people.
00:41:56.409 --> 00:41:59.690
If I've done especially in areas like security where I'm not the expert.
00:41:59.690 --> 00:42:05.289
If I've done my job right, I've hired people who are better at making those technology decisions than I am.
00:42:05.289 --> 00:42:09.844
It's about, you know, setting up the the humans to do their best work.
00:42:09.844 --> 00:42:20.644
But as someone has like come up from the kind of technologic, like the technical side, like how do you like, you still is there some amount of programming, like, you just want to do to kind of like scratch that internal itch?
00:42:20.644 --> 00:42:22.030
You're like, let me cook.
00:42:22.030 --> 00:42:23.070
Yeah.
00:42:23.070 --> 00:42:24.510
Yeah. For sure.
00:42:24.510 --> 00:42:31.150
A lot of it I try to do on sort of side less, you know, mission critical critical path things right now.
00:42:31.150 --> 00:42:33.070
Some of it is actually fun to get.
00:42:33.070 --> 00:42:40.514
So when I was in college, my concentration was then called ML, and then I did my master's in NLP.
00:42:40.514 --> 00:42:46.034
And so I've always really been interested in this side of the house, but this was over ten years ago.
00:42:46.034 --> 00:42:49.429
For for people who might NLP is natural language. Processing.
00:42:49.429 --> 00:42:50.309
Processing. Yeah.
00:42:50.309 --> 00:43:03.429
And it's all about It's basically all of it's like pre LLMs, pre generative AI, how you operated with natural language in, you know, in a statistical sort of modeling sense.
00:43:03.429 --> 00:43:12.195
And actually a lot of the techniques back when I was in college were sort of hilariously primitive versions of what we have today, if that makes sense.
00:43:12.195 --> 00:43:19.739
So But you know, it was it was pretty clear, this was twenty twelve, was pretty clear that this was like, the stuff that they were doing was like, kind of neat.
00:43:19.739 --> 00:43:23.260
And you could like see some interesting properties of language with it.
00:43:23.260 --> 00:43:25.500
But like, your ability you you couldn't generate text.
00:43:25.500 --> 00:43:30.300
Like, your your ability to actually do things, like machine translation was like a big area of study back then.
00:43:30.300 --> 00:43:32.704
And it was like, you know, comically bad. You know?
00:43:32.704 --> 00:43:40.625
It was like, you could get the gist of what the what the text was saying, but it it wasn't anywhere approximating real speech.
00:43:40.625 --> 00:43:47.144
And it was pretty clear to me that like, the techniques that we were learning about as state of the art at the time were like, not it.
00:43:47.800 --> 00:43:52.760
And and so I kind of like left that and went into just core software engineering because I like, I wanna do something real.
00:43:52.760 --> 00:43:54.519
Wanna do something that's like practical.
00:43:54.519 --> 00:44:00.039
And then sure enough, a few years later, Attention Is All You Need was published, and then it's kind like off to the races from there.
00:44:00.039 --> 00:44:02.954
But so so sometimes it's me getting hands on and building things.
00:44:02.954 --> 00:44:05.914
And sometimes it's me just like getting back into that has been really fun.
00:44:05.914 --> 00:44:15.860
And spending some time not just like reading blogs, but like reading the actual underlying papers and like under understanding understanding the research side has been has been super fun over the last couple years.
00:44:15.860 --> 00:44:16.579
Yeah. Oh, that's interesting.
00:44:16.579 --> 00:44:18.820
So you've been kind of diving back into NLP.
00:44:18.820 --> 00:44:20.420
It kind of like reignited.
00:44:20.420 --> 00:44:20.739
Yeah.
00:44:20.739 --> 00:44:23.059
I mean, well, now now it's just, you know, all generative AI.
00:44:23.059 --> 00:44:24.579
And it's like it's sort of a blurred line.
00:44:24.579 --> 00:44:27.780
Large language models do so many more things other than just NLP.
00:44:27.780 --> 00:44:34.635
But yeah, a lot of a lot of it is like, again, at at the foundational level, like, you know, similar techniques.
00:44:34.635 --> 00:44:36.954
It's all just matrix multiplication under the hood anyway.
00:44:36.954 --> 00:44:40.315
But it it is interesting, like, you know, like, named named entity recognition.
00:44:40.315 --> 00:44:42.474
Yeah. Used to be like this really challenging thing.
00:44:42.474 --> 00:44:49.400
You have to have a massive tech stack specifically for that, And now you just, like, throw it at any, like, random It's it does a good job.
00:44:49.400 --> 00:44:52.440
If I'm a barbarian, what's named entity recognition?
00:44:52.440 --> 00:44:56.840
Like, you know, if you've got, like, a blog post and you just wanna extract all the people's names that were mentioned in it.
00:44:56.840 --> 00:44:59.974
Like, that's you can't do that with a regular One of the hard problems naming things.
00:44:59.974 --> 00:45:01.894
Yeah. You know? Yeah.
00:44:59.974 --> 00:45:01.894
Extracting names.
00:45:01.894 --> 00:45:03.255
Extracting names from things.
00:45:03.255 --> 00:45:18.710
Like, I remember a very long time ago, Hilary Mason had this really cool blog post where she scraped, like, two hundred chocolate chip cookie recipes and then created, like, the average chocolate chip cookie recipe, which she then made.
00:45:18.710 --> 00:45:21.510
And, like, back then, that was, like, that was a lot of work.
00:45:21.510 --> 00:45:23.465
I was like, wow.
00:45:21.510 --> 00:45:23.465
That is so cool.
00:45:23.465 --> 00:45:27.784
And that is now something I could do today in, you know, like, thirty minutes worth of Or less than that.
00:45:27.784 --> 00:45:32.105
You just ask ChatGPT for a talk with university, and that's basically prevalent.
00:45:32.105 --> 00:45:33.105
Yeah.
00:45:34.025 --> 00:45:39.719
Ben Stancil mentioned something about this too where, like, and having to do with data work like support.
00:45:39.719 --> 00:45:50.914
If you have like thousands of support tickets, you know, as a data team you might have used to go in and try to analyze it and maybe like calculate a number from support tickets.
00:45:50.914 --> 00:45:52.434
He mentioned yeah.
00:45:52.434 --> 00:46:01.954
Like, nowadays, you might even just dump your support tickets into, like, ChatGPT or something and just ask, like, your question directly.
00:46:01.954 --> 00:46:05.059
Whereas before, we were trying to roll up numbers to try to kind of Yeah.
00:46:05.059 --> 00:46:07.300
You try to have to do some sort of like In between.
00:46:07.300 --> 00:46:07.619
Yeah.
00:46:07.619 --> 00:46:17.179
You have to like take the unstructured thing, and then, you know, run some models over it, maybe some classifiers, maybe some sentiment analysis, maybe you're like extracting specific signal from it.
00:46:17.585 --> 00:46:23.905
And then you get this like very, very limited structured representation of it, topic, you know, topic clustering or whatever.
00:46:23.905 --> 00:46:27.824
And then you like try to figure that out, figure out what's going on based on that.
00:46:27.824 --> 00:46:36.139
And and some of that actually is still very complementary I would say because, you know, LLMs are, you know, nondeterministic and they're not very precise.
00:46:36.139 --> 00:46:47.635
Especially for things that are like, if you're just dumping a bunch of unstructured data into them for things that are require, you know, numerical calculations and things, like marrying those two things can be challenging.
00:46:47.994 --> 00:46:53.994
But for things like, you know, what are the most common, you know, user complaints?
00:46:53.994 --> 00:46:57.155
I've done this in Hex, on Hex Data.
00:46:57.380 --> 00:47:01.380
And you can it's it's actually, like, very, very fast and very powerful to pull out that kind of signal.
00:47:01.380 --> 00:47:01.780
Yeah.
00:47:01.780 --> 00:47:08.019
Are you saying, like, user complaints you can get surprisingly far by putting a lot of the feedback in and just asking.
00:47:08.019 --> 00:47:08.820
Yeah. Yeah.
00:47:08.820 --> 00:47:11.275
Just get straight to the kind of where you're going.
00:47:11.275 --> 00:47:11.514
Yeah.
00:47:11.514 --> 00:47:19.434
And you probably want to move into the more structured well when you're getting to the point where you're like, trying to be like, okay, well, what fraction of users are asking about this?
00:47:19.434 --> 00:47:27.980
And, you know, you know, how does that compare to, like, time to resolve the tickets and and things like that, then you probably do want to start to shift into, like, a more classically structured data.
00:47:27.980 --> 00:47:32.380
But often you just want, like, imagine, like, someone reroll the support tickets, and they're just gonna give you the vibes.
00:47:32.380 --> 00:47:38.114
Like, that's super useful even if they can't tell you, oh, like, twenty five percent of the support tickets are about blah.
00:47:38.114 --> 00:47:38.675
Yeah. Yeah.
00:47:38.675 --> 00:47:42.994
I like I like it's like a alternative data science workflow.
00:47:42.994 --> 00:47:49.795
If you have this like, from like R for data science, you have this workflow that's like import, tidy, transform.
00:47:49.795 --> 00:47:52.570
I like the alternative, which is like copy paste. Yeah.
00:47:52.570 --> 00:47:56.650
Just try to get to Just give me the vibes, and then you, like, figure out then you can iterate.
00:47:56.650 --> 00:48:02.170
But I think I even find that with, like, some of my projects where I'm just like, hey, Claude, just Claude Code, just do this.
00:48:02.170 --> 00:48:06.125
Like, look at this transcript.
00:48:02.170 --> 00:48:06.125
Do this stuff to it.
00:48:06.125 --> 00:48:07.164
And then I'm like, okay.
00:48:07.164 --> 00:48:09.244
This this it feels like this has teeth.
00:48:09.244 --> 00:48:10.844
And then I'm like, okay. Now Claude code.
00:48:10.844 --> 00:48:14.605
Write the code that will do that for me, like, reliably and reproducibly.
00:48:14.605 --> 00:48:22.500
Like, that's such a just to be able to do that very, very quick initial prototype to say, like, is there anything here that I should bother exploring further?
00:48:22.500 --> 00:48:25.780
You have come from, you know, cofounder, your CTO.
00:48:25.780 --> 00:48:31.699
I'm really curious, like, what does success look for you look like for you, especially, you know, in the age of AI?
00:48:31.699 --> 00:48:34.300
Maybe how has that changed throughout your career?
00:48:34.875 --> 00:48:35.434
Yeah.
00:48:35.434 --> 00:48:46.715
I think starting a company has really forced me to like disentangle myself from classic indicators of success, if that makes sense.
00:48:46.715 --> 00:48:47.840
And there's a couple reasons for that.
00:48:47.840 --> 00:48:50.239
One is you'll just drive yourself absolutely insane.
00:48:50.239 --> 00:48:53.280
You're like, that company has how much revenue and they raise how much?
00:48:53.280 --> 00:48:56.079
Like, you just like, you just can't compare yourself.
00:48:56.079 --> 00:48:58.239
Like, a lot of us, you know, come up through school.
00:48:58.239 --> 00:49:09.155
You're basically like ruthlessly, you know, trained to optimize this like one metric and like compare yourself against all the other people to get into and like you just have to just let all of that go.
00:49:09.155 --> 00:49:15.034
Like, you just have to let the numbers go and and focus on, you know, what you can control.
00:49:15.829 --> 00:49:28.469
And I think the other thing about being a cofounder that I found is I found that the hardest part about being a cofounder is that you are, you know, the job changes every six months.
00:49:28.469 --> 00:49:34.684
And and so what that means is just as you're getting your feet under you, you have to do something different.
00:49:34.684 --> 00:49:38.364
And so you're always in this mode of like, I'm not good at the thing I'm supposed to be doing right now.
00:49:38.364 --> 00:49:39.724
I'm learning how to do this.
00:49:39.724 --> 00:49:52.929
And for someone who, you know, generally, you know, had relatively successful career prior to that, you're like, you're used to operating in areas where you're good and you're strong and things that you were hired for and you interviewed well for and you're like, great at software engineering.
00:49:52.929 --> 00:49:55.010
And being a cofounder is just like not like that.
00:49:55.010 --> 00:50:01.905
And so all of these like classic ways of being like, I am successful, just kind of go out the window.
00:50:01.945 --> 00:50:05.465
And I think for me, I just kind of keep coming back to a couple different things.
00:50:05.465 --> 00:50:11.344
One is, are we building something that, you know, is doing good for the world, that is a useful contribution?
00:50:12.110 --> 00:50:15.150
And then am I enjoying what I'm doing?
00:50:15.150 --> 00:50:16.829
Is this a good use of my time?
00:50:16.829 --> 00:50:21.469
And as long as I feel like and you know, you don't have to enjoy everything that you're doing, but like sort of in the macro.
00:50:21.469 --> 00:50:29.505
And as long as you feel like those two things are are happening, then I think that's like a pretty it's actually I feel like it's quite rare to really be nailing both of those things.
00:50:29.505 --> 00:50:34.224
And so that's kind of how I've been thinking about my own personal career success.
00:50:34.224 --> 00:50:39.480
Do you feel like there was one pivot that was like, this is the hardest thing or this is the easiest thing?
00:50:39.480 --> 00:50:43.360
Was there one pivot that felt like it hit harder than the rest?
00:50:45.480 --> 00:50:46.280
I don't know.
00:50:46.280 --> 00:50:54.480
I think in some ways, I kind of, from the beginning of Hex, had to make that pivot even just to start the company.
00:50:55.474 --> 00:51:04.355
And it was a bit of a sort of mental hurdle to get over because prior to that, my career had been extremely linear.
00:51:04.355 --> 00:51:15.519
It was like IC and then manager and then manager of managers and then like, you know, he's just kind of like walking up that and taking a step away from that.
00:51:15.519 --> 00:51:21.840
And and also candidly, like, you know, I I went into Hex being like, I don't know if this is gonna work. Right?
00:51:21.840 --> 00:51:24.454
You know, most startups don't work.
00:51:24.454 --> 00:51:32.574
We may all be back on the job market in a year or two and that would feel like failure.
00:51:33.735 --> 00:51:39.079
But I think getting kind of getting past that and being like, is, you know, is that the worst thing in the world?
00:51:39.079 --> 00:51:41.159
Like, we try something and it doesn't work out?
00:51:41.159 --> 00:51:44.440
Like, maybe you're like slightly embarrassed?
00:51:44.440 --> 00:51:44.920
Like, I don't know.
00:51:44.920 --> 00:51:47.239
Like what like what what is the actual negative outcome there?
00:51:47.239 --> 00:51:49.880
And kind of pushing through that feeling of like, what if it fails?
00:51:49.880 --> 00:51:51.360
Well, what if it does fail?
00:51:51.614 --> 00:51:53.295
So what? We all go, like, we're all software engineers.
00:51:53.295 --> 00:51:54.894
We all go get jobs.
00:51:54.894 --> 00:52:04.534
And and I think that was probably the biggest mindset change and even just like jumping off and then, you know, having to kind of repeatedly remind myself of that over the years.
00:52:04.699 --> 00:52:09.900
One of the frameworks I found really useful is, like, this sort of idea of maximizing versus satisficing.
00:52:09.900 --> 00:52:14.699
Like, it's I think I'll feel like, you know, data people, people have, like, had good grades at school.
00:52:14.699 --> 00:52:18.820
Like, your natural thing is to always, like, find the best, like, maximize.
00:52:18.914 --> 00:52:22.275
Like, I find this a lot when I'm trying to, like, buy a hotel somewhere.
00:52:22.275 --> 00:52:28.275
Like, I do so much comparison shopping because I'm trying to find, like, the best combination of location and price Hotel matches.
00:52:28.275 --> 00:52:28.835
Facilities.
00:52:28.835 --> 00:52:30.195
Is that Exactly.
00:52:30.195 --> 00:52:39.750
Whereas the this this alternative approach of, like, satisficing of just finding the first thing that's gonna be, like, good enough, like, that's such a powerful, like, mindset change.
00:52:39.750 --> 00:52:48.554
Like, you don't have to be like, not everything in your life has to be the best, just like finding the things that, like, make you where you'll be, like, happy and satisfied.
00:52:48.554 --> 00:52:59.155
Like, that's There's an interesting story of, like, the, you know, step one to two feels hard, but what was way harder was step zero to one.
00:52:59.355 --> 00:53:07.880
And, you know, what what mindset mindset shifts do you have to do to even get to a spot where you're at zero versus, you know, continuing to just go like, oh, well, I'm at one.
00:53:07.880 --> 00:53:09.400
I can go to two.
00:53:07.880 --> 00:53:09.400
I can go to three.
00:53:09.400 --> 00:53:11.320
It's like the same with, like, growing a team.
00:53:11.320 --> 00:53:12.519
Like, the first person.
00:53:12.519 --> 00:53:15.014
Like, going from zero to one is really hard.
00:53:15.014 --> 00:53:20.454
From one to two, that's not so bad, and then it just gets gonna get on the downhill slope a little bit.
00:53:20.454 --> 00:53:21.014
Yeah.
00:53:21.014 --> 00:53:34.599
This is a pretty common piece of coaching that I I give to engineers, which is it's it's basically telling them, like giving them permission not to be miserable.
00:53:34.599 --> 00:53:35.719
I love that.
00:53:35.719 --> 00:53:35.800
Yeah.
00:53:35.800 --> 00:53:40.840
They they often sometimes in in engineering sometimes, again, sometimes you gotta do what you gotta do.
00:53:40.840 --> 00:53:49.675
But, you know, sometimes someone like, we had an engineer who moved into an engineering manager role and was miserable.
00:53:49.675 --> 00:53:51.394
They hated it.
00:53:51.594 --> 00:53:54.570
And I was like, you can go back.
00:53:54.570 --> 00:54:04.050
Like, we actually intentionally have set up our org structure such that like senior engineer and engineering manager are the same, no pay difference, they're just the same level, just a different title, different role.
00:54:04.250 --> 00:54:20.735
And coaching people through their first moment of I tried it and I didn't like it and I didn't like make it all the way to something that feels like a success moment, but that's okay.
00:54:20.735 --> 00:54:26.929
I don't have to get there if I'm really unhappy with what I'm doing. I can just stop.
00:54:27.610 --> 00:54:44.485
And that, especially like we've been talking about for for people, high achievers, that can feel like a failure because sometimes you're in a you're in a spot where you're actually not doing a great job at the thing that you're supposed to be doing, and you feel like you have to push through to get to a point where like you've made it.
00:54:44.485 --> 00:54:47.204
But I don't know. We only you only live once. Right?
00:54:47.204 --> 00:54:53.405
Like, you know, what you do with your time matters, and there's only a certain amount of like pushing through.
00:54:53.820 --> 00:54:58.219
And and it really depends on like what the underlying reason is that you're not happy.
00:54:58.219 --> 00:55:03.900
There's like lots of different reasons, but usually I see this in a case where someone just like, they don't like the role.
00:55:03.900 --> 00:55:05.019
And that's not a failure.
00:55:05.019 --> 00:55:07.675
That's just like you would rather be doing something different.
00:55:07.675 --> 00:55:14.954
Obviously, like, there's differences where, you know, you just have you sometimes you do have to power through, like, tough times in the company and you just have to, like, make it out the other side.
00:55:14.954 --> 00:55:16.394
But that those are temporary things.
00:55:16.394 --> 00:55:18.795
Like, the role doesn't change.
00:55:16.394 --> 00:55:18.795
The role is still the role.
00:55:18.795 --> 00:55:20.559
And so if you don't like it, don't do it.
00:55:20.559 --> 00:55:28.559
You mentioned like the kind of uncertainty of starting a startup reflected in this person's shift to engineering manager.
00:55:28.559 --> 00:55:35.159
Like, mentioned, you know, the the fear of starting a startup and thinking like what if it fails?
00:55:35.164 --> 00:55:37.724
And then being able to say, like, so what?
00:55:37.724 --> 00:55:42.764
Like, I tried it, and I'll still have other things to work on all I've learned and growed.
00:55:42.764 --> 00:55:48.045
And I think that, like, mindset shift between, like, this experiment failed versus I'm a failure.
00:55:48.045 --> 00:55:50.329
Like, that's Yeah. Really Yeah.
00:55:50.329 --> 00:55:51.050
Really big difference.
00:55:51.050 --> 00:55:58.969
And to coach an engineer on that, that, like, jumping from to being an engineering manager is a similar, like, experiment as starting a company.
00:55:58.969 --> 00:56:11.414
Because it makes sense for that person, maybe it is probably as big and as scary of a jump, I'd imagine, to try this like new way of operating and that you might not know how you'll do.
00:56:11.414 --> 00:56:22.500
So it seems like honestly so, has gotta be so relieving or helpful to know, like, it's an experiment and you can switch back.
00:56:22.500 --> 00:56:22.739
Yeah.
00:56:22.739 --> 00:56:26.980
And now that we've had we've had this happen in various parts of the organization a couple times now.
00:56:26.980 --> 00:56:30.965
And again, it creates this permission structure for people to try things.
00:56:31.164 --> 00:56:37.485
And that obviously has, like, huge positive externalities when people are, like, less afraid to go out on a limb and try something new.
00:56:37.485 --> 00:56:42.764
I mean, again, being the first person to go from being an engineering manager back to senior engineer, it's, like, super scary.
00:56:42.764 --> 00:56:45.489
Like, if you're the fifth one, like, That that's a good point.
00:56:45.489 --> 00:56:48.369
It kinda, like, snowballs, I guess. Yeah. Yeah.
00:56:48.369 --> 00:56:57.010
But it does feel like right now, like, like, creating an engineering culture where people are excited to try experiments even though they're not gonna succeed.
00:56:57.010 --> 00:57:02.614
Like, that's like, particularly today with AI, just so, so important. Yeah. Okay.
00:57:02.614 --> 00:57:06.695
We went through Hex, life as a CTO.
00:57:06.695 --> 00:57:08.855
How do you like to unwind at the end of the day?
00:57:08.855 --> 00:57:10.414
What's your move?
00:57:10.599 --> 00:57:12.519
Mean, we do a lot of the normal stuff.
00:57:12.519 --> 00:57:19.000
I also have a a toddler, so not everything is unwinding at the end Winding in different ways.
00:57:19.000 --> 00:57:20.079
Yeah.
00:57:20.119 --> 00:57:33.934
Once he's in bed though, actually my husband my husband and I have been we've been playing through together all of the Soulsborne games, which might be a little bit weird for an unwind session.
00:57:33.934 --> 00:57:38.469
But it's like it's a really good way to like, you know, release some tension, you know.
00:57:38.469 --> 00:57:43.909
We gotta, like, start like, I hadn't played them before, but, I got started with Elden Ring, and then we, like, played all the way through Elden Ring.
00:57:43.909 --> 00:57:47.030
And then the and the rule is when you die, you pass the controller.
00:57:47.030 --> 00:57:48.230
So you get a little bit of a break.
00:57:48.230 --> 00:57:50.309
Like, die, you pass it, and then you get another turn.
00:57:50.309 --> 00:57:55.925
Do you do you have different specializations or jobs in Elden Ring, would you say?
00:57:53.445 --> 00:57:55.925
You and your husband, like?
00:57:55.925 --> 00:57:56.405
Yeah.
00:57:56.405 --> 00:58:01.045
I will say he is much better at sort of like learning the timing of the boss mechanics.
00:58:01.045 --> 00:58:03.764
So he's typically like he's he's gonna be a little better than that.
00:58:03.764 --> 00:58:09.219
I tend to be a little bit better at the like mobs because I'm less reckless than he is.
00:58:09.219 --> 00:58:11.619
He just, like, charges in and just get, like, totally annihilated.
00:58:11.619 --> 00:58:15.739
And I'm little more strategic in how I, like, do the placement of the yeah.
00:58:16.179 --> 00:58:26.304
My partner and I played a lot of Elden Ring, and I am very much so that avoid the mobs, run away, like, learning that you don't have to fight everyone, didn't realize that was an option in Elden Ring.
00:58:26.304 --> 00:58:27.985
Life changing. Life changing.
00:58:27.985 --> 00:58:30.905
It's an option in real life too. Yeah.
00:58:32.065 --> 00:58:41.389
I guess for the whole table of Elden Ring players, when you're playing with a person, how is this a collaborative peaceful act or Act of violence.
00:58:41.389 --> 00:58:42.750
You know what I'm saying?
00:58:42.750 --> 00:58:48.204
We actually stopped playing real co op games together because they got too intense.
00:58:48.204 --> 00:58:52.364
And we were like or it's just like we have just different play styles.
00:58:52.364 --> 00:58:58.684
Again, like, his is very, like, you know, just like rushed straight at it, and mine's a little bit like more sort of like careful exploratory.
00:58:58.684 --> 00:59:00.789
And so we were getting really mad at each other.
00:59:00.789 --> 00:59:05.309
Like, tried to play Baldur's Gate co op together and just like that did not go well.
00:59:05.429 --> 00:59:07.989
And so I actually think that this this model works a lot better.
00:59:07.989 --> 00:59:10.869
Like, one person is in control at any given time.
00:59:10.869 --> 00:59:12.150
I feel like that's a hot take.
00:59:12.150 --> 00:59:19.144
I love Baldur's Gate because split screen, you can all almost have a co adventure, like, or completely different adventure.
00:59:19.144 --> 00:59:20.664
Yeah. Oh, you can split off.
00:59:20.664 --> 00:59:26.465
You're saying you took the essence of playing separately, but within a cooperative game Yeah.
00:59:26.744 --> 00:59:27.065
Pretty much.
00:59:27.065 --> 00:59:28.105
You're like, see you.
00:59:28.105 --> 00:59:30.389
And then the screen just splits and you never reunite.
00:59:30.389 --> 00:59:30.630
Yeah.
00:59:30.630 --> 00:59:33.429
Sometimes. And then it's like, oh no, I'm dying.
00:59:33.429 --> 00:59:35.110
Can you come help? Yeah. Yeah.
00:59:35.110 --> 00:59:37.429
And then she'll pop in and have good Yeah.
00:59:37.429 --> 00:59:39.510
Love everybody for me and That's incredible.
00:59:39.510 --> 00:59:40.550
It's a good life.
00:59:40.550 --> 00:59:42.914
Caitlin, really appreciate you coming on the Test Set.
00:59:42.914 --> 00:59:44.355
I Thank you so much for having me.
00:59:44.355 --> 00:59:46.914
Yeah. No. It's okay. I feel like so much to think about.
00:59:46.914 --> 00:59:56.590
I I honestly I hate to admit it, but I feel like I have to I hadn't learned about Context Studio, but it is gonna live rent free in my head for the next You should check it out.
00:59:56.590 --> 00:59:57.469
Probably all my life.
00:59:57.469 --> 00:59:58.829
We'll hook you up.
00:59:58.829 --> 01:00:01.389
I I would love to I mean, always showing Hex to people.
01:00:01.389 --> 01:00:22.175
I feel like I don't actually, what you said about Context Studio really kind of blew my mind because I've I do like a lot of software engineering on data tools, but that strikes me as such a unique maybe not totally unique problem across a lot of domains, but for data people that need to like curate and stamp and endorse.
01:00:22.175 --> 01:00:28.409
It's honestly, like this moment in some ways is like a match made in head of heaven for data people for this workflow.
01:00:28.409 --> 01:00:39.474
Because this is the workflow that data teams have been doing forever, but it's just been so painful and laborious for, you know, relatively little impact.
01:00:39.474 --> 01:00:46.675
Like if you if you think or I mean, obviously there's impact, but like the amount of impact you get out of like all of this labor just sometimes doesn't feel worth it.
01:00:46.675 --> 01:00:56.309
Like, I don't know if you all remember, you know, Ari, I'm sure you remember like when semantic models first kind of became a thing, we just didn't see a ton of adoption of semantic models.
01:00:56.309 --> 01:01:02.389
And it wasn't because people didn't want them, it was just because it was so painful and took so much time to to build them out.
01:01:02.389 --> 01:01:06.974
And now to and and at the end, it's like, only two people are gonna be building your dashboards.
01:01:06.974 --> 01:01:09.775
Right? And now today, that's like totally been inverted.
01:01:09.775 --> 01:01:23.380
Not only can you take, you know, constructs like semantic models and make them available to everybody in the organization a way they can use, but actually also you can do the curation and you can do the governance much much more easily with sort of agents yourself.
01:01:23.380 --> 01:01:29.940
It is a pretty incredible that, like, both the cost has decreased dramatically and the impact has increased dramatically.
01:01:29.940 --> 01:01:32.739
Like, it's pretty unusual to get both of those at the same time.
01:01:32.739 --> 01:01:33.414
It's pretty cool.
01:01:33.414 --> 01:01:36.735
Yeah. Totally. Yeah.
01:01:33.414 --> 01:01:36.735
So a lot to check out.
01:01:37.255 --> 01:01:39.175
Yeah. Can't can't recommend Hex enough.
01:01:39.175 --> 01:01:50.369
I feel like also giving software engineers permission to, like, try things and fail, I think, is so relevant, especially with AI where there's a lot of experimentation going on.
01:01:50.369 --> 01:01:54.289
And then I'm putting in my pocket Elden Ring co op co op strategies.
01:01:54.289 --> 01:01:58.650
I'm gonna find peaceful ways to to play with my partner.
01:01:58.849 --> 01:02:00.929
But, yeah, thanks so much for coming on.
01:02:00.929 --> 01:02:03.454
It's been so nice and been great having you.
01:02:03.454 --> 01:02:04.335
Awesome. It's been fun.
01:02:04.335 --> 01:02:06.375
Yeah. Thank you so much.
01:02:07.215 --> 01:02:13.775
The Test Set is a production of Posit PBC, an open source and enterprise tooling data science software company.
01:02:13.775 --> 01:02:17.969
This episode was produced in collaboration with creative studio, Adjy.
01:02:17.969 --> 01:02:24.170
For more episodes, visit the Test Set dot co or find us on your favorite podcast platform.