WEBVTT
1
00:00:00.031 --> 00:00:24.719
I use AI for research, like to run experiments. But I had this one scary moment. It was like a very long running 12 hour task that it was doing for me. And I just happened to be scanning through and I expanded one of the thinking blocks. And I saw this block of thinking text that was like, oh, metric is 0.0003 off of the cutoff. It was totally undermining the scientific
2
00:00:24.719 --> 00:00:49.913
integrity of the entire experiment, but like dressed up in this language of I'm being fair, I'm doing the right thing. Can you talk about what an invisible failure is? Because I see that everywhere. And I know you talk about a lot. So I'm just curious if you can explain it.
3
00:00:50.116 --> 00:01:13.032
An invisible failure in AI is when something breaks, something goes wrong, something the user isn't satisfied or gets the wrong information. Something breaks down in an AI interaction, but there's no alert. There's no crash. There's no error message. And it's not even something that the user recognizes. They don't give feedback. They don't escalate.
4
00:01:13.032 --> 00:01:33.552
So it's something that goes wrong, but leaves no visible sign or alert or trigger or flag in any dashboard or any kind of our systems that we use to track. And of course, what we found in our research was that the vast majority, 79 percent of failures that are happening in AI interactions are staying invisible.
5
00:01:33.552 --> 00:01:58.460
which means that they're not being, you know, the user isn't catching them, they're not being logged in some dashboard. And the danger of that, of course, is what we can't fix, what we can't detect. We can't hill climb if we don't know what's breaking and what good even means in these interactions. And so we've tried a lot through our research and our product to, you know, be able to now finally see what has been invisible and see these interactional failures that are happening that are letting users down.
6
00:01:58.460 --> 00:02:04.265
Presumably now you're talking about on the customer facing side, but given how much
7
00:02:04.754 --> 00:02:32.328
how many failures there tend to be just as you use it as an individual. Are you surprised that the rate is so high that there aren't harnesses or other guardrails to catch it just in general? I am surprised as a, so I'm both surprised and not surprised as a user myself. I, I myself have go through these phases. I feel like we're sometimes I'm just amazed at how good AI is and how smart it is and how much, how well it does the task that I want it to.
8
00:02:32.328 --> 00:02:51.734
And then other times, you know, it's like this is the dumbest thing in the world. How could it get that wrong? And then I wonder, you know, how many things did I even not catch? And when it's you know, when it when I feel like it's so smart, is that because I'm being really smart? And when I feel like it's being so dumb, is that because I'm using it wrong or is there something about the model?
9
00:02:51.734 --> 00:03:14.262
And so I think that's kind of what's the great story. And as we really started to dig into this in the research as well, is that a single AI model, even used for the same use case, it can just have dramatically different success or failure just based on who happens to be in the room, what the user happens to do, what mindset they happen to bring to that interaction.
10
00:03:14.262 --> 00:03:43.523
And I've certainly seen that in myself, like when I treat it maybe like a vending machine or just hope that like a lottery and I hope it gives me something amazing. Like sometimes it works and sometimes I'm disappointed. And then sometimes when I put in that really hard work, I'm able to like break through on something. And those are often the most frustrating AI interactions that I have where I'm like expressing my frustration in the chat and yet at the same time, the most frustrating interactions are often the ones where I achieve something, I achieve the things with the greatest complexity.
11
00:03:43.523 --> 00:04:08.228
I have the greatest creative breakthroughs. And so it's really hard, I think, for us to speak in this sort of cut and dry way about success and failure. We really have to look at these interactions that are happening between us and the AI, between different styles of users and the AI, and how that has dramatically different impacts on how successful it is. Can you talk through some of those styles and how they impact people?
12
00:04:08.802 --> 00:04:28.158
Yeah. So just for context for folks, we tried to study this in a quantitative way, do a large scale study of actual real production interactions that users had with AI systems. So these are not test interactions. These are real scenarios where users are trying to get some kind of work done. And this spanned domains like coding,
13
00:04:28.158 --> 00:04:46.045
all the way to creative writing and emotional support. So the whole gamut of what people are using AI for and day-to-day. And this was a study done specifically on ChatGPT logs as a public dataset. And what we wanted to do is
14
00:04:46.045 --> 00:05:14.817
try to put some more numbers and rigor behind this kind of this anecdotal thing that we're discussing that's like, man, you know, some people are having a lot of success and other people are not with AI. Like what what what is that? How does that come down to? What's the difference? And for this study, we really looked at how, you know, keeping the model fixed, keeping the product and the user interface fixed and just looked at what do the users do in a session? What behaviors do they exhibit? What do how do they treat the model? What are they you know, how are they trying to do what they're trying to get done?
15
00:05:14.817 --> 00:05:32.165
And I think that the dramatic thing is there is really a big gap and a big difference in how just kind of everyday people approach these tools. And you see a big difference in what they're able to achieve based on what behaviors they exhibit. So kind of roughly speaking.
16
00:05:32.165 --> 00:05:54.001
There's the biggest gap that we found is in users who treat it a little bit more from a delegative perspective, like thinking about how they kind of delegate tasks to AI versus ones that really engage with a more augmentative mindset. So these are users who are like iterating and pushing back and challenging and bringing their own context and critical thinking.
17
00:05:54.001 --> 00:06:21.777
And what you find is that that second group, which is a tiny percentage, by the way, it's about 3% of really high fluency, high augmentation users. That small percentage are able to achieve significantly more complicated tasks with AI. And there's this interesting paradox where if you just look at those augment of high fluency users, they're actually failing more. Like there's more visible failures. There's more things that don't work.
18
00:06:21.777 --> 00:06:37.522
But at the same time, they're also trying to achieve significantly more complicated things, on average, twice as complicated, and they're succeeding more at those tasks. And so it's that behavior that they bring and, frankly, the hard work that they're putting into that interaction.
19
00:06:37.522 --> 00:06:59.864
that allows them to achieve just different outcomes compared to this vast majority of users in this study who were a little bit more delegative, more treating it like a slot machine, seeing what came out, maybe walking away if they didn't like it. In either of those cases, how do you define right and wrong in a non-objective output scenario?
20
00:07:00.421 --> 00:07:20.333
That's the real question, right? Is what does good mean? But I think this is like what I've struggled with since I first started trying to build products on top of modern LLMs. So my kind of first cutting my teeth on this was I was VP of AI at BetterUp and we're building an AI coach. So you have now an LLM that is going to
21
00:07:20.333 --> 00:07:49.426
hopefully do so many of these things, like supporting people, helping them with self-reflection and self-insight and motivation and learning and growth. And it's this phenomenal opportunity that now, you know, you're shipping a conversation that you can kind of ship at an incredible scale and just help people anytime a day, anywhere. And so how do you define, you know, you're shipping AI conversations now that are doing coaching. How do you define what a good coaching session is when you've deployed an AI coach? How do you measure that?
22
00:07:49.426 --> 00:08:14.165
And it's hard to pin down. And so the obvious starting point is, well, you know, what does it mean to be a good human coach? And that's also hard to pin down. But there's frameworks, there's certifications, there's competencies. And so, of course, like the first thing that we tried was, well, we want to make sure that the AI coach that we're shipping is like a really good coach from from kind of an expert coach's opinion. Yeah.
23
00:08:14.165 --> 00:08:40.456
And so we had coaches in the loop training it. We did evals that were based on all these competencies. And there's this one moment that I had where we've really invested in a new coach. We're doing this pilot and we're about to ship this new coach. And they would really like it was significantly better at coaching as judged by expert human coaches. And so from the objective metric of what does good coaching look like, this was a way better model, way better system.
24
00:08:40.456 --> 00:09:07.523
But when we A-B tested it, which I thought was like a pro forma thing, I'm going to be able to measure how much better this thing is. Users liked it significantly less. The engagement was way lower. What's your North Star? Is it how objectively good the coaching is? Is it whether users like it? Neither of those really by itself can be the North Star. If you're just optimizing for user engagement, you'll get sick of fancy. You won't like a good coach that should be difficult. Sometimes there should be friction in those experiences.
25
00:09:07.523 --> 00:09:24.770
But if you're just optimizing for the standard human definition of what a good coach is, well, you're shipping something that people don't actually want to use. You're shipping something that's like an AI medium, but now you've used this human analog and tried to retrofit it. And so the real question to me, it's how do you determine what good is?
26
00:09:24.770 --> 00:09:51.145
How do you measure that? And you can't rely on proxies from the human world. You can't just rely on lagging metrics like engagement and re-engagement. You have to actually look at what's happening in these sessions. What job is the user hiring this AI to do for them? And then evaluate, like, you know, to what extent is it fulfilling those goals? In non-consumer facing, but in internal tools
27
00:09:51.652 --> 00:10:06.400
that companies are now trying to build, do you see that same bifurcation happening in terms of the inability to define good? Once you get outside of the workflow stuff, which I think is probably easier.
28
00:10:06.991 --> 00:10:22.972
I think the best place to maybe look at this is in AI coding. So most highly used internal application and the hottest area for AI doing actual work output.
29
00:10:22.972 --> 00:10:50.900
So we did a study on trying to understand, like, what is a token actually buy you when you're doing AI coding? And how do you actually measure, if you look at a group of 100, 200, 300 users that are all using Cloud Code, how are they all getting the same productivity gains from it? How are some getting more gains? What distinguishes those who are seeing how more productivity gains from those who aren't? And what can each group learn from each other?
30
00:10:50.900 --> 00:11:19.840
And I think what's, again, so difficult about that question is theoretically for AI coding, you should have more objective metrics for output, for work output, because, you know, you can look at PRs, you can look at lines of code drafted, you can look at their crude, but they are, you know, there are just raw, countable things about engineering throughput and productivity. But I think what's so, so difficult to measure about that is that
31
00:11:19.840 --> 00:11:37.846
the way that people actually use AI coding agents, there's just like any AI product, there's this really large variety of different usage styles. And on one end of the spectrum, let's say it's people who really treated like a code writing tool.
32
00:11:37.846 --> 00:11:53.945
And so you have a very detailed spec. You tell it exactly what you want it to write. And in that way, it is a machine that is more directly sort of, you can see it as like, this is a code writing utility. And you can measure on how much code did it write? Did it pass all the tests? Did it get merged, et cetera?
33
00:11:54.383 --> 00:12:15.798
But if you look at how a lot of how I use Cloud Code, that way up the spectrum of, I really treat it like a collaborator. I treat it like a teammate, like a peer. Like I expect it to know things about our code base, know things about our product vision, know things about our goals and our users. And I expect it to help me think through things and be creative and iterate.
34
00:12:15.798 --> 00:12:44.367
And in that world, you can't just measure it based off of like my utility based off of how much code is written, because that doesn't you know, how do you capture maybe the inspiration that I have from a good idea or from from something that it showed me that I wouldn't have been able to, you know, to iterate on as quickly just by myself. So even in a domain that's technically verifiable and has raw kind of output metrics like coding, even there, I think the hardest question is how do you actually measure value?
35
00:12:44.671 --> 00:13:11.317
There's a presumption, I guess, behind all of that, that the model, even if you are using it consistently, is reacting consistently back to you. And have you found that to be the case that within a certain model setting that your style is consistently producing similar outputs? That is, again, something that is hard to nail down, but I think anybody who's
36
00:13:11.317 --> 00:13:41.303
used AI for something now and tried to do the same thing a couple of months ago, it's very clear that everything is changing very fast about these models and about the harnesses. And a lot of that is happening in ways that we as users don't see. So we don't have visibility into what memories ChatGPT is storing about us or what cloud code is creating based on the work.
37
00:13:41.303 --> 00:14:05.283
We don't know how we can't even, you know, a lot of these models, more and more of the tokens are going to thinking. That's not even something that the user can see that even if we wanted to read all that, that we only get summaries of that. And so more and these models, the harnesses, everything is changing behind the scenes. And that's just another variable in this very difficult world where you have a model that keeps changing.
38
00:14:05.283 --> 00:14:28.047
A user who's also changing and has different behaviors and different mindsets, different moods they might be in at a given day. And the product of what happens in an AI session is somehow like this chemical reaction at a given moment between those two. And we did try to study this as well. And because I saw just how much token usage is increasing.
39
00:14:28.047 --> 00:14:56.701
And that's one of the most common things a lot of organizations measure in general is like, well, it's hard to measure what the value is that we're getting out of it, but we can at least measure how much we're using this thing. Yeah. And what's so interesting is that even for the exact same task, the exact same user kind of doing the same behaviors, you've seen a steady increase in how many tokens are being used to fulfill that task. Because the models are just in general, each successive generation becoming more token hungry
40
00:14:56.701 --> 00:15:22.891
and putting more of that into the thinking. I read that you coined this term tokenflation. Yeah. Can you just define it real quickly? We're just trying to find a word to describe this phenomenon that we found in the research that a token today buys you less raw work output than a token yesterday. And so it's the same as inflation. You know, if you go to a grocery store, how many eggs and how many apples can you buy for a dollar?
41
00:15:22.891 --> 00:15:37.893
And the consumer price index tracks that over time. So we did a study doing the exact same thing for tokens. And again, looking at AI coding outputs. So looking at these engineering outputs, which we've talked about, they're imperfect. They don't capture everything.
42
00:15:37.893 --> 00:16:00.050
But they are a good raw kind of unimpeachable just metric of like how much code is being written, how many PRs are being merged. And so if we look at kind of a broad scale, so we looked at February to April, Opus 4.6, kind of the same cohort of about 115 users. So we looked at all of the work that they were doing, all of their sessions on Cloud Code.
43
00:16:00.050 --> 00:16:15.355
And we just tried to study like, well, what how is the purchasing power of a single token changing over time? So our basket of goods here is things like, you know, how much how many tokens does it take to buy a PR? How many tokens does it take to buy one line of code that's drafted?
44
00:16:15.355 --> 00:16:40.685
And what we found was that there's kind of similar to inflation. There's just been this steady decrease where, you know, a 10,000 token session in February, it bought about a third of a PR, eight files touched, about 600 lines of code drafted. By April, the same amount of tokens, 10,000 tokens was buying a sixth of a PR, four files touched, 90 lines of code, about.
45
00:16:40.685 --> 00:17:01.171
And is the cost of the token consistent or is it going down or how is that impacted there? So that's why we focus just on what is a token buy rather than just what is a dollar buy, because the model providers, they change their pricing. Also, some users are on subscription plans where there's like certain.
46
00:17:01.171 --> 00:17:28.222
rebates or the economics of how that translates to dollars is it's changes a lot. And I think that's just another thing for anybody who's trying to like project future spend and think about ROI should keep in mind is like the cost of a token is changing. How many tokens are required to do a piece of work is changing. Like all of these variables are changing and you can't rely on them. Going back to your core statement, why would the same token produce less output even though they're thinking is the output
47
00:17:28.222 --> 00:17:52.943
more calculated and defined, or is it just the way the models are now being designed? Because that's counterintuitive, I would think. Yeah. So, and I can't answer that definitively. So what we can tell is where are the extra tokens going? So you can track that to, you know, earlier it was like most of the tokens would actually go to tool calls to the actual code generation task.
48
00:17:52.943 --> 00:18:08.840
Now, more of those tokens proportionally are going into thinking and explanation to the user. So you can kind of say, okay, the role of the token is changing a little bit. And then you think, well, what is that? What impact is that having? What is that buying us or what's the trade that you get from that?
49
00:18:08.840 --> 00:18:34.996
And, you know, I think that the two obvious kind of answers or kind of explanations would be one, users are able to do more complicated things now. And so they're just doing more complicated things. And that's why it costs more tokens. The data, the study that we did, it didn't we didn't find evidence for that. Users were kind of doing the same things that the token was increasing. So then the second the second explanation would be what's buying quality.
50
00:18:34.996 --> 00:18:52.546
Like your more thinking means that the line of code that it writes is better. Maybe it didn't need 600 lines of code. It only needed 90. And because of extra thinking, it was able to find the right 90. And there is some evidence for this. So we studied code survivability. So every line of code that's written in a session
51
00:18:52.546 --> 00:19:19.192
how long does it survive in the code base? How quickly is it overwritten or deleted or changed as kind of a signal of what's the kind of durable value of that code. That's an interesting metric, yeah. And if we look at just like kind of seven days survivability of the line of code, it goes from 90% in February to 95% in April. So there is an increase there of 5% or roughly half the gap. I can't make a value judgment about whether that was worth it.
52
00:19:19.192 --> 00:19:49.162
But I think what we're trying to do and just trying to push, I think, the industry towards is we have to be able to measure the trade and get better at measuring what do we get when we spend a token? Because as more and more of our costs are going to AI to make really smart decisions about how to leverage that spend and where can we get the most out of it, like unless we're able to measure what the token is buying us and where it buys us more value than in other contexts, we're not going to be able to make the decisions we need to make.
53
00:19:49.162 --> 00:20:11.977
And what metric do you look at internally or then do you recommend other people look at? I, again, this is, I wish I had like the perfect set of metrics. Well, there's all these startups, right, that are starting now to measure and do this. And, you know, it feels like every couple of weeks somebody else is creating a layer of cost management. But I don't know what metric they're looking at though.
54
00:20:11.977 --> 00:20:35.956
But the hard part for me is like, it's easy to track cost relatively. Like we think about ROI, the investment part of that equation is easy to track. It's the return part that we should really be focused on. And I think that's where the most of the time the answer tends to be quite business specific. It depends on your strategy as a business, right? It depends on what you actually care about.
55
00:20:35.956 --> 00:20:57.134
What environment you're operating in. And so for my team, there's two things that I pay the most attention to. One is just what percentage of our token spend and AI spend in general is going towards use cases that I consider like compounding.
56
00:20:57.134 --> 00:21:24.843
So, where it's not just a transactional, like getting something done, it's like a token spent here improves something else later, like something accrues because of it. And that to me is like a, just a kind of a rough value signal of like, how much are we leveraging the spend in a way that we're like, where something is compounding and growing rather than it's just kind of a spend that we have to stomach and we're like cost of doing business.
57
00:21:24.843 --> 00:21:49.126
And the other thing that I really track is how much not just helping with an individual's productivity, like I used to do XYZ task and it took me two days. Now, because of AI, I can do it in one day or one hour. I actually pay a lot of attention to what are our systems and feedback loops that we have where people now can...
58
00:21:49.126 --> 00:22:10.895
not just do something faster, but do something that they just fundamentally previously could not do. They can now play a role that they were not able to play. So for example, for me, I've been an engineer in the past, but I'm not a really experienced front-end engineer. I get really nervous with backend data migrations, all that kind of stuff.
59
00:22:10.895 --> 00:22:38.485
But I have really great skill files in our repo that have been curated and created by the experts on my team that mean that my Cloud Code can just do a really good job of following our code bases conventions, following our design principles, and following the right workflows where now I can take on meatier features that I was not able to do previously. And so that's not just an example of it saving me time. It's just an example of I now am capable
60
00:22:38.485 --> 00:23:02.853
of doing things that I wasn't capable of doing before. I talk a lot about how we need to get better measurements for what's the actual value that we're getting from AI. And I'm still trying to figure that out for our business. But I think those are the things that at a meta level, I think, are like, how do you make sure you're funneling that spend and that energy into the most productive areas?
61
00:23:04.287 --> 00:23:29.617
So you mentioned these different types of AI users and in a company, oftentimes different personalities are what drives the success of the company. We come together and we all come with different points of view. Do you need all those different types of AI users to get to the right outcome or are six or five of them fundamentally wrong and we need to do better training internally to get to the right outcome?
62
00:23:30.562 --> 00:23:53.579
I think both can and should be true. And let me try to be nuanced, but also succinct, which is if you look at, you know, if you look just at the data at, you know, power users of AI. Again, let's look at cloud code as an example, like modern agentic system. The most productive power users, they actually don't have just one way of interacting with AI. Yeah.
63
00:23:53.579 --> 00:24:22.722
they will change how they're interacting with AI based on what they're trying to accomplish in that session. So, you know, big meaty tasks where you're really trying to work through something where there's strategy questions. Maybe that's when you want to be in pair programmer mode. But then for, you know, your kind of routine janitorial things where you're cleaning up, you're getting rid of dead code or you're cleaning things up like for that, you want a little bit more of a like, all right, I have a good spec. I'm just going to let it run. I'll come back the next day. I'll like verify in the PR that it did what it's supposed to do.
64
00:24:22.722 --> 00:24:39.496
And so there isn't one single mode or a set of magic words that unlocks things. It's different user behaviors that fit different tasks. That's my my caveat for like, you know, there's not just one mode. I will say in general, the data is very clear.
65
00:24:39.496 --> 00:25:06.547
that there is a worryingly large percentage of users that treat it like a slot machine or like a vending machine. Yeah. And when they don't get what they want, they'll just say, like, try again or no, that was wrong. And they're just very passive. And that, you know, if you just look at the data, those types of users just do not succeed as much. That's maybe uncomfortable because we we talk a lot about how
66
00:25:06.935 --> 00:25:30.543
I feel like a lot in the discourse, it's like AI should, we should all be on a beach while AI is working for us in the background. Like we should all be able to just sit and have a cold drink where I think in reality, like AI requires a lot of hard work and critical thinking and iteration if you really wanna achieve meaningful things.
67
00:25:30.897 --> 00:25:56.378
So that's interesting because you look in most companies now and there's the board, the CEO. We have these people that came on and described. We yell at the director who says, where's our strategy and why aren't we cutting costs and et cetera, et cetera. And so when you think about all those real world applications versus the expectations that are currently sitting in a lot of organizations and on management teams, is that just a complete disconnect versus expectations?
68
00:25:57.138 --> 00:26:22.450
today's achievability versus a reality of where we are. I think you can see some of the stories that are being written about now organizations that token maxed and that have spent incredible budgets on AI and they didn't magically outcompete all their competitors. They didn't magically, like their product and metrics didn't like 10 X overnight.
69
00:26:22.450 --> 00:26:46.109
And a lot of them have found that like, wow, we spent millions and millions of dollars on tokens and we don't like we're just not seeing like where's the return on that? Where's the where's the real impact? And I think what that shows is that, yeah, that it isn't just a machine where like, you know, the more money you put in, the more tokens you get out, the more like business outcomes come out the other side.
70
00:26:46.109 --> 00:27:11.236
is that there's real differences in how you use those tokens and what you're actually doing and who you're giving them to, who's exercising them. That determines whether or not companies are going to see return on all the usage that they're trying to promote across employees. So putting coding aside, if you were sort of sitting
71
00:27:11.388 --> 00:27:35.823
with someone who runs a many hundred million dollar business who was trying to figure out where to start implementing and what team and people they need around the table. Given what you've learned, what would your recommendation be? The trap we can fall into, and I think just through inertia what often happens, is that kind of charge goes to some AI tiger team.
72
00:27:35.823 --> 00:27:56.123
you know, or some, you know, some we're going to find our most AI forward engineers or, you know, our best like technical tiger team, and we're going to give them that task and they're going to go figure something amazing out. And that tends to not work for a couple of reasons. But the biggest one I've found is that the, the, the problem of how you unlock
73
00:27:56.123 --> 00:28:12.307
business value with AI or how do you build a great product with AI. It's not, the key bottleneck is not a technical bottleneck. It's not, can you build technical systems to serve the AI? There's a lot of things that are very complicated in that world, and I don't want to diminish that work.
74
00:28:12.307 --> 00:28:40.201
But that's not what makes or breaks a great product or a business changing application of AI. What makes or breaks that is actually it depends on the people who are closest to the actual business problem or closest to the actual user who they're trying to serve. And those are the people who tend to have these creative insights who are actually closest to the real nitty gritty of the work.
75
00:28:40.201 --> 00:29:09.260
Those are the ones who actually often have those creative insights about where and how AI can actually be leveraged in the most augmentative way. And so I think the first and foremost, like we can't treat it as something that some technical tiger team is gonna go solve and that like technical expertise is what's needed. I'd also say like another trap that I've seen is assuming that model capability buys better outcomes.
76
00:29:09.260 --> 00:29:28.008
So a lot of teams are like, well, we tried something with AI. It didn't work that well. So we're going to try again with the next model. Once GPT 6, 7, 8, 9, 10 comes out, once now finally Fable is no longer blocked by the government and now all these things will be unlocked.
77
00:29:28.008 --> 00:29:49.625
And it is definitely true that capability improvements have unlocked a whole new category of use cases. But for the vast majority of things that we want AI to do, like a couple of IQ points isn't what's holding the model back from being able to achieve that. It's how we're using it, how we're applying it, the behaviors that we're exhibiting when we use it. That's the real bottleneck.
78
00:29:49.625 --> 00:30:10.600
internally as you roll this stuff out, you have to, spy is the wrong term, observe maybe is the nice way of saying it, how employees are using each of the things or how internally people are using the tools and figure out ways to move the organization forward to help educate people or help help them
79
00:30:10.870 --> 00:30:21.502
translate the reality of how people interact with it into how you want to work with the AI. What's your point of view on setting up that observability?
80
00:30:21.755 --> 00:30:41.161
Definitely the single best source of truth for how do our, you know, is our AI working? How do we make it better? Are our employees being successful with AI? What are they actually able to achieve that they weren't able to achieve? The single best source for answering that question is in the actual transcripts in the sessions.
81
00:30:41.161 --> 00:31:00.061
It's not, you know, a dashboard can tell you, you know, we had 10 users. They, you know, had sessions today. Each of them had three turns. It doesn't actually tell you what happened. Did they succeed? So by the way, you're addressing my personal frustration is I've seen our AI usage skyrocket. I know we're using it. I just don't know how and why.
82
00:31:00.230 --> 00:31:26.150
Yep. And if you don't know how and why, like, how do you know where it needs to be improved or what can you hill climb on? How do you build in that observability layer and how do you make it useful but not, you know, 1984 ish call it. Right. Yeah. So we'll talk just about what what the Big Spin product does and kind of our our answer to that is. Yeah. So we help teams with exactly that kind of like observability of what's actually going on in these sessions.
83
00:31:26.150 --> 00:31:50.247
And we specifically build this for teams that are deploying user-facing AI products in high-stakes environments, high-expertise environments. And so there's often not even that just teams want to know what's going on, but it's also an imperative to being able to scale and deliver that AI is that you're able to catch these failures. You're able to catch what's going wrong. You're able to improve the product.
84
00:31:50.399 --> 00:32:16.083
One of the great challenges and the great dynamics of this space is the things that we want AI to do meaningful, consequential things for people, then the data will be very sensitive by definition. And so even within organizations, you really want to protect that data and protect people's privacy and their confidentiality.
85
00:32:16.083 --> 00:32:33.346
And so you have this, that makes the problem even more acute because you have in many cases, employees, there's no single employee who is allowed to see the actual transcript data because it's, for good reason, there's policies and guardrails in place.
86
00:32:33.346 --> 00:32:56.887
But then how can you improve your system? How can you catch the things that are going wrong if like nobody can audit these, you know, long, complicated conversations that are happening? And so this is where LLMs actually are, you know, not just something that makes it scalable, but also like are a privacy preserving approach where when you have, you know, these secure LLMs.
87
00:32:56.887 --> 00:33:23.363
doing the first pass of reading redacted transcripts and annotating it for quality issues. That means you can get the visibility and the insights that you need from these, you know, from empirically what's actually happening between users in the AI without having kind of unprivileged or having human data access to those transcripts. Do you think with those transcripts, you can adjust your internal corporate
88
00:33:23.566 --> 00:33:32.678
AI models so that they self-learn against sort of your own organization's way of engaging, or do you think it comes back to employee training?
89
00:33:33.505 --> 00:33:57.147
So we've talked a lot about how important user behaviors are in success and failure. I am very firmly on the side that it shouldn't be on the user to know all the behaviors that they need to exhibit. We should be demanding of the companies that design and build and ship and deploy big AI products
90
00:33:57.147 --> 00:34:19.793
that they are designing not just for model capability, but they recognize that they're designing an interaction because there are actually very powerful things that you can do in product that change user behavior, that change which behaviors they're likely to express. And in fact, how the model behaves changes how users respond. Like that's the beauty of a conversation is like what you say changes how I behave. And
91
00:34:19.793 --> 00:34:41.123
If you make me feel comfortable and safe, then I am going to, you know, when we're shipping and building AI products, we're not just shipping a capability. We're really designing an interaction that happens between the user and the AI system. And to your question about, well, you know, what can we learn from what's happening in those interactions?
92
00:34:41.123 --> 00:35:02.167
I think this is one of the most slept on maybe things, but everybody who's shipping user facing interactive AI or every organization that is using internally some sort of, you know, use interactive AI. You have every session. There's a human in the loop that's doing evaluation and that's the user.
93
00:35:02.167 --> 00:35:32.153
explicitly or implicitly, when they push back, when they say, oh, no, I actually realized this, or when they express frustration, those signals of what they're actually doing in those sessions, those signals are, you know, we started this conversation talking about how do you know what good means? Well, a really good place to start is look at what users are already expressing in the session. Look at what people are already doing in the session. I know that for myself, for my own
94
00:35:32.153 --> 00:35:55.947
So many things that get surfaced in that session are things that define what good means in future sessions. So, you know, the most simplest things could be like, you know, we use this tailwind class rather than that, or we have this convention rather than that. It's not something that the model could have, you know, it's not something that depends on how smart the model is. It's something that's application or business specific.
95
00:35:55.947 --> 00:36:20.753
And those things get surfaced in these sessions. And if we just have ways to capture them, to log them, to actually learn from them, to have them not die, you know, once the session ends, that is this amazing, you know, data asset and feedback loop that's just waiting to be activated. So every organization that has internal, you know, chat logs between with employees trying to do real work,
96
00:36:20.753 --> 00:36:47.753
They're sitting on this incredible latent artifact that's just waiting to be mined for feedback, for these pushback moments to surface like, okay, what does good mean for us as an organization, for our employees when they're trying to achieve these tasks? So maybe this is getting too dystopian or meta, but if this is the way humans interact with models, as we move more towards an agent-based world,
97
00:36:48.563 --> 00:36:54.453
Do the individual agents also face this issue when interfacing with the models and with other agents?
98
00:36:54.790 --> 00:37:22.567
I do think in five years, the vast majority of interactions, conversations will be conversations between agents trying to figure stuff out on your behalf or on behalf of some other agent. And I think those conversations will look different because LLMs are different from humans, but they will still be conversations and they will still have these interactional failure modes and their own ways of going south.
99
00:37:22.567 --> 00:37:38.311
And I do think the danger for us or the challenge is, you know, the bottleneck is not, you know, going and having models generate stuff or having models go do work like they will. They can go generate a bunch of work.
100
00:37:38.311 --> 00:37:58.240
The bottleneck- Faster than I can review it. Exactly. They can be created faster than I can review it. That's the bottleneck is verification. And there are very few domains where verification is something that you can do mechanically, where verification where you have some sort of oracle that can do verification. For most things like knowledge work and most things like complicated things we want to use AI for.
101
00:37:58.240 --> 00:38:15.301
it's more subjective. There's more expertise and judgment that are required. And so I think the dynamic that we should be thinking about is like, well, if it's hard right now in this current moment to verify all the work that AI is doing and all of that output,
102
00:38:15.301 --> 00:38:32.277
Well, how much more difficult is that going to be in five years when you have 10x, 100x, the amount of work that's happening now between agents and the amount of tokens that like, how are you going to verify all of that? How do you know what to pay attention to? Yeah, so what's the answer? I don't know.
103
00:38:32.277 --> 00:38:52.240
I think it's turtles all the way down. You need trusted LLMs that watch the other LLMs and that flag certain things. You need LLMs to help mediate our attention, help us figure out what to pay attention to. Because we're never going to want to take our judgment out of it. We're never going to want to take our human-in-the-loop verification out of it.
104
00:38:52.240 --> 00:39:21.468
We just need to be really targeted and smart about, well, what are the moments when, you know, what are the highest leverage moments where we do need to verify, where we do need to express judgment, where we do need to, to, you know, have, make sure our expertise is expressed. And then how do we, when those moments happen, how do we learn from them? How do we capture from them so that then the system can incorporate them moving forward? And in big Spence case in particular, do you imagine a world where a lot of the usage is
105
00:39:21.991 --> 00:39:42.528
you observing the agents talking to the agents and then allowing. Yeah. That is, we, that's, can you talk me through that real fast? That, that's, yeah, we, we believe, you know, in five years or, you know, maybe less like the majority of right now, what our system monitors and where we kind of, what we mine for quality signals and,
106
00:39:42.528 --> 00:40:12.515
It's interactions between users and AI systems. It's all user in the loop. But even there, you've seen the, you know, every day that passes pretty much, there's more happening on the AI side behind the scenes. There's more tools being called. There's more tokens going to thinking. Like more and more of it is more and more of just like the raw text that you kind of need some system to verify and monitor. More and more of that is happening on the LLM side.
107
00:40:12.515 --> 00:40:28.377
It's very common now to have it spawn sub-agents as part of a single response that it's giving a user. More common for it to have really long-running tasks where the user comes back the next day to see after it had worked for a couple hours.
108
00:40:28.377 --> 00:40:53.099
And we're just going to see that continue. We're going to take the, you know, harnesses are going to be capable of taking on longer running tasks. So you're going to have more thinking happening, more tool calls happening behind the scenes, but you're also going to have more autonomous agents that are just off in the world doing things for you. You know, you're going to have, you know, one agent that's like negotiated, that's representing you, that's trying to negotiate the best price for a plane ticket from some agent that's representing the airline company.
109
00:40:53.099 --> 00:41:09.923
And there's just going to be an explosion of more and more agents talking to agents on your behalf. Or the product that you're shipping is just going to be more and more systems and teams of agents trying to accomplish things.
110
00:41:09.923 --> 00:41:35.877
And so it's just going to like this precipitous trend of the AI just producing more and more tokens, more and more output, doing more and more work kind of on your behalf in the background, in a room that nobody else is in, where nobody can verify, nobody can pay attention, nobody can flag things that might need your attention. That is just going to become more and more of an acute dynamic. And so having these systems in place now...
111
00:41:35.877 --> 00:42:05.105
that let us gain trust, that let us, you know, that direct our attention to the right moments, that learn what are those moments where we need human attention. That's gonna be critical in a world like where you're trying to verify and make use of all the work that's happening on your behalf in the background. In a lot of ways, it's almost like I'm imagining a world where there's just people watching and reading through the discussions between the agents, helping the educate,
112
00:42:05.830 --> 00:42:31.767
Agents, I guess that's the world we're heading towards as you describe it, right? Where you're... Yeah, in a way, we're already doing that on a small scale every time we're in a session with the agent. And what I'll notice myself is AI responses are very long. And a lot of the time, I don't read the full response. I kind of skim around or scan around and maybe read part of it.
113
00:42:31.767 --> 00:42:59.408
So it is already a world in which like our, you know, there's a lot of a lot happening that we're, you know, even in these sessions where we're theoretically fully present, fully human in the loop, even there, there's a lot that no human is looking at. I mean, I had a frightening moment where I use AI for research, like to run experiments, to help do these large scale experiments. And in general, it's been this phenomenal productivity booster for me.
114
00:42:59.408 --> 00:43:15.608
But I had this one scary moment where I just happened to, you know, I was scanning. It was like a very long running 12 hour task that it was doing for me. And I just happened to be scanning through and I expanded one of the thinking blocks. And I was just and something caught my eye.
115
00:43:15.608 --> 00:43:33.142
And what caught my eye was this line where the thinking block said, you know, it had run an experiment and there was a preregistered number of like, OK, if we, you know, this scientific experiment, if we match this, if we hit this threshold, we're going to say this hypothesis failed. If we if we don't, it's going to say that the hypothesis was proven true.
116
00:43:33.142 --> 00:44:01.120
And I saw this block of thinking text that was like, oh, you know, metric is point zero zero zero three off of the cutoff. I will adjust the cutoff for a fair and true comparison. It was totally undermining the scientific integrity of the entire experiment. But like dressed up in this language of I'm being fair, I'm doing the right thing. That's not like like how many of those were happening that I didn't that I hadn't caught.
117
00:44:01.120 --> 00:44:28.475
So this prompted me to do a massive audit where I had it like do an audit of every single research thread that it had done for me recently. And like looking at all of the, you know, all of the thinking texts and finding all of these potential, you know, potential issues. And I think that just really made me, you know, like even me as a very highly engaged user, like there are just things that I wasn't paying attention to that are like wildly important and critical for like how much I trust the output. What does that mean? If you're a big company,
118
00:44:29.302 --> 00:44:48.185
and you're entrusting this stuff and they're just arbitrarily disregarding your goals. How do we view this in the next couple of years of enterprise AI adoption here? It means if you don't have systems that are actually looking at what your AI is doing, if you are flying blind,
119
00:44:48.185 --> 00:45:16.838
then that's on you in many ways. Like these systems, they're not predictable. Just because you've given them what you think are good instructions doesn't mean that they are going to be doing right by your business. It doesn't mean that they know what even what good means for your business. So if you don't have systems that let you actually see what your AI is doing, understand where it's falling short from what your business needs or what your users need,
120
00:45:16.838 --> 00:45:33.325
and then improve on those and measure whether the improvements work. If you don't have those feedback loops, if you're not able to have the visibility to catch failures and hill climb, then you are best case shipping slop and worst case shipping something that could actively harm users.
121
00:45:33.325 --> 00:45:54.908
And so what does that tech stack look like then if you're looking at forward a few years, like either specific companies or layers? So we already see pretty established that, you know, the kind of tracing category. There's some, you know, database where the traces and kind of the raw exhaust.
122
00:45:54.908 --> 00:46:13.268
can be stored. But right now, most teams use those tracing platforms mostly for kind of technical debugging. And I think like what we're now starting to see and you're starting to see that the tracing platforms as well start to move kind of upwards in this direction. But I would imagine that
123
00:46:13.488 --> 00:46:38.041
like this is actually where there's going to be a lot of innovation, you know, a dedicated category pretty soon is then, you know, well, how, what's actually reading all the traces? What's actually paying attention to all the traces, not from a technical perspective of like, is the latency, you know, P90, what we expect, but from a business and product perspective is like, are the user goals being met? Are the business requirements being met?
124
00:46:38.041 --> 00:47:00.974
more LLM inference is going to go to verifying and testing and paying attention to what, you know, what AI products are doing, then those AI products are using themselves. So like we will be using LLMs more to help watch LLMs and help us understand when to pay attention to LLMs than like, you know, the LLMs doing the generation.
125
00:47:00.974 --> 00:47:18.355
Yeah, I think this old Dr. Seuss, you know, book of the places you'll go. This hawk hotcher is like the watch who watches the watcher watches and watch. Who watches the watcher. That's how I'm imagining this. And I think like ultimately the most important thing we have
126
00:47:18.457 --> 00:47:44.832
like on a team and in organizations and just in general, right. And individually is our human attention. It's like, you know, there are hundreds and thousands of things every day competing for our attention and like where we spend our attention that determines like, well, what, where are we, you know, where are we funneling our judgment, our taste, our expertise, our energy, our motivation. And so to me, this is just a variant of that question, which is,
127
00:47:44.832 --> 00:48:10.837
you know, using AI, AI working on our behalf, it's going to demand our attention and it's going to be, you know, the raw scale and volume of it is gonna be more than, you know, any single, more than there is human attention to address. And so the key question that, you know, that we need to answer here is like, well, how, what are the systems that we have that make sure we are paying attention to the right parts?
128
00:48:10.837 --> 00:48:40.520
that make sure that we are spending that attention that we have, that our teams have, that our experts have, and we're spending that attention in a really effective way. And so that's in many ways what I see the Big Spin product doing, what I see our research doing, is trying to push forwards on how do we find better systems to direct our attention on the things that actually matter, the things where we actually have leverage to improve outcomes that our AI products and our AI systems are delivering.
129
00:48:41.465 --> 00:48:46.645
That's a perfect way to end. Thank you so much. This was awesome. Thanks. This was a little scary, but very awesome.
1
00:00:00.031 --> 00:00:24.719
I use AI for research, like to run experiments. But I had this one scary moment. It was like a very long running 12 hour task that it was doing for me. And I just happened to be scanning through and I expanded one of the thinking blocks. And I saw this block of thinking text that was like, oh, metric is 0.0003 off of the cutoff. It was totally undermining the scientific
2
00:00:24.719 --> 00:00:49.913
integrity of the entire experiment, but like dressed up in this language of I'm being fair, I'm doing the right thing. Can you talk about what an invisible failure is? Because I see that everywhere. And I know you talk about a lot. So I'm just curious if you can explain it.
3
00:00:50.116 --> 00:01:13.032
An invisible failure in AI is when something breaks, something goes wrong, something the user isn't satisfied or gets the wrong information. Something breaks down in an AI interaction, but there's no alert. There's no crash. There's no error message. And it's not even something that the user recognizes. They don't give feedback. They don't escalate.
4
00:01:13.032 --> 00:01:33.552
So it's something that goes wrong, but leaves no visible sign or alert or trigger or flag in any dashboard or any kind of our systems that we use to track. And of course, what we found in our research was that the vast majority, 79 percent of failures that are happening in AI interactions are staying invisible.
5
00:01:33.552 --> 00:01:58.460
which means that they're not being, you know, the user isn't catching them, they're not being logged in some dashboard. And the danger of that, of course, is what we can't fix, what we can't detect. We can't hill climb if we don't know what's breaking and what good even means in these interactions. And so we've tried a lot through our research and our product to, you know, be able to now finally see what has been invisible and see these interactional failures that are happening that are letting users down.
6
00:01:58.460 --> 00:02:04.265
Presumably now you're talking about on the customer facing side, but given how much
7
00:02:04.754 --> 00:02:32.328
how many failures there tend to be just as you use it as an individual. Are you surprised that the rate is so high that there aren't harnesses or other guardrails to catch it just in general? I am surprised as a, so I'm both surprised and not surprised as a user myself. I, I myself have go through these phases. I feel like we're sometimes I'm just amazed at how good AI is and how smart it is and how much, how well it does the task that I want it to.
8
00:02:32.328 --> 00:02:51.734
And then other times, you know, it's like this is the dumbest thing in the world. How could it get that wrong? And then I wonder, you know, how many things did I even not catch? And when it's you know, when it when I feel like it's so smart, is that because I'm being really smart? And when I feel like it's being so dumb, is that because I'm using it wrong or is there something about the model?
9
00:02:51.734 --> 00:03:14.262
And so I think that's kind of what's the great story. And as we really started to dig into this in the research as well, is that a single AI model, even used for the same use case, it can just have dramatically different success or failure just based on who happens to be in the room, what the user happens to do, what mindset they happen to bring to that interaction.
10
00:03:14.262 --> 00:03:43.523
And I've certainly seen that in myself, like when I treat it maybe like a vending machine or just hope that like a lottery and I hope it gives me something amazing. Like sometimes it works and sometimes I'm disappointed. And then sometimes when I put in that really hard work, I'm able to like break through on something. And those are often the most frustrating AI interactions that I have where I'm like expressing my frustration in the chat and yet at the same time, the most frustrating interactions are often the ones where I achieve something, I achieve the things with the greatest complexity.
11
00:03:43.523 --> 00:04:08.228
I have the greatest creative breakthroughs. And so it's really hard, I think, for us to speak in this sort of cut and dry way about success and failure. We really have to look at these interactions that are happening between us and the AI, between different styles of users and the AI, and how that has dramatically different impacts on how successful it is. Can you talk through some of those styles and how they impact people?
12
00:04:08.802 --> 00:04:28.158
Yeah. So just for context for folks, we tried to study this in a quantitative way, do a large scale study of actual real production interactions that users had with AI systems. So these are not test interactions. These are real scenarios where users are trying to get some kind of work done. And this spanned domains like coding,
13
00:04:28.158 --> 00:04:46.045
all the way to creative writing and emotional support. So the whole gamut of what people are using AI for and day-to-day. And this was a study done specifically on ChatGPT logs as a public dataset. And what we wanted to do is
14
00:04:46.045 --> 00:05:14.817
try to put some more numbers and rigor behind this kind of this anecdotal thing that we're discussing that's like, man, you know, some people are having a lot of success and other people are not with AI. Like what what what is that? How does that come down to? What's the difference? And for this study, we really looked at how, you know, keeping the model fixed, keeping the product and the user interface fixed and just looked at what do the users do in a session? What behaviors do they exhibit? What do how do they treat the model? What are they you know, how are they trying to do what they're trying to get done?
15
00:05:14.817 --> 00:05:32.165
And I think that the dramatic thing is there is really a big gap and a big difference in how just kind of everyday people approach these tools. And you see a big difference in what they're able to achieve based on what behaviors they exhibit. So kind of roughly speaking.
16
00:05:32.165 --> 00:05:54.001
There's the biggest gap that we found is in users who treat it a little bit more from a delegative perspective, like thinking about how they kind of delegate tasks to AI versus ones that really engage with a more augmentative mindset. So these are users who are like iterating and pushing back and challenging and bringing their own context and critical thinking.
17
00:05:54.001 --> 00:06:21.777
And what you find is that that second group, which is a tiny percentage, by the way, it's about 3% of really high fluency, high augmentation users. That small percentage are able to achieve significantly more complicated tasks with AI. And there's this interesting paradox where if you just look at those augment of high fluency users, they're actually failing more. Like there's more visible failures. There's more things that don't work.
18
00:06:21.777 --> 00:06:37.522
But at the same time, they're also trying to achieve significantly more complicated things, on average, twice as complicated, and they're succeeding more at those tasks. And so it's that behavior that they bring and, frankly, the hard work that they're putting into that interaction.
19
00:06:37.522 --> 00:06:59.864
that allows them to achieve just different outcomes compared to this vast majority of users in this study who were a little bit more delegative, more treating it like a slot machine, seeing what came out, maybe walking away if they didn't like it. In either of those cases, how do you define right and wrong in a non-objective output scenario?
20
00:07:00.421 --> 00:07:20.333
That's the real question, right? Is what does good mean? But I think this is like what I've struggled with since I first started trying to build products on top of modern LLMs. So my kind of first cutting my teeth on this was I was VP of AI at BetterUp and we're building an AI coach. So you have now an LLM that is going to
21
00:07:20.333 --> 00:07:49.426
hopefully do so many of these things, like supporting people, helping them with self-reflection and self-insight and motivation and learning and growth. And it's this phenomenal opportunity that now, you know, you're shipping a conversation that you can kind of ship at an incredible scale and just help people anytime a day, anywhere. And so how do you define, you know, you're shipping AI conversations now that are doing coaching. How do you define what a good coaching session is when you've deployed an AI coach? How do you measure that?
22
00:07:49.426 --> 00:08:14.165
And it's hard to pin down. And so the obvious starting point is, well, you know, what does it mean to be a good human coach? And that's also hard to pin down. But there's frameworks, there's certifications, there's competencies. And so, of course, like the first thing that we tried was, well, we want to make sure that the AI coach that we're shipping is like a really good coach from from kind of an expert coach's opinion. Yeah.
23
00:08:14.165 --> 00:08:40.456
And so we had coaches in the loop training it. We did evals that were based on all these competencies. And there's this one moment that I had where we've really invested in a new coach. We're doing this pilot and we're about to ship this new coach. And they would really like it was significantly better at coaching as judged by expert human coaches. And so from the objective metric of what does good coaching look like, this was a way better model, way better system.
24
00:08:40.456 --> 00:09:07.523
But when we A-B tested it, which I thought was like a pro forma thing, I'm going to be able to measure how much better this thing is. Users liked it significantly less. The engagement was way lower. What's your North Star? Is it how objectively good the coaching is? Is it whether users like it? Neither of those really by itself can be the North Star. If you're just optimizing for user engagement, you'll get sick of fancy. You won't like a good coach that should be difficult. Sometimes there should be friction in those experiences.
25
00:09:07.523 --> 00:09:24.770
But if you're just optimizing for the standard human definition of what a good coach is, well, you're shipping something that people don't actually want to use. You're shipping something that's like an AI medium, but now you've used this human analog and tried to retrofit it. And so the real question to me, it's how do you determine what good is?
26
00:09:24.770 --> 00:09:51.145
How do you measure that? And you can't rely on proxies from the human world. You can't just rely on lagging metrics like engagement and re-engagement. You have to actually look at what's happening in these sessions. What job is the user hiring this AI to do for them? And then evaluate, like, you know, to what extent is it fulfilling those goals? In non-consumer facing, but in internal tools
27
00:09:51.652 --> 00:10:06.400
that companies are now trying to build, do you see that same bifurcation happening in terms of the inability to define good? Once you get outside of the workflow stuff, which I think is probably easier.
28
00:10:06.991 --> 00:10:22.972
I think the best place to maybe look at this is in AI coding. So most highly used internal application and the hottest area for AI doing actual work output.
29
00:10:22.972 --> 00:10:50.900
So we did a study on trying to understand, like, what is a token actually buy you when you're doing AI coding? And how do you actually measure, if you look at a group of 100, 200, 300 users that are all using Cloud Code, how are they all getting the same productivity gains from it? How are some getting more gains? What distinguishes those who are seeing how more productivity gains from those who aren't? And what can each group learn from each other?
30
00:10:50.900 --> 00:11:19.840
And I think what's, again, so difficult about that question is theoretically for AI coding, you should have more objective metrics for output, for work output, because, you know, you can look at PRs, you can look at lines of code drafted, you can look at their crude, but they are, you know, there are just raw, countable things about engineering throughput and productivity. But I think what's so, so difficult to measure about that is that
31
00:11:19.840 --> 00:11:37.846
the way that people actually use AI coding agents, there's just like any AI product, there's this really large variety of different usage styles. And on one end of the spectrum, let's say it's people who really treated like a code writing tool.
32
00:11:37.846 --> 00:11:53.945
And so you have a very detailed spec. You tell it exactly what you want it to write. And in that way, it is a machine that is more directly sort of, you can see it as like, this is a code writing utility. And you can measure on how much code did it write? Did it pass all the tests? Did it get merged, et cetera?
33
00:11:54.383 --> 00:12:15.798
But if you look at how a lot of how I use Cloud Code, that way up the spectrum of, I really treat it like a collaborator. I treat it like a teammate, like a peer. Like I expect it to know things about our code base, know things about our product vision, know things about our goals and our users. And I expect it to help me think through things and be creative and iterate.
34
00:12:15.798 --> 00:12:44.367
And in that world, you can't just measure it based off of like my utility based off of how much code is written, because that doesn't you know, how do you capture maybe the inspiration that I have from a good idea or from from something that it showed me that I wouldn't have been able to, you know, to iterate on as quickly just by myself. So even in a domain that's technically verifiable and has raw kind of output metrics like coding, even there, I think the hardest question is how do you actually measure value?
35
00:12:44.671 --> 00:13:11.317
There's a presumption, I guess, behind all of that, that the model, even if you are using it consistently, is reacting consistently back to you. And have you found that to be the case that within a certain model setting that your style is consistently producing similar outputs? That is, again, something that is hard to nail down, but I think anybody who's
36
00:13:11.317 --> 00:13:41.303
used AI for something now and tried to do the same thing a couple of months ago, it's very clear that everything is changing very fast about these models and about the harnesses. And a lot of that is happening in ways that we as users don't see. So we don't have visibility into what memories ChatGPT is storing about us or what cloud code is creating based on the work.
37
00:13:41.303 --> 00:14:05.283
We don't know how we can't even, you know, a lot of these models, more and more of the tokens are going to thinking. That's not even something that the user can see that even if we wanted to read all that, that we only get summaries of that. And so more and these models, the harnesses, everything is changing behind the scenes. And that's just another variable in this very difficult world where you have a model that keeps changing.
38
00:14:05.283 --> 00:14:28.047
A user who's also changing and has different behaviors and different mindsets, different moods they might be in at a given day. And the product of what happens in an AI session is somehow like this chemical reaction at a given moment between those two. And we did try to study this as well. And because I saw just how much token usage is increasing.
39
00:14:28.047 --> 00:14:56.701
And that's one of the most common things a lot of organizations measure in general is like, well, it's hard to measure what the value is that we're getting out of it, but we can at least measure how much we're using this thing. Yeah. And what's so interesting is that even for the exact same task, the exact same user kind of doing the same behaviors, you've seen a steady increase in how many tokens are being used to fulfill that task. Because the models are just in general, each successive generation becoming more token hungry
40
00:14:56.701 --> 00:15:22.891
and putting more of that into the thinking. I read that you coined this term tokenflation. Yeah. Can you just define it real quickly? We're just trying to find a word to describe this phenomenon that we found in the research that a token today buys you less raw work output than a token yesterday. And so it's the same as inflation. You know, if you go to a grocery store, how many eggs and how many apples can you buy for a dollar?
41
00:15:22.891 --> 00:15:37.893
And the consumer price index tracks that over time. So we did a study doing the exact same thing for tokens. And again, looking at AI coding outputs. So looking at these engineering outputs, which we've talked about, they're imperfect. They don't capture everything.
42
00:15:37.893 --> 00:16:00.050
But they are a good raw kind of unimpeachable just metric of like how much code is being written, how many PRs are being merged. And so if we look at kind of a broad scale, so we looked at February to April, Opus 4.6, kind of the same cohort of about 115 users. So we looked at all of the work that they were doing, all of their sessions on Cloud Code.
43
00:16:00.050 --> 00:16:15.355
And we just tried to study like, well, what how is the purchasing power of a single token changing over time? So our basket of goods here is things like, you know, how much how many tokens does it take to buy a PR? How many tokens does it take to buy one line of code that's drafted?
44
00:16:15.355 --> 00:16:40.685
And what we found was that there's kind of similar to inflation. There's just been this steady decrease where, you know, a 10,000 token session in February, it bought about a third of a PR, eight files touched, about 600 lines of code drafted. By April, the same amount of tokens, 10,000 tokens was buying a sixth of a PR, four files touched, 90 lines of code, about.
45
00:16:40.685 --> 00:17:01.171
And is the cost of the token consistent or is it going down or how is that impacted there? So that's why we focus just on what is a token buy rather than just what is a dollar buy, because the model providers, they change their pricing. Also, some users are on subscription plans where there's like certain.
46
00:17:01.171 --> 00:17:28.222
rebates or the economics of how that translates to dollars is it's changes a lot. And I think that's just another thing for anybody who's trying to like project future spend and think about ROI should keep in mind is like the cost of a token is changing. How many tokens are required to do a piece of work is changing. Like all of these variables are changing and you can't rely on them. Going back to your core statement, why would the same token produce less output even though they're thinking is the output
47
00:17:28.222 --> 00:17:52.943
more calculated and defined, or is it just the way the models are now being designed? Because that's counterintuitive, I would think. Yeah. So, and I can't answer that definitively. So what we can tell is where are the extra tokens going? So you can track that to, you know, earlier it was like most of the tokens would actually go to tool calls to the actual code generation task.
48
00:17:52.943 --> 00:18:08.840
Now, more of those tokens proportionally are going into thinking and explanation to the user. So you can kind of say, okay, the role of the token is changing a little bit. And then you think, well, what is that? What impact is that having? What is that buying us or what's the trade that you get from that?
49
00:18:08.840 --> 00:18:34.996
And, you know, I think that the two obvious kind of answers or kind of explanations would be one, users are able to do more complicated things now. And so they're just doing more complicated things. And that's why it costs more tokens. The data, the study that we did, it didn't we didn't find evidence for that. Users were kind of doing the same things that the token was increasing. So then the second the second explanation would be what's buying quality.
50
00:18:34.996 --> 00:18:52.546
Like your more thinking means that the line of code that it writes is better. Maybe it didn't need 600 lines of code. It only needed 90. And because of extra thinking, it was able to find the right 90. And there is some evidence for this. So we studied code survivability. So every line of code that's written in a session
51
00:18:52.546 --> 00:19:19.192
how long does it survive in the code base? How quickly is it overwritten or deleted or changed as kind of a signal of what's the kind of durable value of that code. That's an interesting metric, yeah. And if we look at just like kind of seven days survivability of the line of code, it goes from 90% in February to 95% in April. So there is an increase there of 5% or roughly half the gap. I can't make a value judgment about whether that was worth it.
52
00:19:19.192 --> 00:19:49.162
But I think what we're trying to do and just trying to push, I think, the industry towards is we have to be able to measure the trade and get better at measuring what do we get when we spend a token? Because as more and more of our costs are going to AI to make really smart decisions about how to leverage that spend and where can we get the most out of it, like unless we're able to measure what the token is buying us and where it buys us more value than in other contexts, we're not going to be able to make the decisions we need to make.
53
00:19:49.162 --> 00:20:11.977
And what metric do you look at internally or then do you recommend other people look at? I, again, this is, I wish I had like the perfect set of metrics. Well, there's all these startups, right, that are starting now to measure and do this. And, you know, it feels like every couple of weeks somebody else is creating a layer of cost management. But I don't know what metric they're looking at though.
54
00:20:11.977 --> 00:20:35.956
But the hard part for me is like, it's easy to track cost relatively. Like we think about ROI, the investment part of that equation is easy to track. It's the return part that we should really be focused on. And I think that's where the most of the time the answer tends to be quite business specific. It depends on your strategy as a business, right? It depends on what you actually care about.
55
00:20:35.956 --> 00:20:57.134
What environment you're operating in. And so for my team, there's two things that I pay the most attention to. One is just what percentage of our token spend and AI spend in general is going towards use cases that I consider like compounding.
56
00:20:57.134 --> 00:21:24.843
So, where it's not just a transactional, like getting something done, it's like a token spent here improves something else later, like something accrues because of it. And that to me is like a, just a kind of a rough value signal of like, how much are we leveraging the spend in a way that we're like, where something is compounding and growing rather than it's just kind of a spend that we have to stomach and we're like cost of doing business.
57
00:21:24.843 --> 00:21:49.126
And the other thing that I really track is how much not just helping with an individual's productivity, like I used to do XYZ task and it took me two days. Now, because of AI, I can do it in one day or one hour. I actually pay a lot of attention to what are our systems and feedback loops that we have where people now can...
58
00:21:49.126 --> 00:22:10.895
not just do something faster, but do something that they just fundamentally previously could not do. They can now play a role that they were not able to play. So for example, for me, I've been an engineer in the past, but I'm not a really experienced front-end engineer. I get really nervous with backend data migrations, all that kind of stuff.
59
00:22:10.895 --> 00:22:38.485
But I have really great skill files in our repo that have been curated and created by the experts on my team that mean that my Cloud Code can just do a really good job of following our code bases conventions, following our design principles, and following the right workflows where now I can take on meatier features that I was not able to do previously. And so that's not just an example of it saving me time. It's just an example of I now am capable
60
00:22:38.485 --> 00:23:02.853
of doing things that I wasn't capable of doing before. I talk a lot about how we need to get better measurements for what's the actual value that we're getting from AI. And I'm still trying to figure that out for our business. But I think those are the things that at a meta level, I think, are like, how do you make sure you're funneling that spend and that energy into the most productive areas?
61
00:23:04.287 --> 00:23:29.617
So you mentioned these different types of AI users and in a company, oftentimes different personalities are what drives the success of the company. We come together and we all come with different points of view. Do you need all those different types of AI users to get to the right outcome or are six or five of them fundamentally wrong and we need to do better training internally to get to the right outcome?
62
00:23:30.562 --> 00:23:53.579
I think both can and should be true. And let me try to be nuanced, but also succinct, which is if you look at, you know, if you look just at the data at, you know, power users of AI. Again, let's look at cloud code as an example, like modern agentic system. The most productive power users, they actually don't have just one way of interacting with AI. Yeah.
63
00:23:53.579 --> 00:24:22.722
they will change how they're interacting with AI based on what they're trying to accomplish in that session. So, you know, big meaty tasks where you're really trying to work through something where there's strategy questions. Maybe that's when you want to be in pair programmer mode. But then for, you know, your kind of routine janitorial things where you're cleaning up, you're getting rid of dead code or you're cleaning things up like for that, you want a little bit more of a like, all right, I have a good spec. I'm just going to let it run. I'll come back the next day. I'll like verify in the PR that it did what it's supposed to do.
64
00:24:22.722 --> 00:24:39.496
And so there isn't one single mode or a set of magic words that unlocks things. It's different user behaviors that fit different tasks. That's my my caveat for like, you know, there's not just one mode. I will say in general, the data is very clear.
65
00:24:39.496 --> 00:25:06.547
that there is a worryingly large percentage of users that treat it like a slot machine or like a vending machine. Yeah. And when they don't get what they want, they'll just say, like, try again or no, that was wrong. And they're just very passive. And that, you know, if you just look at the data, those types of users just do not succeed as much. That's maybe uncomfortable because we we talk a lot about how
66
00:25:06.935 --> 00:25:30.543
I feel like a lot in the discourse, it's like AI should, we should all be on a beach while AI is working for us in the background. Like we should all be able to just sit and have a cold drink where I think in reality, like AI requires a lot of hard work and critical thinking and iteration if you really wanna achieve meaningful things.
67
00:25:30.897 --> 00:25:56.378
So that's interesting because you look in most companies now and there's the board, the CEO. We have these people that came on and described. We yell at the director who says, where's our strategy and why aren't we cutting costs and et cetera, et cetera. And so when you think about all those real world applications versus the expectations that are currently sitting in a lot of organizations and on management teams, is that just a complete disconnect versus expectations?
68
00:25:57.138 --> 00:26:22.450
today's achievability versus a reality of where we are. I think you can see some of the stories that are being written about now organizations that token maxed and that have spent incredible budgets on AI and they didn't magically outcompete all their competitors. They didn't magically, like their product and metrics didn't like 10 X overnight.
69
00:26:22.450 --> 00:26:46.109
And a lot of them have found that like, wow, we spent millions and millions of dollars on tokens and we don't like we're just not seeing like where's the return on that? Where's the where's the real impact? And I think what that shows is that, yeah, that it isn't just a machine where like, you know, the more money you put in, the more tokens you get out, the more like business outcomes come out the other side.
70
00:26:46.109 --> 00:27:11.236
is that there's real differences in how you use those tokens and what you're actually doing and who you're giving them to, who's exercising them. That determines whether or not companies are going to see return on all the usage that they're trying to promote across employees. So putting coding aside, if you were sort of sitting
71
00:27:11.388 --> 00:27:35.823
with someone who runs a many hundred million dollar business who was trying to figure out where to start implementing and what team and people they need around the table. Given what you've learned, what would your recommendation be? The trap we can fall into, and I think just through inertia what often happens, is that kind of charge goes to some AI tiger team.
72
00:27:35.823 --> 00:27:56.123
you know, or some, you know, some we're going to find our most AI forward engineers or, you know, our best like technical tiger team, and we're going to give them that task and they're going to go figure something amazing out. And that tends to not work for a couple of reasons. But the biggest one I've found is that the, the, the problem of how you unlock
73
00:27:56.123 --> 00:28:12.307
business value with AI or how do you build a great product with AI. It's not, the key bottleneck is not a technical bottleneck. It's not, can you build technical systems to serve the AI? There's a lot of things that are very complicated in that world, and I don't want to diminish that work.
74
00:28:12.307 --> 00:28:40.201
But that's not what makes or breaks a great product or a business changing application of AI. What makes or breaks that is actually it depends on the people who are closest to the actual business problem or closest to the actual user who they're trying to serve. And those are the people who tend to have these creative insights who are actually closest to the real nitty gritty of the work.
75
00:28:40.201 --> 00:29:09.260
Those are the ones who actually often have those creative insights about where and how AI can actually be leveraged in the most augmentative way. And so I think the first and foremost, like we can't treat it as something that some technical tiger team is gonna go solve and that like technical expertise is what's needed. I'd also say like another trap that I've seen is assuming that model capability buys better outcomes.
76
00:29:09.260 --> 00:29:28.008
So a lot of teams are like, well, we tried something with AI. It didn't work that well. So we're going to try again with the next model. Once GPT 6, 7, 8, 9, 10 comes out, once now finally Fable is no longer blocked by the government and now all these things will be unlocked.
77
00:29:28.008 --> 00:29:49.625
And it is definitely true that capability improvements have unlocked a whole new category of use cases. But for the vast majority of things that we want AI to do, like a couple of IQ points isn't what's holding the model back from being able to achieve that. It's how we're using it, how we're applying it, the behaviors that we're exhibiting when we use it. That's the real bottleneck.
78
00:29:49.625 --> 00:30:10.600
internally as you roll this stuff out, you have to, spy is the wrong term, observe maybe is the nice way of saying it, how employees are using each of the things or how internally people are using the tools and figure out ways to move the organization forward to help educate people or help help them
79
00:30:10.870 --> 00:30:21.502
translate the reality of how people interact with it into how you want to work with the AI. What's your point of view on setting up that observability?
80
00:30:21.755 --> 00:30:41.161
Definitely the single best source of truth for how do our, you know, is our AI working? How do we make it better? Are our employees being successful with AI? What are they actually able to achieve that they weren't able to achieve? The single best source for answering that question is in the actual transcripts in the sessions.
81
00:30:41.161 --> 00:31:00.061
It's not, you know, a dashboard can tell you, you know, we had 10 users. They, you know, had sessions today. Each of them had three turns. It doesn't actually tell you what happened. Did they succeed? So by the way, you're addressing my personal frustration is I've seen our AI usage skyrocket. I know we're using it. I just don't know how and why.
82
00:31:00.230 --> 00:31:26.150
Yep. And if you don't know how and why, like, how do you know where it needs to be improved or what can you hill climb on? How do you build in that observability layer and how do you make it useful but not, you know, 1984 ish call it. Right. Yeah. So we'll talk just about what what the Big Spin product does and kind of our our answer to that is. Yeah. So we help teams with exactly that kind of like observability of what's actually going on in these sessions.
83
00:31:26.150 --> 00:31:50.247
And we specifically build this for teams that are deploying user-facing AI products in high-stakes environments, high-expertise environments. And so there's often not even that just teams want to know what's going on, but it's also an imperative to being able to scale and deliver that AI is that you're able to catch these failures. You're able to catch what's going wrong. You're able to improve the product.
84
00:31:50.399 --> 00:32:16.083
One of the great challenges and the great dynamics of this space is the things that we want AI to do meaningful, consequential things for people, then the data will be very sensitive by definition. And so even within organizations, you really want to protect that data and protect people's privacy and their confidentiality.
85
00:32:16.083 --> 00:32:33.346
And so you have this, that makes the problem even more acute because you have in many cases, employees, there's no single employee who is allowed to see the actual transcript data because it's, for good reason, there's policies and guardrails in place.
86
00:32:33.346 --> 00:32:56.887
But then how can you improve your system? How can you catch the things that are going wrong if like nobody can audit these, you know, long, complicated conversations that are happening? And so this is where LLMs actually are, you know, not just something that makes it scalable, but also like are a privacy preserving approach where when you have, you know, these secure LLMs.
87
00:32:56.887 --> 00:33:23.363
doing the first pass of reading redacted transcripts and annotating it for quality issues. That means you can get the visibility and the insights that you need from these, you know, from empirically what's actually happening between users in the AI without having kind of unprivileged or having human data access to those transcripts. Do you think with those transcripts, you can adjust your internal corporate
88
00:33:23.566 --> 00:33:32.678
AI models so that they self-learn against sort of your own organization's way of engaging, or do you think it comes back to employee training?
89
00:33:33.505 --> 00:33:57.147
So we've talked a lot about how important user behaviors are in success and failure. I am very firmly on the side that it shouldn't be on the user to know all the behaviors that they need to exhibit. We should be demanding of the companies that design and build and ship and deploy big AI products
90
00:33:57.147 --> 00:34:19.793
that they are designing not just for model capability, but they recognize that they're designing an interaction because there are actually very powerful things that you can do in product that change user behavior, that change which behaviors they're likely to express. And in fact, how the model behaves changes how users respond. Like that's the beauty of a conversation is like what you say changes how I behave. And
91
00:34:19.793 --> 00:34:41.123
If you make me feel comfortable and safe, then I am going to, you know, when we're shipping and building AI products, we're not just shipping a capability. We're really designing an interaction that happens between the user and the AI system. And to your question about, well, you know, what can we learn from what's happening in those interactions?
92
00:34:41.123 --> 00:35:02.167
I think this is one of the most slept on maybe things, but everybody who's shipping user facing interactive AI or every organization that is using internally some sort of, you know, use interactive AI. You have every session. There's a human in the loop that's doing evaluation and that's the user.
93
00:35:02.167 --> 00:35:32.153
explicitly or implicitly, when they push back, when they say, oh, no, I actually realized this, or when they express frustration, those signals of what they're actually doing in those sessions, those signals are, you know, we started this conversation talking about how do you know what good means? Well, a really good place to start is look at what users are already expressing in the session. Look at what people are already doing in the session. I know that for myself, for my own
94
00:35:32.153 --> 00:35:55.947
So many things that get surfaced in that session are things that define what good means in future sessions. So, you know, the most simplest things could be like, you know, we use this tailwind class rather than that, or we have this convention rather than that. It's not something that the model could have, you know, it's not something that depends on how smart the model is. It's something that's application or business specific.
95
00:35:55.947 --> 00:36:20.753
And those things get surfaced in these sessions. And if we just have ways to capture them, to log them, to actually learn from them, to have them not die, you know, once the session ends, that is this amazing, you know, data asset and feedback loop that's just waiting to be activated. So every organization that has internal, you know, chat logs between with employees trying to do real work,
96
00:36:20.753 --> 00:36:47.753
They're sitting on this incredible latent artifact that's just waiting to be mined for feedback, for these pushback moments to surface like, okay, what does good mean for us as an organization, for our employees when they're trying to achieve these tasks? So maybe this is getting too dystopian or meta, but if this is the way humans interact with models, as we move more towards an agent-based world,
97
00:36:48.563 --> 00:36:54.453
Do the individual agents also face this issue when interfacing with the models and with other agents?
98
00:36:54.790 --> 00:37:22.567
I do think in five years, the vast majority of interactions, conversations will be conversations between agents trying to figure stuff out on your behalf or on behalf of some other agent. And I think those conversations will look different because LLMs are different from humans, but they will still be conversations and they will still have these interactional failure modes and their own ways of going south.
99
00:37:22.567 --> 00:37:38.311
And I do think the danger for us or the challenge is, you know, the bottleneck is not, you know, going and having models generate stuff or having models go do work like they will. They can go generate a bunch of work.
100
00:37:38.311 --> 00:37:58.240
The bottleneck- Faster than I can review it. Exactly. They can be created faster than I can review it. That's the bottleneck is verification. And there are very few domains where verification is something that you can do mechanically, where verification where you have some sort of oracle that can do verification. For most things like knowledge work and most things like complicated things we want to use AI for.
101
00:37:58.240 --> 00:38:15.301
it's more subjective. There's more expertise and judgment that are required. And so I think the dynamic that we should be thinking about is like, well, if it's hard right now in this current moment to verify all the work that AI is doing and all of that output,
102
00:38:15.301 --> 00:38:32.277
Well, how much more difficult is that going to be in five years when you have 10x, 100x, the amount of work that's happening now between agents and the amount of tokens that like, how are you going to verify all of that? How do you know what to pay attention to? Yeah, so what's the answer? I don't know.
103
00:38:32.277 --> 00:38:52.240
I think it's turtles all the way down. You need trusted LLMs that watch the other LLMs and that flag certain things. You need LLMs to help mediate our attention, help us figure out what to pay attention to. Because we're never going to want to take our judgment out of it. We're never going to want to take our human-in-the-loop verification out of it.
104
00:38:52.240 --> 00:39:21.468
We just need to be really targeted and smart about, well, what are the moments when, you know, what are the highest leverage moments where we do need to verify, where we do need to express judgment, where we do need to, to, you know, have, make sure our expertise is expressed. And then how do we, when those moments happen, how do we learn from them? How do we capture from them so that then the system can incorporate them moving forward? And in big Spence case in particular, do you imagine a world where a lot of the usage is
105
00:39:21.991 --> 00:39:42.528
you observing the agents talking to the agents and then allowing. Yeah. That is, we, that's, can you talk me through that real fast? That, that's, yeah, we, we believe, you know, in five years or, you know, maybe less like the majority of right now, what our system monitors and where we kind of, what we mine for quality signals and,
106
00:39:42.528 --> 00:40:12.515
It's interactions between users and AI systems. It's all user in the loop. But even there, you've seen the, you know, every day that passes pretty much, there's more happening on the AI side behind the scenes. There's more tools being called. There's more tokens going to thinking. Like more and more of it is more and more of just like the raw text that you kind of need some system to verify and monitor. More and more of that is happening on the LLM side.
107
00:40:12.515 --> 00:40:28.377
It's very common now to have it spawn sub-agents as part of a single response that it's giving a user. More common for it to have really long-running tasks where the user comes back the next day to see after it had worked for a couple hours.
108
00:40:28.377 --> 00:40:53.099
And we're just going to see that continue. We're going to take the, you know, harnesses are going to be capable of taking on longer running tasks. So you're going to have more thinking happening, more tool calls happening behind the scenes, but you're also going to have more autonomous agents that are just off in the world doing things for you. You know, you're going to have, you know, one agent that's like negotiated, that's representing you, that's trying to negotiate the best price for a plane ticket from some agent that's representing the airline company.
109
00:40:53.099 --> 00:41:09.923
And there's just going to be an explosion of more and more agents talking to agents on your behalf. Or the product that you're shipping is just going to be more and more systems and teams of agents trying to accomplish things.
110
00:41:09.923 --> 00:41:35.877
And so it's just going to like this precipitous trend of the AI just producing more and more tokens, more and more output, doing more and more work kind of on your behalf in the background, in a room that nobody else is in, where nobody can verify, nobody can pay attention, nobody can flag things that might need your attention. That is just going to become more and more of an acute dynamic. And so having these systems in place now...
111
00:41:35.877 --> 00:42:05.105
that let us gain trust, that let us, you know, that direct our attention to the right moments, that learn what are those moments where we need human attention. That's gonna be critical in a world like where you're trying to verify and make use of all the work that's happening on your behalf in the background. In a lot of ways, it's almost like I'm imagining a world where there's just people watching and reading through the discussions between the agents, helping the educate,
112
00:42:05.830 --> 00:42:31.767
Agents, I guess that's the world we're heading towards as you describe it, right? Where you're... Yeah, in a way, we're already doing that on a small scale every time we're in a session with the agent. And what I'll notice myself is AI responses are very long. And a lot of the time, I don't read the full response. I kind of skim around or scan around and maybe read part of it.
113
00:42:31.767 --> 00:42:59.408
So it is already a world in which like our, you know, there's a lot of a lot happening that we're, you know, even in these sessions where we're theoretically fully present, fully human in the loop, even there, there's a lot that no human is looking at. I mean, I had a frightening moment where I use AI for research, like to run experiments, to help do these large scale experiments. And in general, it's been this phenomenal productivity booster for me.
114
00:42:59.408 --> 00:43:15.608
But I had this one scary moment where I just happened to, you know, I was scanning. It was like a very long running 12 hour task that it was doing for me. And I just happened to be scanning through and I expanded one of the thinking blocks. And I was just and something caught my eye.
115
00:43:15.608 --> 00:43:33.142
And what caught my eye was this line where the thinking block said, you know, it had run an experiment and there was a preregistered number of like, OK, if we, you know, this scientific experiment, if we match this, if we hit this threshold, we're going to say this hypothesis failed. If we if we don't, it's going to say that the hypothesis was proven true.
116
00:43:33.142 --> 00:44:01.120
And I saw this block of thinking text that was like, oh, you know, metric is point zero zero zero three off of the cutoff. I will adjust the cutoff for a fair and true comparison. It was totally undermining the scientific integrity of the entire experiment. But like dressed up in this language of I'm being fair, I'm doing the right thing. That's not like like how many of those were happening that I didn't that I hadn't caught.
117
00:44:01.120 --> 00:44:28.475
So this prompted me to do a massive audit where I had it like do an audit of every single research thread that it had done for me recently. And like looking at all of the, you know, all of the thinking texts and finding all of these potential, you know, potential issues. And I think that just really made me, you know, like even me as a very highly engaged user, like there are just things that I wasn't paying attention to that are like wildly important and critical for like how much I trust the output. What does that mean? If you're a big company,
118
00:44:29.302 --> 00:44:48.185
and you're entrusting this stuff and they're just arbitrarily disregarding your goals. How do we view this in the next couple of years of enterprise AI adoption here? It means if you don't have systems that are actually looking at what your AI is doing, if you are flying blind,
119
00:44:48.185 --> 00:45:16.838
then that's on you in many ways. Like these systems, they're not predictable. Just because you've given them what you think are good instructions doesn't mean that they are going to be doing right by your business. It doesn't mean that they know what even what good means for your business. So if you don't have systems that let you actually see what your AI is doing, understand where it's falling short from what your business needs or what your users need,
120
00:45:16.838 --> 00:45:33.325
and then improve on those and measure whether the improvements work. If you don't have those feedback loops, if you're not able to have the visibility to catch failures and hill climb, then you are best case shipping slop and worst case shipping something that could actively harm users.
121
00:45:33.325 --> 00:45:54.908
And so what does that tech stack look like then if you're looking at forward a few years, like either specific companies or layers? So we already see pretty established that, you know, the kind of tracing category. There's some, you know, database where the traces and kind of the raw exhaust.
122
00:45:54.908 --> 00:46:13.268
can be stored. But right now, most teams use those tracing platforms mostly for kind of technical debugging. And I think like what we're now starting to see and you're starting to see that the tracing platforms as well start to move kind of upwards in this direction. But I would imagine that
123
00:46:13.488 --> 00:46:38.041
like this is actually where there's going to be a lot of innovation, you know, a dedicated category pretty soon is then, you know, well, how, what's actually reading all the traces? What's actually paying attention to all the traces, not from a technical perspective of like, is the latency, you know, P90, what we expect, but from a business and product perspective is like, are the user goals being met? Are the business requirements being met?
124
00:46:38.041 --> 00:47:00.974
more LLM inference is going to go to verifying and testing and paying attention to what, you know, what AI products are doing, then those AI products are using themselves. So like we will be using LLMs more to help watch LLMs and help us understand when to pay attention to LLMs than like, you know, the LLMs doing the generation.
125
00:47:00.974 --> 00:47:18.355
Yeah, I think this old Dr. Seuss, you know, book of the places you'll go. This hawk hotcher is like the watch who watches the watcher watches and watch. Who watches the watcher. That's how I'm imagining this. And I think like ultimately the most important thing we have
126
00:47:18.457 --> 00:47:44.832
like on a team and in organizations and just in general, right. And individually is our human attention. It's like, you know, there are hundreds and thousands of things every day competing for our attention and like where we spend our attention that determines like, well, what, where are we, you know, where are we funneling our judgment, our taste, our expertise, our energy, our motivation. And so to me, this is just a variant of that question, which is,
127
00:47:44.832 --> 00:48:10.837
you know, using AI, AI working on our behalf, it's going to demand our attention and it's going to be, you know, the raw scale and volume of it is gonna be more than, you know, any single, more than there is human attention to address. And so the key question that, you know, that we need to answer here is like, well, how, what are the systems that we have that make sure we are paying attention to the right parts?
128
00:48:10.837 --> 00:48:40.520
that make sure that we are spending that attention that we have, that our teams have, that our experts have, and we're spending that attention in a really effective way. And so that's in many ways what I see the Big Spin product doing, what I see our research doing, is trying to push forwards on how do we find better systems to direct our attention on the things that actually matter, the things where we actually have leverage to improve outcomes that our AI products and our AI systems are delivering.
129
00:48:41.465 --> 00:48:46.645
That's a perfect way to end. Thank you so much. This was awesome. Thanks. This was a little scary, but very awesome.
