文字起こし 🔗
speaker-0: you
speaker-1: Thank you guys for coming. We're gonna get started. So my name is Basil, and ⁓ we have Glean's co-founder, Tony Gentlecore. ⁓ today we're gonna be talking about the future of SAS. So yeah, let's get started. So yeah, could you tell me a little bit about your background, the origin story of Glean, and you know how the product has evolved? Yeah, ⁓ that's great. ⁓ It's nice to be here. This is a beautiful, beautiful building. Good to see, good to see everybody had a couple, a couple nice ⁓ meetings, ⁓ like interactions before this. ⁓ let me see. So Glean, who who do who do we have in here? Who's a ⁓ or do we have any any folks that are Glean customers? I know I I talked to at least at least one. ⁓ so pretty pretty pretty pretty new to Glean then. ⁓ so ⁓ we started in 2019. And my background, I spent about I spent about 10 years at Google. About half of that on the web search, ⁓ and then about half of that on ⁓ the Chrome team. I started a team there called the Chrome Speed Team, all about making the browser fast. ⁓ There I met ⁓ Arvind Jane, who's Glean's CEO, and ⁓ he was a distinguished engineer. He was working on making the whole web faster and all of all of Google faster. And I we we worked together in that capacity quite quite a bit. ⁓ he tried to take me with him when he left, he he left Google. I think he left around 11 or 12. ⁓ And ⁓ wasn't quite ready to leave yet at that time, but he started a company called Rubrik. And ⁓ any anybody familiar with Rubrik here? The yeah, so a lot a lot more. ⁓ so Rubrik as you know connects to all of your SAS apps, all of your data. Does back up to disaster recovery? They do more now, but ⁓ at the time that was kind of the core thing. And through that, he kind of identified this this big customer pain point, which is like, hey, we we can't find anything across our companies. you know, we it like you've backed up all the information, can't you do something useful with it? And So around that time, I'd since left Google, done another little startup. We were gonna try to do some some social good. That didn't work out. We returned a little bit of money to investors. I could talk about that afterwards, maybe if that's interesting. But we we had this idea that you should know what your company knows instantly. And and we wanted to we wanted to approach that differently. And we're thinking about the way Google got started. And and Google, when they got started, I don't think it's well known now. But it was really like, hey, search on the web is solved. There's already Alta Vista, it works good, there's there's ads. Larry and Sergei, as we know, like it's Valley Lore now, they have page rank or the back rub and said it's just this really simple idea. The number of links to a page is how relevant that is. You do more relevant results, and now the rest is history, like w one of the largest companies in the world, obviously. And we said, what's the equivalent of that for the enterprise? And meanwhile, our investors, you know, they they said, hey, Arvin, ⁓ here's a blank check because of his reputation and what he had done done before. but they were also saying, yeah, here's here here's a check, but but like work on anything except enterprise search. Like it's dead, people have tried this, all this other stuff. But we were thinking, no, we can do there's there's actually a ton of signals in the enterprise. There's a there's a ton of ways that we can know from all the activity, from all the links about Hey, was this document shared in this Slack channel that you're in? We started to think, hey, we could we could build, we could really start to build a graph about who works with whom, about what's relevant, about what's canonical, what's deprecated, what what's not only like globally on those things, but maybe there's an account executive in Tokyo that has an entirely different mental model of the company from a software engineer in San Francisco. And you know, when people asked things they or or need to know things they need know very different things so we just set out saying hey we want to solve we want to solve that one problem and we got going we had ⁓ one of ⁓ one of meta or facebook's ⁓ original original tech leads of the Facebook news feed ⁓ Vish and we had also a guy named Piyush who went on to form another startup Piush Pradlaka ⁓ called Ida and ⁓ he he was one of the one of the early tech leads on Google search ranking and we just said, hey, let's just let's just focus on this problem. Let's build Google for your enterprise. ⁓ I think back in the nineties one of the H HP execs said it really well, like, if my company only knew what my company knows, we'd be 10 times as productive. And that was kind of maybe maybe I'll pause there for a minute because that was kind of how we got up to like 2019 origin, why did we start Glean? Was it just a total coincidence that GPT2 was launched in 2019? It was a coincidence, GPT2. So so ⁓ it was a coincidence in in that ⁓ we thought this was a really big problem. I think we we always had this idea that it's more, that it's not 10 blue links. So I remember that very first summer, and we started out in Kleiner Perkins basement on Sandhill, Sand Hill Road, and we were we were working there, we got an intern the very first summer, and we had him working on a a Slack bot. That was trying to bring direct answers. But at the time we were using models like like BERT, you know, things that are I I think I think there was a time when that was considered a large language model. And now now you would definitely call that a an SLM now these days. ⁓ but but it was a coincidence with GPT too. And it wasn't really it wasn't really until we saw the power of chat GPT ⁓ coming out and the the rapid adoption that I think I think like maybe there was really a A a a large a a large change in terms of how we were approaching things. So what was that change in terms of how you were approaching things? Yeah, I
Load full transcript
mean, so okay, so we had this thing and it was working really well for for humans to find what they needed, but the vision the vision was already was always more. And at that point we realized ⁓ like as soon as that was out that like okay, the tech is here. So G GPD two, the tech wasn't really ready yet. You know, it wasn't It wasn't until that that initial Chat GPT launched that I think it was there. And and I think we found ourselves like it you know, it was it was business was good. Like we definitely had had good traction and things like that, but not like great. But it was a it was that that was a real inflection point where I think any any startup, you know, th those are thinking about this, you you know, it's like you you need to have a little bit of luck and a little bit of tailwinds. Like you you definitely need to It you know, somebody asked me once, like, is it is it is it all is it all like execution or is it all luck? And it's like it's like, no, it's not one or the other. You you actually need both. And so I feel like we found ourselves in this really fortunate position where LLMs, they're reasoning engines, they're like human brains, right? Like hum human brains are like the pinnacle computer in the in the in the world, like the the best compute thing, but which is amazing, like we're so smart, but we're we're also like really dumb. We forget where our keys were. ⁓ we we hallucinate, we have biases, we, we, we, like, we get all kinds of things wrong. And we need humans, we need context, we need like Google search. You know, I'm kind of using that as an analogy of what it's like when you research and you look, you look something up versus when you just go off the top of the head. And that's kind of that's kind of the same for an LLM. So we found ourselves in that position where we had built exactly what you need to have an LLM. give you the answers that are that are grounded, that are ⁓ cited the that aren't hallucinated. So early on as a customer, like what was I using Glean for? Like like what was like from start to end, let's say like, you know, I start my day, what am I using Glean for like early on? Yeah, so early on. So let's say, let's say you're in the 10 blue link era or you're in the the the basic chatbot era. This is you you connect you connect ⁓ Whatever your chat and communications are, be it Microsoft Teams or Slack or ⁓ email, whether it's Gmail or Outlook or you know, whatever ecosystem, you connect your code base, you connect ⁓ you know, GitHub, ⁓ or GitLab or Bitbucket, whatever you're using there. You connect all of your doc systems, you know, Office or G Suite. ⁓ you you connect your ticketing systems, EndS, Jira, ⁓ whatever else, your CRMs like Salesforce. ⁓ it goes on and on and on. Like today it's ⁓ what like close to I think ⁓ two seventy five or something different connectors that the that that we index really well. So you connect all these as a business and and ⁓ you're you're ⁓ basically like able to quickly find things like in the same way that you would find something in your personal life on the public web, you're you're finding those things across ⁓ across your company's internal knowledge. And so it was a day-to-day productivity tool. Like it at the time it was primarily a Chrome extension that acted as your new tab page. You you open it up and it's like you're you're searching across your company. Okay. So how like early on, how were you measuring how accurately it was pulling in the correct information? ⁓ yeah, evals, that's that that's a good question. So well we we we actually we took a lot we took a lot of ⁓ of inspiration from how how google search worked, you know, so there you measure all the things around around every interaction that you can on on the from the page. Like did did how much did they scroll? What was their dwell time? What did they click? And then when they get to when they get to the the ultimate destination, is that a good click or a bad click? And we would base that on like dwell time. Was it a did they stay there for a long time or did they come back? Did they Refine the query again, or did they stay on the query? And so we would build these evaluation sets. And it was actually interesting for us. It was a little bit different. I remember, I remember when I first started at Google, you walk in and there's this giant board on the wall that maybe people have seen and stuff like that. That's like every live query that's that's happening, and you know, they're just scrolling through the wall, ⁓ you know, where it's all public and we could look at everything. But here we have really we have really sensitive. customers and we can't we don't always have access to those things. So we had to, so it's kind of a different problem where we as engineers couldn't directly go in and like read all that information. It's way too sensitive, you know, and and so from that point we had to develop some some interesting things that would let us like say, okay, maybe I'm gonna make a shadow ranking experiment. Or like what if things were you know we we have some hypothesis about what this was, let me run this. It will run within the environment, I'll get the I'll get the results back. And and look at that. So eval eval was actually a big challenge. I think eval only gets harder as we move forward into the modern age where you're you're taking actions, you're ⁓ you're you're writing agents, things like that. I I feel like it's it's so much harder. I mean, by one, like at least in the 10 blue link era, you always had a click. You you always knew like if they were getting value, they were clicking. But now most things you're just reading. And so I think there's there's entirely new challenges where I mean there's a couple ways around that. shares and copies and you can still look at dwell time and kind of understand some things but I think largely largely e eval eval is getting even harder these days. So I guess early on it was a it was
a search product. ⁓ You just alluded to I guess now you guys are building agents as well or helping companies build their own agents. ⁓ Could you talk about that? Like when did you guys start doing that? Yeah, I think ⁓ so that was probably maybe I maybe it was a year later than like the kind of the you know really that initial like get to chat bot, but you know like twenty twenty yeah I felt like I felt like at this point when when ⁓ LLMs got better at reasoning. ⁓ what was ⁓ what was the era here? Maybe maybe like four one ⁓ was the was the time when you could actually do like some reasoning to take To take an action based on that. ⁓ I mean, it it kind of varies based on how how complex that was. But it turns out like if you're going to take an action, if you're going to actually do some work, it's like that much more important to have the right context that you're acting on, that you're that you're not, ⁓ you know, not not hallucinating, not making those things up. So we we we found our we found ourselves in a in in a value a valuable position there as well, too, as it as it progressed. And I kind of look at it as, There was a time when it was experimental, and now we've moved into the thing where it's like all scale. It's scale, it's governance. ⁓ I don't know what what what what you were gonna ask next, but now it's cost. Now, now tokenomics is the whole thing. It's not like is this possible, but is this ⁓ is this affordable? Is is a is a trade-off there, is there high ROI? ⁓ these these are really the questions. So I feel like we're We're out of the experimentation stage, the can this work thing, and to how do you how do you how do you scale this? When you knew that agents, hey, like these things can actually be valuable for companies, like what were some of those use cases that you thought companies would use the agents for? Yeah, I I guess maybe our maybe like some of our initial conceptions and like what actually gets used maybe maybe it varied a little bit. I guess initially I think we were thinking that it was a little bit more deterministic. You kind of tell it tell it what to do more a little more like workflow based and and ⁓ You know, for instance, this this ticket comes in, you're gonna take this context, you're gonna run some prompt against that against that ticket, try to resolve it, now you're gonna respond back with the with the with with a potential solution or next step. But very that's very much like a workflow. ⁓ and I think I think we had the the hypothesis that agents were gonna look like that. ⁓ today, ⁓ and you know, I like to talk a little bit about like what what I see directly, like what we actually use internally and what works really well. And I'd say the most valuable agent for us now is a is a on-call shepherd. We call it support shepherd or it's really an on-call resolution bot or agent, I should say. ⁓ but this thing, it's just, it's just kind of you can think of it like a claw. It's just kind of free in in in
in Slack with with the context and everything. And these escalations come in or these outages or these issues. And now we're seeing the surprising behaviors, the to be the behaviors that we didn't expect. Which I think is when you like that's when you know it's working really well. So instead of just like, hey, I'm gonna try to solve this and suggest a resolution, it's like, ⁓ this person hasn't responded to the tag in fifteen minutes. I should ⁓ actually tag their manager now. And like it, you know, it's figuring out things with you know, with Glean, it's like connected to the or to to to the org chart and things like that. And so it's able to kind of start to make those those inferences. And we haven't we haven't actually released this thing yet because we're really still ⁓ figuring out like when when it when does it have the the quality that ⁓ that that we're happy with. We want to make sure it works really well, that it really generalizes, it's not annoying people. ⁓ I it's been fun seeing the reception go from I would say internal detractors and doubters to like people just love it now. And it's like saving a lot of time. ⁓ but here here we're We're kind of on that on call agent is a big one. ⁓ TPM is another one that we think ⁓ a technical program manager is another one that we think benefits a lot from having a a lot of organizational context. A lot, you know, this involves a ⁓ you know, really understanding trade-offs, really understanding roles and projects and people and lots of data sources and and also kind of like nudging things forward, you know, knowing knowing when to ping, who to ping, those sorts of things. And then ⁓ a third one we're thinking about is executive assistant. It's very much, very much the the same sorts of you need every piece of context to do to do executive assistant work. So these are these are some of the things that we're seeing now are like way more powerful than a hey, just like reply on a ticket. Not that that's not valuable too. I mean there's a you know a ton of deterministic cases where you do want that repeatability and you want the guardrails, but I'm I'm really excited about the much more open agentic things now. Yeah. It sounds like since since founding that it's been up and to the right. What are I I'm assuming that's not the case. That there's probably some things that maybe you guys got wrong. Like what are some of those things that maybe you look back and you say, ⁓ like that was really dumb. Why were we doing that? That's a great question. So if I look back, one of one of the main things I I I I think is like not betting against the models getting better and better and better. So when I I even was talking about this with the with the ⁓ With the agent thing where we thought at certain points in time, hey, the reasoning isn't quite there yet. ⁓ you know, we should build a lot around around workflows and things like that. And then, you know, come come six months later or nine months later, ⁓ it's
like, hey, the ⁓ the you know, the the ⁓ the reasoning is so good that anything that you come up with as a human that's like a deterministic workflow, ⁓ you're actually limiting yourself more. And so the kind of the guardrails and what you wanna what you want to encode it a as as as a human, I think changed really quickly. So I would I guess I guess the the recommendation would be like, ⁓ you know, make sure may like d don't don't build for the model of today, like try to extrapolate that forward six months, nine months, a year. It's really hard to extrapolate into the future, but I you know, we don't actually know the rate of change and what's gonna happen. But I think I think we can all assume more and more powerful. And so I think I think when when building like it takes time to get things to market and things like that. And you know, always always ⁓ you know, predicting, looking where the puck is going, not not where it is. So what would you say, yeah, where is the puck going? ⁓ like what are the agents bad at right now that you think in a year from now, hey, like they're gonna probably be pretty good at that? Yeah, I mean I could give like trite cliche ⁓ you know, like, hey, you know, it's gonna be this much more powerful and stuff like that. But like when you say what are agents bad at now, ⁓ it's actually it's actually interesting. We did this, if if you if you ⁓ look up later this ⁓ work AI index that ⁓ we released, ⁓ big study ⁓ last week. ⁓ we we we ⁓ the the takeaway is that like some of the problems right now are what I would what I would call like ⁓ or what the report calls like bot sitting and then bot shitting. I don't know if we're allowed to say shitting, but yeah. ⁓ the the the bot the bot sitting is like we were just looking across across productivity and serving a lot of enterprises and things like that. And if we just limit to personal productivity level, you know, average are coming back saying, Hey, we're s we're saving about eleven hours a week ⁓ per per employee at a at a personal productivity layer. And this isn't with Glean, this is this is like Asking them about AI in general, right? ⁓ this is a like a industry survey, not like a glean survey. you know, performed by us, but not about our product. And however, with that, they said, hey, now we're spending an additional six hours a week just kind of reprompting, like re-reworking with with this, like like reviewing the output, this other things. And we call this, we you know, they we call this like bot sitting now, where like You know, some of the I think I think people maybe, maybe I mean, certainly I as a as a builder, I feel like some of the fun parts are gone and now now some of this is like ⁓ you know, it's not quite the matrix yet where we're plugged in as a battery. And, you know, but we're definitely doing a little more a little more rote work on just on just babysitting these things. And so I think there's a lot to build around kind of the harnesses and how we interact with these and ⁓ and in the productionization of them, ⁓ the reliability, the the verifiability and trustability, observability, all of these things I think is a big thing that that needs to improve on agents so that they're actually just like fire and forget. spending all your time just like babysitting them instead of doing it yourself. So that that's one thing. The bot shitting is actually worse, which is which is, you know, when you're when you're just a producing AI slop ⁓ with it. And I I think I've I've I've started I you know this is this is a problem where I think everybody thinks like AI, or not everybody, I think maybe there's a trend or some people think that AI is just automatically everybody is doing better work. But I look at it more as a force multiplier. And that force can be a force for good or a force for bad. Like if you know, if you have good taste, you have good judgment, ⁓ those sort of things, like that's multiplied. ⁓ if you have bad taste, bad judgment, that like that sort of thing is multiplied. And so I started I I I you know, I I'll go I'll switch back from like what people were reporting about the ⁓ about the consequences of this and just talk myself personally. I'm now getting and reading more documents than ever in my life. And half the time, like I'll poke at the thing and I'll ask about this. And they're like, ⁓ well, I didn't actually mean that part. Like the the LLM just wrote that. Right. And I I think, yeah, everybody's laughing. So like I think I think we all have this. And I'm like, well y you know, don't send it to me that like don't make me review it for you. Like you still need to to like these are your tools. ⁓ you know I compare it like if you were a lumberjack and you're really good at using your handsaw. And all of sudden one day you get a chainsaw and you start like cutting down the wrong trees. You don't say, like, it wasn't me, the chainsaw did it. It's like, no, that's your tool. You're expected to be a master of it, to learn, to learn to use it well. ⁓ one of the more, one of the more extreme examples I did hear from a customer. ⁓ this is a very AI-forward startup, ⁓ you know, to the point where they were telling me that they have ⁓ you know, there's the token maxers out there using, using, you know. 10 or $20,000 a month kind of thing, which I've heard a lot for engineers, like we have that for engineers, everybody does. But they were saying, hey, we've got business users in that, in that category of spend per month in in a good way, and they're doing all that, but they actually have a rule in their company. They're that AI forward, they have a rule, no AI writing. All documents must be written by humans. And and that and I thought that was really interesting. We haven't, we haven't applied that internally ourselves yet. But I am starting to see, you know, judgment matters, taste matters, and and I think I think there there there will be a little bit of a trend or a resurgence of like
what what sort of things should you do? Not just like token max everything, but what makes sense for you to do ⁓ and and what not. And you know, when are you when when are I I think I think the other thing like making sure making sure folks on your team and things like that understand that if ⁓ You know, if they're putting out AI output, that that's still their work product. Like it's still it's still their reputation on the line. Like when I see a a as an engineer and maybe I see a 3,000 line vibe coded PR and there's all these problems in it and things like that, it's like, well, you can't just say that that was AI. Like that that's you. You're producing it. You can't like put the burden on somebody else. So I think you know, there's a really long winded answer to like what are agents bad at? But you know, right right now it's like, you know, making sure the human judgment is there, that that you're applying your tastes, and then also making sure that we're not just spending all of our time on the road like babysitting work on them as well too. So are you limiting the amount of ⁓ tokens or I guess like the cost for engineers to use LLMs? ⁓ is it on limited budget? This is a this is a really interesting one. ⁓ I think every company in the world is struggling that with this right now, and it's actually really exciting to me. ⁓ we, you know, I spent like there there was this real push to tell everybody use this as much as possible. And in fact, like the only the only metric that that companies really had was spend for a while. Like how much is everybody wanted like your a your employees to be AI forward. Why did they use it a lot? And then you assume that, well, the only way I can tell that they're using it is how much they spend. So there's this real push for this. And now it is out of control everywhere. Like every enterprise we I talk to, every time I sit down with like a CTO or CIO or something, they're like, actually this was like back in April. They would say we've like two X'd our planned token budget for the whole year, or like, you know, and we're only a quarter in. ⁓ and and and and so I think, I think that that like flipped so hard where we pushed so hard for it and then all of a sudden it it was it was out of hand and now we're trying to say look we don't want to hurt anybody's productivity. ⁓ we want everybody to use this as much as possible, but now we need to make sure the ROI is there. And what I've seen is that you want to look for you want to look for waste. So We went through our own AI spend and actually found and so it's out of control, like at Glean internally, it's out of control. I won't I won't say numbers, but like a good portion of it was automatic ⁓ automated agents that were running in the background. And when we actually audited this thing and looked through it, there were so many of them that could just be turned off. Like they they you know, they they didn't actually need to to be there. And so this is one thing that we're, you know, as a product, like we had to build for ourselves. To make sure the observability was there and the g, you know, that you can actually see how much these are well, how much they're running, how much they're consuming. And ⁓ you know, we're starting to work on, you know, or kind of org level budgets for the for these things as is as well too, that that these can be controlled. But the other thing on the on the personal side, the other thing we found is that you know, everybody wants the best, right? Like you don't your time is valuable, you don't want to put up with a sonnet response, you wanna Opus response or or or whatever. So I think everybody now just sets it and forgets it at 4.8. And and ⁓ but not everything needs that. Like it literally sometimes is like rebase my PR and and like that doesn't need 4.8, right? Like go back to auto mode or or or whatever for that for that query. And and and I think A lot of the a lot of the tools and stuff they're at the point where they're saying, like, all right, what is our what is our tool to control spend is to set hard caps. And but ⁓ but a hard cap in my opinion doesn't work because it doesn't change behavior. Like you hit your hard cap on the 20th of the month, and then if your admin doesn't approve like raising that cap, like you may as well just go home or take vacation for the last 10 days of the month, right? ⁓ and so there's really no option. Like they have to keep they have to keep. ⁓ upping upping your cap. So what we're doing instead, you know, we're we're now ⁓ you know, you ask about the the evolution of Glean, you know, really the context layer has turned a lot more into a to an MCP gateway and now our we're very model agnostic across all the all the families and then all of the ⁓ all of the open ones as well too. We're exposing that as an LM gateway that you can You can use across your across your coding tools, but then you know across across Glean as well too. Like hook these up to whatever tool and and Glean. But what we're finding is actually effective is is more of a you know, the limits on the the expensive models that are much more granular, like throttles and daily limits that change behavior. You know, it's like it's as simple as like, hey, you're done with four or you're done with Opus for an hour, you know, try like go go back to Sonnet a little bit and then try again. try again in an hour. Like it sounds, it sounds, it it sounds simple, but I think if you're ⁓ I think there are actually incentive issues. Like if you're if you're a company and your your whole business model is based on inferencing, like you may not actually want to change behavior. The way we've been approaching it, what we've seen effectively internally and stuff is is is going about those ways to actually change behavior. So that people are being ⁓ being thoughtful about what they're using and and and ⁓ and have real-time feedback, you know. I look at it more like how your your parents
probably told you as a kid, like, you know, turn the light off when you leave the room, you know, it costs electricity. And I think like nobody's been told yet, like, turn f turn opus off when you don't need it. I feel like every week there's just you know, new stuff coming out all the time. There's like MCP skills, like sub-agents, blah blah blah. How do you guys decide that, hey, like this is something that we should look into and hey, like just you know, that's noise. Let's not like waste time looking into you know agent teams and blah blah blah, whatever. Like whatever the new thing of the week is. ⁓ for us as a company, we try everything. ⁓ and and we I mean, I I don't know that this is what I would encourage for everybody to just have like AI sprawl, but I do see it a lot. I think AI sprawl is this this huge problem. shadow IT shadow AI I should say and like AI sprawl. But as a company that's building this, like we really have to use everything. So we use it, understand it, ⁓ all of those things. ⁓ and then you get a feel really quickly when you're when you're in the you're in the heat of it. And like skills skills are skills are not noise. They're like an insanely powerful ⁓ primitive that I I you know I I I don't think I don't think that'd be con controversial right now. Yeah. Yeah. So let's switch to the business mode. ⁓ so on Twitter, I feel like there's a lot of people saying that, hey, there's no point in building any new product because the AI labs are just gonna eat you in a couple months. ⁓ like what do you think about that? No point in building a product. ⁓ yeah, ⁓ I think I think it's important to think about the fundamental value that you're that you're adding. I do think that I mean we've already seen it, like certain things that are ⁓ you know, the the the pejorative is a wrapper, you know, we call it a wrapper company is like the pejorative way to put it. And then I don't think anybody would say like, ⁓ this is a rapper. But I do think I do think it's important to think about the core value, the the the core mode. for You know, for for and I can't say what that is for every for every company. you know, for us, I think there's I think we're just finding again and again and again how much value there is in a context layer that's connected across like at clean, a context layer that's connected across everything at your company. ⁓ we're we're finding that with the tokenomics thing again here, ⁓ Like MCP is a pretty is a is a pretty great ⁓ it it's it's a pretty great trick for getting connectivity that works at a certain ⁓ to a certain degree. ⁓ however, like I think the the best part about MCP or the it would be if you're the one selling tokens. Like it like MCP is great if you're selling tokens. ⁓ I say that because Way it works is like, okay, so you're using you you're you're pulling on all this context ⁓ through through it, but you're using like fundamentally bad retrieval and bad search. And so what what does that mean? You
might have to like if there's a certain thing you're trying to see, you might have the the LLM itself might have to come up with all the synonyms, all the stems, and like issue multiple queries. So you might have You might say, like, ⁓ I need to I need to search 10 different ways in this in this system. You may have to search across 10 different systems because you don't know where it is. And then for each one, like the results retrieved may not be good. So you you ⁓ you have to retrieve a large page of results. And ⁓ it it's it's it's great if you want to fill up your million token context window and you're happy to pay for those those million input tokens, like you're gonna get it done. ⁓ we we re we we released some some ⁓ benchmarks recently, but we're seeing we're seeing something like l ⁓ l less than a third of the token usage on like on on like across our across our our query set just by having like at the you know at the same quality level, same model, just a third of the token usage from having having like the right context instead of having to to stab around in the dark for it. and you know I guess I I guess I well That's a long answer to, you know, your original question, which was like w ⁓ w you know, what do you think about not building anything anymore? And I was I was just talking a little bit about, you know, I think I think for for us that's a context layer. There's a lot of benefits to this. I think there's a l ⁓ there's a lot of things that are that are not possible without like what you know, we talked about the efficiencies of it, but there's also like you get to a certain point, you're hitting rate limits and things like that as well too. So ⁓ that's what we're considering or what we're what we're considering right now. ⁓ and I think I think it's worth just you know ask asking what you know what is what is the next step on this? You know, we talked a little bit about seeing where the you know the puck is going. I remembered what I was gonna ask before. it's kind of related. So do you like how do you guys think about training your own models for you know using on in the glean product? Yeah, training your own models, this has changed. it's changed back and forth. This was another thing. I was almost gonna say this when you were asking about something I regretted. So there was a time where we took one stab at fine-tuning, and I remember we spent all this money. We got it to a certain point, it was better than the current model generation. And then by the time we got it out there, like the next model generation came out, and like the non fine-tuned one was better. ⁓ however, open source has completely changed that now. And ⁓ you know, what we're seeing now with ⁓ Nemotron ⁓ from NVIDIA especially, but also ⁓ the the Chinese ones are more controversial in the enterprise ⁓ standpoint, but also like G GLM and Kimmy and stuff are are like insane quality wise. And so the approach we took, we actually now ⁓ you know, the majority ⁓ by
by default, the majority of glean traffic is using model called that we call Waldo. We name we named it mole you know, it's about searching or finding, right? You find Waldo on the page. Like you might have to be a little older to remember who the red and white striped guy is. But ⁓ that's a that's a fine-tuned version of NemoTron. And we fine-tuned it on like one principle, one principle only, is like know what you can handle and know when to escalate. And so the whole name of the game here is like run run a bunch of the a bunch of the workload at a tenth of the cost of of you know the the the larger models. But no, like as soon as we get to the point that we're you you know, with with with Glean you can do all the the the co-working things as well too, you know, slide decks or iterate on a dock or you know take take actions and things like and as soon as we get into those sorts of things, it's like, well, yeah, if you're making ⁓ if you're making a live interactive dashboard that's querying Salesforce and Zen Desk and all this other stuff, like yeah, max it out. That's when you know you call call opus or you know, Fable ever comes back, call call that one. ⁓ but ⁓ For us now, fine-tuning is like the name of the game and this is how you this is this is I I I think I think from a from a cost perspective, like I we're I'm I'm super super bullish on on on open source right now. You know, it's kind of like call the call the call the call the premium ones when you need them, but beyond that, ⁓ there there's ⁓ you know when you when you when when when you're using a model that's In the inferencing is done by a company that's not having to balance the other half of the balance sheet that was burnt on the pre training with their inferencing margin. The margin is just like the you know, the inferencing is just so much cheaper. It's literally order of magnitude. So ⁓ very bullish on on fine on finding. Why are enterprises not okay with using a Chinese open source model as like a base? It doesn't make any sense. Like yeah, yeah. I think I think I think it's FUD. I think I think there's two two there's two things. ⁓ I think largely it's it's like this idea that it's it's not safe or something something like that. But I think it boils down to two things. One is a misconception. I think that there really are people out there still who think like Chinese model means Chinese in it means inferencing in China, like they don't want their data to ingress. But like so that one is easy to clear up. It's like, no, this is running in on bedrock or on vertex or you know, where or fireworks or whatever. ⁓ that's easy. The second one, I think the other the other objection more comes from the public arena, which is like, hey, was this trained with something like the ⁓ you know, take take like the like the Russian election interference thing? And like was this trained with some intent like that, you know, and like yeah, maybe you care about that if you're running a social network or you know, some public thing. But when you're taking the model and you're distilling it and you're you know, like you in and you're like deciding Whether you're, you know, the ticketing system to call is Jira or Zendesk or something, like all of that stuff, like even if you subscribe to that, and I'm not saying that I do, but even if you thought there was some malicious thing, it it it's just not there. And then but we do still see that a lot where there's like enterprises that say no, no Chinese models, like we we can't have that. But right now it's just kind of like I I don't think this is gonna hold more than a couple months. Cause as soon as people see their token bill and then you tell them, It's a tenth of the token, like immediately, even if there were principles or something, like they're out the window. It's the same as like, you know, where, you know, when when you know people say like or we should take like go back decades when people said like buy American and then ⁓ and and but you know, it's when Walmart is gonna sell something at a tenth the price, like you're going to Walmart, you know. And and ⁓ I think I think the same thing happens to models. One interesting thing is actually is actually like the national sovereignty though. I think to to me that's like the more interesting story of will there be ⁓ will there be a similar backlash against US models in in Europe and things like that. You know, maybe I shouldn't like get too geopolitical in my commentary, but I think that's where you know I've definitely ordered a little bit more popcorn for my watching of these events to unfold. Also random, but ⁓ I think back in 2024, apparently like Sam Altman, like he named Glean as a company that their investors should not invest in. Like what was that all about? Yeah, ⁓ I I I don't know. You tell me what it was about. I I will say that we ⁓ we took it as very flattering ⁓ at the time when you know I think they were they they were, you know, 50 X our size or 100 XR size or something like that. And meanwhile, like the, you know, one of the major players in the space is is is worried about our our our little startup ⁓ to to the point where they want to make sure nobody else invests in it. ⁓ so I to ⁓ I was happy like ⁓ yeah ⁓ to to make you know make somebody like that ⁓ a little afraid of ⁓ what we're doing. All right so let's ⁓ switch to org structures. ⁓ how has Glean changed the way that you run your org structures maybe compared to companies that you've worked at previously? So is there like a difference in the org structure compared to previous companies you've worked at? You know, maybe like the teams are smaller, maybe you know, you don't need as many PMs, blah blah. Yeah, ⁓ you know, we haven't actually started changing our org structure, ⁓ just transparently. We I would say the roles have changed. The roles within this. And I really I I say this all the time, but like really, really soon, there's not gonna be like product
manager, product designer, product engineer. There's just gonna be product builder. ⁓ I know that like even even large companies like Google, I think has this as a new a new job track in internally now. ⁓ that they're that they're experiencing that that they're experimenting with. ⁓ and certainly now like more so than ever, we've kind of switched to a model where we're doing a lot more V teams, a lot more virtual teams, and there we we always want to name one DRI, one directly responsible individual. That way we don't get this like, hey, three people kind of blocked on each other and each responsible for the thing. And usually we take, you know, sometimes it's the sometimes it's the engineer that's the most senior, has the most clout, you know, that sort of thing. And they're getting, you know, you know, of course they respect the product manager's opinion and the designer's opinion and things, but they're they're really the one the the one responsible. And then sometimes it would be a product manager or design. It kind of depends a little bit on the thing, but we're definitely seeing ⁓ now all of all of our product designers are writing code, not not directly, they're but they're make they're making PRs, you know. ⁓ and then ⁓ I would say more and more projects are going end to end with just ⁓ with just an engineer. Didn't you know there's no more like waterfall. ⁓ not that we ever did waterfall really, but you know, now now like that that that PRD stage and those designs, you just go s you just go straight to prototype, you see if it works, you dog food internally, you build it, you see if the team loves it, and then and then you get it out. So I would say I would say like Or org structure hasn't quite changed yet, but certainly certainly functions ⁓ feel roles and functions feel entirely different now. I've heard a few people say, like, ⁓ let's call these product builders, but I feel like there needs to be a sexier name. I love calling myself a builder. I I I I I identify with the term a lot. ⁓ but yeah, I'm all I I don't know, is is is ⁓ so what did we have before? Manager, designer, and engineer. They're not particularly sexy. I don't know. It wasn't like like I don't know, we could call it product rock star or something. But yeah, yeah, maybe. ⁓ what do you think work structures are gonna look like a couple years from now? Like a lot has changed just in the last 12 months. So what do you think they're gonna look like, you know, in a couple years? I'm trying to think where where org structures are going to go. I I I I I can I can foresee some change the the the org thing, how people are going to relate, I'm I'm not actually sure I how that's going to go. There there are there are some things I think I think will change. Like ⁓ you know, the way the way we work on some on some time horizon. I feel like everything is is is voice driven. You work from wherever you are. You could be on a like you could be on a trail in the in the in the Sierras
and like with Starlink, you're you're fully connected and everything. You're talking, you're talking, you're doing your work, you're able to, you're able to interact with every system, like write code, like get whatever you want. ⁓ and then and then what does that mean? What does that mean for the office? If you're using more and more voice, like ⁓ office, I think starts to look a little bit more like maybe a business class lounge than a open row of desks, you know, because actually like the open like the open desk row is actually like not good for like it's actively not good if you're if if you're working in this way. But I think I think in this in in this way, like whatever screen you're nearby, be it like a small screen like this or a larger screen like that, I think it becomes about presentational for you, hoping you understand the responses and the work that that you're doing and that you're getting back. But I r I really think it it becomes more human. It becomes more, you know, the like we already have the you know, there there's there's there's HCI, like human computer interface, but like we already have the best human to human interface. Like it's it's conversation and then and then we we use illustrations, we use graphs or charts or materials or things like that to help explain our ideas better. And I think that's a little more how we'll we'll we'll interact with computers in ⁓ in that way in the future. And I think it'll have a Have a lot, a lot of implications. ⁓ but the you know, where where where does that mean for for teams and stuff like that? I think maybe it's just like sh you know, I I think it's I think it's really hard to say like where they're gonna go and how they're gonna shape up. Like, you know, there's kind of the Doomer way of thinking of like there's no jobs left to do. I'm more of a maximalist of like, you know, the comp the the the ones I think are gonna go far, the ones that say, like, look. Pie is getting bigger. There's there's more things than ever before. I think that taste and judgment matter more than ever before. ⁓ If I'm worried for one thing, it's it's I'm not sure how the bootstrapping is gonna look for new grads. Like the things that LLMs are getting better and better and better are the things that look like entry level ⁓ in your career. And then the things that are more and more valuable are the the the things that like you've learned from some mistakes and things like that. And I'm sure we'll figure it out. I'm sure we'll figure out like what it you know how do we how do we train better? How do we or like what, you know, that there will be there will be an adjustment. But that's that's the one I'm I'm just trying to figure out is like, you know, ⁓ how how how do how does that how does that start to shape up? Like how do you how do you get to the point where you have the the the judgment and the experience to give the LM. It's more like everybody's it's more like everybody's a manager, right? And a manager takes, you know, a certain level of experience before you would give to that person. And so we'll we'll have to figure it out. I have all night. So Yeah, I guess my next question was, yeah, how do you think software engineering is going to evolve over the next few years? A lot of people that I've been talking to are saying, I mean, these are like really strong engineers, and they're saying, Yeah, I haven't looked at a line of code in a couple months. And that's kind of crazy. Yeah. ⁓ I like the software engineers who say I haven't written a line of code in months. ⁓ the ones that say I haven't looked at a line of code in a couple of months, ⁓ I don't really want on my team. What I mean by that is there's there's there's There's vibe coding and there's AI coding. And I think I think ⁓ what we're you know, I think that the same things that make a code base or an architecture or an API friendly to a human that makes your your your code base maintainable and safe and quality, that's the same things that make it that way to an LLM too. Like they're basically reading that too, and like, yeah, you know. If you have this spaghetti mess that was put together, like yes, you could throw like many tokens and much re like much ⁓ reasoning at that thing and and it would probably get some result. But it's gonna be way, way better if you if you have the same kind of APIs, encapsulation, you have the same you you have the the same good software engineering principles that that make a code base good quality. I think also mean that like when LM looks at like that, it can stay within a ⁓ abstraction and not have to understand all the leaky abstractions through the code base to to make the right decision and stuff like that. ⁓ And so yeah, I think maybe this gets back to the bot shitting or the AI slop or whatever where I'm I'm pr I I'm pretty I'm pretty I I I still like there's there's a there's a there's a real place for for vibe coding. There's a real place where you're prototyping or you have things like that. There's another stage where like you know that this is a big production system that many people have to work on and that is gonna be around for a while. And I think code quality matters. And I just see all the time where it's it's like so easy for the LM to go off the rails. ⁓ I've seen, I've seen it like, hey, there's a there's a you know, thousandline PR where like changed all these call sites and everything and tried to do something at every call site. And then like a human looks at it who actually knows the code base, and they're like, ⁓ no, this is a one-line change at the at at the at the at the function level, not at every call site level. You know? And I j I just I just see those things and I really worry with the like I don't I I don't look at code anymore. How much of your code base would you say is written by agents? ba based like most. I I mean when when I when I say that, like the the bottleneck has shifted to reviewing the code, not
not writing the code. ⁓ I think hand coding hand coding's not really a thing. It's just a different it's just a different level of abstraction. ⁓ I I mean I could just repeat and stress that like I think I think the reviewing the reviewing and the understanding and the judgment on that is still ⁓ absolutely essential. So no code review agents? Well the they kind of have they they kinda I mean they they kind of suffer from the the same like if it didn't I mean don't get me wrong it can help but like if it didn't figure out the pattern like if if one didn't figure out the pattern then the one reviewing it like why would it figure out the pattern? ⁓ so I mean co c c code review agents help. Like we use those as well too and you know I I am seeing the thing where like It found a missing security issue or something like that. Like there is the adversarial thing there. But yeah, I don't think it replaces like someone understanding this yet. I don't know. Maybe I should take my own advice and be like, hey, next year ⁓ it's gonna be better than than anything we could ever possibly come up with. But we'll we'll see how that how how that unfolds. All right. ⁓ last section, SAS pocalypse. So Satya Nadella ⁓ says that hey, like any app that's just a cred database ⁓ with some business logic, like that is going to be a dead company. Like do you agree with that? The SAS apocalypse is real. It's real in the following way. Sixty-five million years ago there was an apocalypse, right? There was an asteroid that hit, wiped out all the dinosaurs, right? I mean, did it? Like dinosaurs are one of the most successful species on the planet today. Like ⁓ birds. Like birds, birds literally are dinosaurs. They're everywhere. Dinosaurs, dinosaurs, they changed a lot. ⁓ the point I'm trying to make is that it it it was a forcing function, the asteroid was a forcing function for for rapid evolution at a pace that never you know was wasn't there. I mean there were other extinction events like that too. ⁓ but ⁓ you know we wouldn't be here if it wasn't for that. ⁓ you know mammals wouldn't be here in the way that we are now, if it wasn't for that. And so I think the SAS pocalypse looks a little more like that. It's a it's a big evolution. This is ⁓ it's gonna look nothing like it looks today. ⁓ I I do I I think strong systems of record enforcing certain business logic, enforcing ⁓ certain ⁓ you know there's a lot of things on compliance and audibility and you know th and and connection to other things. And I think I think there will be strong need for systems record, you know, to go. But I think, you know, what I was saying before about like working with your voice and generating UIs automatically, I think, I think over time we'll we'll we'll like it there'll be a little bit less of like, ⁓ I have to navigate into the CRM and enter things myself and use that UI and stuff like that. So yeah, SAS pocalypse is real, but ⁓ you know, it's an evolution. It's not a it's not a ⁓ it's not SAS Armageddon. So I guess one of those things that is changing a lot about SaaS is previously SaaS was on a you know priced on a per seat basis. And I think is that how Glean is ⁓ priced right now? Yeah, we do we do a a ⁓ a platform access fee or seat. That's that's still there. We want that per user, and that's actually where we capture a lot of our value, but then we do consumption on tokens, and that's where we That's where we pass it through. And I think everybody has to do that because that's the that's the thing. You can you can scale this to any any level. Like you can spin up as many agents as you want. It's just like fundamentally it's a consumption-based product. It's not ⁓ it'd be like it'd be like asking if the electricity for this building should be a seat base. It's like, no, it matters like if you're running generators and refrigerators in here or just a couple of light bulbs, right? Like you know, and and I think that's the way. ⁓ that that's the way ⁓ that the that that's the way everything is now. So yeah, we've you know, we charge charge by the token and charge but we try to just pass that through. And we want to make that as cheap as possible. We want to bring in the, you know, like I said, bring in bring in the weldos, use as much open source, like minimize the context, you know. We're we're trying to ⁓ save that, like make the make it so you can use as much of it as possible and also govern it in the best way. Some companies like Sierra and Decagon, they've been charging on resolved tickets. What do you think about that kind of pricing? Like pricing on outcomes? I don't think it's gonna hold. and the reason the reason I say that is it's kind of like ⁓ you know, the analogy I made or the analogy I was thinking of is like, what if ⁓ take like Photoshop? If they were to say, what is the outcome of the thing you produce? Like, hey, you produced a billboard and it generated this amount of revenue, so like you have to you have to pay us it in like commensurate to that marketing campaign. Like it just doesn't hold, you know, you would say like, no, there's a there's like a fixed value and I and I used it. And I think if you're charging outcomes, it's it's assuming that you're the only one who can get that outcome and that, you know, I I would say like at that point, like if I if I was being charged at like a level that's outcome based and and then I could like build it myself, but for way for way cheaper. And you were talking about how it's much easier to build things now and things like that. So I I don't know. I'm not up, I'm not, you know, I maybe maybe I'll be wrong. Like if we if we do this chat again, you know, maybe in a year I'll come back and I'm totally wrong. I just don't see the durability of outcome base. It seems like it can always be undercut. All right, cool. Well, that's all I had. So I think we can leave it to the audience for audience questions. ⁓ we should get you guys a mic though, so I'll do that. Hey
Tony, thanks so much for coming and speaking with us. you mentioned that people Are still accountable for work for their work and their quality, right? Your reputation is attached to it, which totally makes sense. ⁓ but I wonder if there is a new trend that is emerging where people are unable to keep up with AI-produced content or code. Like it's just hard to efficiently go through it and actually just spend the time reviewing it and being able to error catch, right? Like catch those errors. So do you think there needs to be another layer that it needs to sort of evolve, like a logic layer almost that ⁓ sits between the AI-produced content or code and use and essentially condenses that information like a chief of staff would for a president. Like, yeah, it's really it's really interesting. I mean, the the the joke on this is that like I write three bullet points, I pay an LLM to convert it to a letter, and then somebody on the other end like pays an LLM to convert it down to three bullet points again. And I feel like, yeah, if I was selling tokens, I would really like that world. ⁓ but you know, I kinda wonder like you know, like the company that said, Hey, don't do AI writing, you know, should we just all say, Hey, should we communicate in the three bullets again? I I totally hear what you're saying. I it like i in and and but I I'm not I'm not sure I'm not sure how it how it how it act I think I think this is a thing like we're you know, we're all as a s as a society, as companies and things like that gonna have to figure out how we how we work with this in a in a sustainable way. But I'm not I'm not I'm not sure we all wanna just like ⁓ we wanna like token max everything and build up inefficient layers on on that. You know, it could be I I I c it it might it might be unavoidable. You know. ⁓ so yeah, we'll see. Thank you. ⁓ so for coding agents, ⁓ clock code is heavily subsidized. Do you fear that the same sort of dynamic is gonna play out ⁓ for Chat GPT enterprise? ⁓ and how do you feel about positioning yourself against that? Well l let me see. So for for me, I don't I don't feel positioned against it. I mean, ⁓ so Glean Glean is a is a pro is a like a user-facing product and a platform. ⁓ a lot of our customers they use ⁓ they use Cloud Cowork or Cloud or Cloud Code or ChatGP Enterprise and they hook up the you know Glean ⁓ MCP gateway or or LM gateway and then we see like the cost benefits, ⁓ the the the the the quality improvements, ⁓ thing things like that. ⁓ so you know when it comes to the like token subsid subsidies and stuff like that, for me like Also, what makes me so bullet. You know, I talked a lot about open source and and I and and you know for me the durability is just really like at the end of the day, if you're doing all the pre-training and you're lighting you're you're like incinerating cash on that side, like it forces you to to have to do more on the right side. ⁓ I was actually looking at it as ⁓ you know, I saw the press too where where they said, Hey, ⁓ ChatGPT is gonna like slash their prices. ⁓ either model prices, right, to compete with anthropic. And I saw that as a really, really good thing. So for me, the only the only ⁓ the only AI future I don't want to live in is one where like one company controls the world like a like a monopolistic way. And as long as there's as long as there's there's there's like healthy competition and there's back and forth ⁓ and and there's not like lock in then I'm gonna I'm gonna be like really happy with that. So I I I I like the ⁓ I like the competition, I like the back and forth. To me it means like there's that much more reason that now, you know, a a customer is gonna say like, ⁓ gosh, we we we we we totally locked into anthropic and everything and now now open AI is cheaper and we we want to use those models instead. And so I say like okay yeah we've built the model agnostic layer and ⁓ you know switch switch workload over to this model. or or whatever whatever's like dial in the cost performance you want today or tomorrow and don't don't like lock into one. Hi ⁓ Get. I do have a question about context layers. Do you believe that context should be interoperable? Because like right now there's many company brains, second brains, everything is about context layers. But at the end, when I'm thinking about like enterprises, they already have like data breaks, snowflakes where the real data is. And for me it's like for enterprises they own the data, but context layers, it's not like do you believe that enterprises will own the context as well as a derivative over the data, like using ⁓ like process mining techniques and like building this context. Yeah, so when I say context layer, I should break it down a little bit. There's there's structured and there's unstructured data. I think I think Snowflake, Databricks, ⁓ yeah, that's where the structured data lives. You know, the ⁓ but I think the majority of the context is on un unstructured data. And you look at all the stuff we talked about, communications and docs and things like that, those don't live ⁓ in there. And this is when you're when you're really understanding you under and under like understanding how to optimize your your business and and you know, be like much more productive. Like when I say context layer, it's really like the most powerful things come when you're blending the the structured and the unstructured ⁓ data. And then I believe very much that it should be interoperable and that companies should own this. So when we you know we we we look at this as a layered system where we we put I mean I'm not trying to do an ad, but I'll at least talk about how how we do it. It's like The each one of our customers they own and their tenant, we let them pick their cloud. We support all the clouds, right? We terraform into that. They own that. They can lock us out at any time. They can own the encryption keys, you know, give that to us or not. They can use their
LM keys or our LM keys. Like we, you know, we don't care about that. We have our ⁓ protect layer, which is which is really important for like ⁓ finding overshared content. Once you once you do this, like ⁓ all the time like AI is coming for your data. If you have some like security through obscurity, it is gonna find it. And so we do all that remediation of of things that are overshared as well too. But even that, you know, if you want to swap it out for Palo Alto Network's version or Sentinel One's version, swap that out for that. ⁓ of course we we connect every app, even even back in the search days, there were customers that were like, ⁓ yeah, we were trying to migrate from this. I at the time I remember it was like Quip and there were companies trying to might migrate off Quip. to to like say Google Docs or Notion or something. It's like this made it easier to migrate, you know. And so it's all for me it's always been about like that openness and and letting letting like empowering customers to like not have lock in to own their data, use it how they want. We, you know, we expose everything via API and we we actually treat now, you know, when I talk about third ⁓ you know third party non-glean assistants or agent frameworks or something, we treat those as like peers to our our first party ⁓ UI and assistant. We look at that as like, of course, you can't build any sort of platform or or context layer or something without building the hero app yourself on top of that. But we don't look at that as a necessity. We have really large customers who like don't even know they're using Glean. They've just like totally built their own or I mean the the customer does, but the users in the company don't because they've built their own UI or they connect it to another system. But it's just helping them govern it. Hello. Thank you for the interesting conversation. So ⁓ Tony, do you see increasing number of CIOs and CTOs who believe they can build, rebuild Glean in-house? And how do you win those deals over? Yeah, it ha it happens all the time. ⁓ and ⁓ I can think of some some some where it's happening now. It happens in different ways. ⁓ And I and I would say, I don't know if you're asking because you're like, hey, what's a good way to approach enterprises? So I would I would answer from that point of view. one one way is to let it let it go. So we you know, time and time again, like we know that we've got ⁓ you know hundreds of engineers that have been working on this for seven years and like have been go you know going through all these problems, and then I see these things where they're ⁓ yeah, we're gonna have three people and we're gonna, you know, build it in two months and stuff like that. And it's like Yeah, I mean we built the first version in twenty nineteen with three people in two months, you know, but like it you know, that that that that's nowhere near where it is today. So there's a little bit of like ⁓ there's been many, many times where we
see that and we just keep engaging and you know, we offer to help and we offer to show where it is and and go back. But then the other thing, you know, if you're if you're in a thing where you sell to enterprises and stuff too, is like multi-channeling is really important. So I know ⁓ multi-threading, I should say, where Where like sometimes you're working with a team and they're and cli they see you as their competitor. Like they see you as like, I'm not gonna have a job if if if the company buys this thing. and some so then sometimes about finding the right champion who under understands. So I can think of one one earlier customer where they had a team of like 20 people building like their internal ⁓ knowledge platform and stuff like that. And at first the team was totally hostile to us. Like they wanted to, you know, it was it was like really, really hard. ⁓ but in the end, like we found the right champion and and ⁓ they and but they actually like repurchase a team to work on their product, like instead of an internal tool, like now they're working on on the the product itself there and the and the company was coming across saying, Hey, we saved you know twelve million dollars in in salaries on this internal tool, and meanwhile we bought this other tool, you know, we bought clean for a fraction of that. And it was kind of like win-win-win. So I r I really think, you know, a lot of times I'll I'll say like I I recommend, I I recommend that like every company you you know what the value is that you're providing the world, you know what your product is, and you put your you put your expertise and your resources into that, not to like d you know, reinventing internal tools that you could that that you could pull in. ⁓ so I I guess it's different every count ⁓ every account and every every customer is different, but those are the two The two major things is like if you know what you've built and you're confident, let them try. And second, you know, make sure you're multi-threading to the right champions and you don't stop it like a detractor. Let's go to that side first. So we get some of them and then we'll come back. ⁓ thanks for all your insights. It's great. ⁓ my question is about enterprise companies and beyond Glean. ⁓ you know, a lot of companies, as you said, is experimenting today, giving them all the tools, GPT, anthropic, claude, whatever. go try figure it out and every team there's sort of a lot of shadow AI, shadow IT, and they're burning so many tokens and now IT has to go after them and say, What did you build? How did you build it? I'm doing a ⁓ I'm literally doing MCP registry, agent registry, solving AI governance, things that we shouldn't be doing in the first place. Just wanted to get your thoughts on how this would you see this in the future. I mean Glean does a great job, at least you have an admin console for tech, but I don't think the others do. So just your thoughts on that. Yeah, no, th I I I see this all the time. I to literally talked to a a like a C CIO yesterday who was saying exactly that. He's like, I have ⁓ whole whole teams running this on like ⁓ like a a credit card that was not designed for I IT spend. Like he's like, I don't even know what's going on ac across all this. and you know, if I'm if I'm looking at an industry trend, ⁓ So so one one just validating this is this is completely this is completely the case everywhere. Like the the AI sprawl and the shadow AI thing, like it's it's it it's up like it's it's there. There's not a question of if it's there or not. The question is like what is the next step and what is a way out of it? And I think there will be a way out of it. And what what what's happening right now is unprecedented level of security breaches. You know, and a lot of this is from it's from LLMs. They're getting They're they're like so good at finding vulnerabilities and everything. So we're seeing supply chain attacks everywhere. Like we're seeing like ever everything's getting popped all over the place. And I think that there's gonna be some some ⁓ more I like I would predict over the coming months or so, we'll see some really high profile like data breaches and things like that. And I think like I I usually see in large enterprises that there's like the gas and the brakes. The g you know, there's some gas in the g in the business unit that's like just use AI for everything and don't worry about it and like buy everything and the brakes is like the the the the CIO, the CISO and everything saying like, Hey, hey, but we have to do this safely, like there's still some rules and thing you know, things like that. And right now the gas is winning over the brakes and I think we'll see enough. I I th I think we're already starting to see a little bit with all the supply chain attacks and stuff, but I think there'll be some breaches that'll like do that. And yeah, somebody told me recently ⁓ you know, in exact ⁓ something effective like ⁓ well I mean yeah, you see you see it where the you you you see where like the organizations are not always aligned at really big companies and you know, I remember one of them he's just like, Yeah, you you win security, you win the enterprise, you know, and and like Yeah, lots of things will get through that business unit thing, but then the one that's gonna stick and that's gonna stay is the one that gets like really gets the good buy-in from security. Awesome. Thank you. One quick one minute follow-up question to that. If you are an enterprise search company or a tool in selling, should you focus on business units or focus should focus on security or you're you're asking like walk back to twenty nineteen, Glean, and what should you focus on? Security versus business units because You know, like head of CRO, CRO is willing to pay the bill, but security is like, I'm gonna block this because I don't think it's safe. So how do you equally partner with both parties and make sure your tool is right? You you can't focus on everybody if you're a small startup, right? I I would say this. I would say look at your look at your ICP. Is your ICP large enterprises? Is it is it Fortune 500? Is it at that level? Is it I don't know, where would I draw the line? five thousand plus person companies, that sort of thing. If that's it, you better focus on security. ⁓ if your ICP's SMBs, ⁓ ICP ideal customer profile, i is is it small businesses, is it 30 person startups and stuff like that, just just focus on the on the product and the b the business unit. And some somewhere in between there's a there's a trade off. ⁓ and so yeah focus I think I think focus has always really been important for us. Like we we have this tendency to like go horizontal and try to do so many things. But yeah, it it it can be it can be important to pick like what you're gonna do where and decide like are we going, are we more of like sales led growth, more of product led growth? Are we more of like large enterprise or more of SMB? That ⁓ that kind of thing. So that'd be that'd be how I'd look at it. Thank you. You were flattered by open AI, ⁓ thinking of you earlier, but who do you see as competitors or anybody doing anything interest interesting in the space that you're in. I mean I love I love what we're building so I and I and I want I want to make every part of it open and and and swappable. And I guess I guess what would be what what is it what is a a a competitor would be like if there's if there's not a you know not a perceived need ⁓ for for what we're offering and th those sorts of things. I usually You know, when I talk to our our account teams and things like that, it's like once once I try it, like once I get to pilot phase and everything, I'm always totally I'm totally confident. Like the only thing that really scares me is when they don't when they don't try it, when we can't we can't get through that like initial hurdle on security of connectivity or some some you know, like actually getting the systems connected and stuff. That's that's the thing that scares me. So maybe, maybe you know Yeah, to the to the extent that we're not able to tell the story or position why it's important. ⁓ I know it's it's like a it's like a partial dodge on the question, but ⁓ it'll be aware I'll leave it. All right. So that's all the time we have for questions. If you have a question, you know, you can always come up to Tony. ⁓ but yeah, let's give it up for Tony. This was fun. These were these were really fun questions. You yeah. Thank you.