Kshitij Grover: Hello everyone. Welcome to another episode of Tractable. I'm your host, Kshitij, CTO at Orb. And today I have with me David, who is the CTO at Sentry. Sentry is an error monitoring and performance monitoring tool that probably every developer on the internet has heard of with customers like Sonos and GitHub and Reddit and some 90,000 more.
Kshitij Grover: So really excited to dive in with David. David, welcome to the podcast.
David Cramer: Cool, yeah. Thanks for having me.
Kshitij Grover: Before we dive into the technical details of Sentry, I'd love to just hear a little bit about the origin story of Sentry in your words. Obviously, you've been working on Sentry for a while.
Kshitij Grover: How did it come to be? and, give us the kind of playback of the last decade or so.
David Cramer: So, I mean, Sentry is super old at this point. We started it in, or rather I started it, I think 2008. Open source project dashboard for errors is basically how you can think about it. And from there, I want to say it took on a little bit of a life of its own.
David Cramer: I think useful things get adoption and we saw some of that where people wanted what it meant built and I'm the kind of persona that if there's adoption, I actually want to work on it. It's like a motivating factor for me. And so hacked on it on the side for a few years and then eventually started the business.
David Cramer: We bootstrapped that. Raised some money, hired a bunch of people, and have been, continuing to kind of grow it out from there. But fundamentally it's very similar to what it was when we started the project. It's still a dashboard of your errors. We have some other products now, of course, but, it's a really simplistic idea, I guess.
Kshitij Grover: Yeah, that's, it's interesting that it's very easy to describe the product, especially to a developer who has to deal with errors. Do you think there's anything about the positioning of the product or the meat of the product that has really changed in that time or anything where you feel like "Oh, I'm surprised that we started going into this territory," whereas before I was just thinking it would, you know, maybe be a simpler product or just be dashboard of errors?
David Cramer: Yeah, I think when we raised very different industry when we raised our funding, which was 2015. And I didn't have a lot of conviction that errors like error monitoring as a product category was enough to build like a hundred million dollar sort of recurring revenue business, which is how you need to think about venture.
David Cramer: Right. and so we're like immediately like, well, we have to build more with, it's going to be this and then something else, which is probably performance monitoring. And we basically just took the idea of like, we messed things up a lot in production. Let's just build more tools for us to help us resolve those issues.
David Cramer: Kind of in the same guise as Sentry did because the bread and butter of the core product experience was just, there's an error, it's in your inbox, all the details there. It kind of took away this like frustration of you didn't know what's going on, you didn't know how to debug it, and we're like, well, there's a lot more problems that we could probably apply that same formula to. And I think the surprising thing over the years was one, the core product and what most people pay for is actually still.. it's obviously different, but it's mostly the same thing at like a surface level.
David Cramer: We expanded to like new audiences. So like new technology frameworks and things like that. But I think something like 80 percent of our revenue comes from error monitoring, which I would not have believed, if you told me, you know, in 2015, that was possible. So I guess that's cool. But yeah, nothing overly surprising.
David Cramer: I think we're still figuring out product two, three, four, five, and really getting us to something that we're proud of. But otherwise the core has done really well.
Kshitij Grover: So you, you've obviously had a large part in building the core of Sentry. One thing I was curious to dig into is, you know, it's easy to describe, but I imagine there are a lot of very tricky technical challenges, especially as you go from, you know, adopting it from a few teams to tens of thousands of teams.
Kshitij Grover: So, so what do you think are the kind of trickiest technical challenges you've had to architect around or, kind of push on over the years?
David Cramer: I think, you know, if you look at the bread and butter of Sentry, the goal is instrument every application, whenever there's an error, sort of process that, make it human readable, do whatever needs to happen to make it so you can consume that really quickly, and then decide if you need to notify somebody or not, right?
David Cramer: And, the interesting aspect.. I mean there's two fundamental things that are hard about that problem. One, how do you do this in every technology? But that's like, it's not a hard problem. It's just a tedious problem for the most part. And so it's just like constant, you know, write more code, write new language, deal with new framework and over So it's not a fun problem, but it's an easy problem, right? The other problem, which I wouldn't say is a hard problem, but maybe you wouldn't expect it early on, is one of the biggest challenges of Sentry is dealing with massive spikes of inbound errors and things like that. And this actually gets worse as we add more products.
David Cramer: And so for a long period of time, Sentry's fundamental goal was how do you just handle like denial of service nonstop? Like somebody's gonna have a big outage. They're going to send us all the errors. We can throw them away. That's great. Like the data loss is not a problem, but we need to stay online. And ideally it only impacts that customer that say we're delaying their data or something like that.
David Cramer: And so I, a good analogy, actually, if, I assume Gmail still does this, but back in the day, actually with Sentry, we flooded our inboxes because we took down the site and we were sending ourselves. This might not have been Sentry. This might've been something else, but we were flooding ourselves with error emails, basically it was like an email per error or something. And what Gmail does is it basically delays all inbound at that point. And it basically makes your email unusable. It's not fun. But no other customer gets affected, like it doesn't delay your inbox because I flooded my inbox. Right. Right. So Sentry has that same problem and that spent, or that took us a lot of time for, how do we really optimize things?
David Cramer: It really handles a lot of traffic pretty smoothly. How do we optimize things so when, especially when there's a repetitive concern, that's pretty tight to manage. And like I said, it's not as hard as it might sound. There's actually a lot of easy technologies to approach these problems with. But what it meant over time was like, we just built up many layers of a firewall.
David Cramer: So there's like an IP level, like, like pure IP tables, firewall level. That's like the cheapest version to remove data from the system. But also at least, flexible and accurate and things like that. And then we just have a few more of those. And so it's just a lot of dealing with things like that, which let me tell you is not fun when it's your project, because you're just on call 24/7 and you're also getting paged all the time, this kind of thing.
Kshitij Grover: Well, one question I have about that is. how do you think about the different use cases of error monitoring? So, you know, you could imagine some people are totally okay with losing a bunch of errors because they know there's a big outage and, you know, they don't need to see all those errors and in Sentry, whereas some people it's possible want like a pretty precise count of how many occurrences of this error were there, even in the presence of a big spike. So, you know, is that like an architecture level trade off, versus, you know, a customer by customer sort of give people as much configurability as they want so that they can, you know, in, in some sense, get the experience they want out of the product.
David Cramer: Turns out configurability is like one of the worst things that can exist. And so early days of Sentry, what we'd do is like, you have errors come in, we would fingerprint them, basically like hash the stack trace. And then, if we had seen it a lot, we would just throw them in the trash can. Right. So analogy here is like, maybe the first 100 errors, we captured all of them for the first 100, like instances of an error.
David Cramer: Right. And then the next 100, maybe we only captured 10 of them or something like that. And it, amounted to something like if there were a million copies of the same error, we might only store a thousand, which is awesome, right? When you think about, flexibility, there's some challenges with that because what you don't care about is the duplicate information in error, but you do care about all the unique information.
David Cramer: So you care about things like which browser was it happening on? If there's like a large variance of browsers, you actually need that metric effectively. And so we had to build abstractions to kind of make that cheap. And then over the years, we found there were more and more versions of that.
David Cramer: Well, I want to answer this kind of specific question about this error that would require me to have a lot more data than what's been sort of denormalized already. And so over time, we went from this sort of sampling algorithm to, we're just going to store everything. We're kind of charging for that anyways, which is where configurability can't work.
David Cramer: And it was kind of fine. And, what I mean by that is like, if you imagine we're trying to like operate on margins or something, because Sentry doesn't do value based pricing. We effectively do cost based pricing. And so it costs us a dollar for something. We charge you, 9 dollars. I don't know. I'm not sure the math on hand.
David Cramer: That's the goal though, is something like significant from a margin point of view, because if people don't know how this works, you got to pay for engineers. And what you're paying for is the continued development, not just the maintainability of software. Right. Anyway, so if I'm telling you it's a dollar per error and you're like, well, I want you to store less of those, but I still want you to process all of them.
David Cramer: I'm like, well, how much does that cost us? But all of a sudden you're in this complicated scenario where it's like, I don't want to deal with these problems, you know? And so we just decided to make it all the same. And here we are storing lots and lots of data.
Kshitij Grover: And it's interesting. You say you don't do value based pricing. Instead, you kind of do cost based pricing. I imagine it is very very hard to build up the same level of product experience that centering has built, even if you could store all the errors, right? I mean, you all have an alerting product, even just the ability to group and filter by dimension, like, all of those product capabilities, you know, have taken years of R and D. So is that a conversation you have with folks who maybe you're on the phone with trying to get to adopt Sentry? Or is it just like pretty obvious because so much of the product has already been used by developers that people know, like, yeah, there's no way I want to build this. And that category has just formed, especially over the last, I'm guessing, five, seven years.
David Cramer: Yeah, I don't think we have as much of a problem these days about like, should I build my own? There's a question on do I need something like Sentry versus just logs? And that question's persisted forever.
David Cramer: It's become less and less common because it's like, yeah, if you have a problem in production, it's like, does it matter to you? What matters to us because we ship changes all the time. So we're breaking things all the time. So we need to be able to fix it quickly. Right. And that's, the status quo of tech, but not everybody operates that way.
David Cramer: So they're like, ah, if there's a problem, we'll just figure it out. Or we'll just go dig in the logs and it's fine if it's harder or something. And that's fine for some people. Our sort of pitch has always been, and I actually still believe this to be the truth. Like I have a side project. I just run Sentry on it.
David Cramer: I actually, I don't have Datadog or metrics or any of these random things. And honestly, and I run on GCP often and I'm like, if I have to go dig in GCP's logs. And I'm not saying GCP is good at this, to be fair, they're not. But if I had to go dig in logs or something, I'm like, what am I doing? Like, all of my problems are errors or something, they should show up in Sentry.
David Cramer: I'm not at a scale where I should have other, like, contrived problems that I need something else. And that's a very big value prop right at the end of the day and, sort of Sentry's whole belief over a long period of time was we will be able to solve more and more needs of businesses. And so over time we'll be able to actually grow sort of more alongside a business versus trying to play catch up to what their needs might be. And that, that works out pretty okay, I would say.
Kshitij Grover: And that philosophy, I think, is actually very, different. You know, if you look at something like Datadog, they have a huge product suite. So, you know, obviously Sentry has multiple products, but Datadog has 30 products or 40 products, right? Do you think that is It's just a reflection of them being dragged into different markets and, you know, some business fundamental thing? Or do you think that is a different stance on, where they think instrumentation should go and like some technical philosophy stance?
David Cramer: I mean, I obviously can't really speak for them, but I think there's some mechanics that are kind of standard in businesses, right? There are different modes of operation.
David Cramer: First off, like some people choose, they want to build a thing. Right. And then they'll figure out how to get an adoption. Other people are like, tell me what you want customer. I'm just going to build that and sell it to you. That latter thing I think is what almost everybody does. And it's probably significantly easier as a business.
David Cramer: We don't do that. We say, what do we think is interesting to build? We're going to build that. We're going to have a thesis behind like, yeah, there are customers that want it and it's valuable. And usually that starts with, it's valuable to us. But that latter point, I think also becomes as you get bigger and bigger.
David Cramer: It's hard to just do like pure R and D, right? Like you are, dealing with demands of the market at the end of the day. And Datadog is very, big. And you look a lot at what they build and it's not very exciting for the most part just like AWS or GCP or any of these big enterprises, they build a lot of really boring check boxes that are not innovative.
David Cramer: They're not doing anything different than they ever did before. They're just like helping pick the most unexciting corporate logo you can think of. They're helping those companies consolidate on a mediocre product experience. Right? And that's fine. It is what it is. We try to avoid anything that looks like a checkbox.
David Cramer: And I, think the other thing is, at least in our market, there's this, there's this like confusing situation that's always existed where monitoring is infrastructure. It's like the I.T. Team that's running the production infrastructures in charge of monitoring and everything. And if you look at every tool that's kind of existed outside of Sentry or sort of some of these newer ideas, they're always just like, here's my logs, here's my metrics, here's my dashboards.
David Cramer: It's always the same damn thing over and they're never actually connected to like what we do, which is just change the code over and over. And so we kind of took the stance of like, that is systems monitoring and that's an important industry. But that industry has plateaued because at the end of the day, there are more and more people writing the business logic. Less and less net new people writing sort of the platform layer. Like I don't monitor disk failures, right? GCP monitors disk failures, I don't want to have those problems, right? And so our sort of belief is that wave continues and we continue to sort of abstract these low level problems. But if that's true, then there needs to be technology that helps you actually monitor more of the business logic, the application code and focus on those problems, third party vendors that you're using those problems rather than underlying hardware problems or something like that. And so, so that's kind of been our approach and that's why our error monitoring is so coupled to like the stack trace has to be really enriched.
David Cramer: It has to be service oriented. We don't care about CPU and memory. They're just not good tools in this kind of problem space. And that's almost how we think about the differentiation of things.
Kshitij Grover: And so would it be accurate to say that Sentry is much more closely coupled to the product engineer, whereas, you know, an infrastructure monitoring tool more generically would be for, as you were saying, the platform team? Or maybe even there's a stronger thesis there of that's going to just be less and less required as AWS or GCP build higher order abstractions where, you know, you don't have to have like infrastructure instance monitoring if you're using lambda, right? because that's kind of just been abstracted away from you.
David Cramer: And that's the fundamental belief. Whether you agree or not with serverless, the concept of outsourcing the platform has always like Heroku started the big wave here, in my opinion. Maybe not Heroku started it, but they were the big name at the time. We still want the same thing. Like, I don't want to run any of this stuff.
David Cramer: I spent the last two days fighting with GCP for my side project. Just like make it not cost a couple hundred dollars a month. You know? All I want to do is deploy some application images, have a database that exists, have it all securely managed for me, and that's it. Right. And that should get cheaper and more efficient over time.
David Cramer: I don't think cloud vendors are doing that, of course, but you see all the past investment these days, like Vercel, Render, Fly, like Railway. There, there's so many of them. and so the demand is still there and you, we bet on that demand. We bet that exactly what we want is the truth. Nobody wants to run that shit.
Kshitij Grover: Yeah, why do you think it is? And maybe this is a tangent from Sentry specifically, but why do you think it is that the experience of deploying code doesn't feel like it's gotten simpler or maybe you disagree, there is clearly demand for things to be simple. Obviously, Sentry has a very simple instrumentation experience in large part across lots of different environments, but you're still having to deal with deploying containers and you're still having to deal with observability issues. And yeah, it doesn't feel like we've made that much headway.
David Cramer: I think.. I don't know. there's like two sides of this, right? Big tech has gotten more complicated. And so Sentry needs something like GCP to fully outsource the systems layer, but we're building our own platform there on top.
David Cramer: That's fine. And to do that and to deal with customers like Sentry, you do actually need at least some degree of complexity or available complexity, I guess. Yeah, I think that, well, the reality is that's where the money is, right? It's servicing customers like Sentry, not servicing my side project.
David Cramer: Yeah, at least for cloud vendors. And so I just don't think they're incentivized and or they're just not good at simplicity. Because they've tried, of course, like you've seen, at least I'm not familiar as much with AWS, but like you had App Engine and all this stuff from Google that tried. And so I think that's like one challenge of it.
David Cramer: I think there is, a truth that technology has gotten more complex for two reasons. I think one, we're trying to do things that we didn't do before. And so I think that's richer, sort of more native feeling, responsive applications, and that there's inherent complexity to do that versus I've got a web server and it just returns HTML.
David Cramer: And so, okay, a little bit of complexity there. I think there's also just a lot of complexity in the scale of the internet and all this stuff has got, it's different, you know. That actually doesn't impact most people, which is odd because the complexity is still there no matter what. If I just had to put my finger on it, in my opinion, it's like everything's more complex because some things are more complex, which is not an outcome to be fair, but it is, yeah, it is difficult these days.
Marker
---
Kshitij Grover: Do you think it's possible that part of it is, and I think you were getting at this, you know, Sentry has, say 90,000 whereas most of these infrastructure providers or infrastructure tools, I'm guessing a lot of them start with some very big customers who have very complex requirements because that's where the money is to begin.
Kshitij Grover: And so you end up with a roadmap that's largely driven by that complexity rather than a very opinionated kind of product forward, almost consumer looking product because I imagine Sentry is not really beholden to one big customer that's driving a checkbox set of features, right?
David Cramer: Yeah, and that's very much true And I hate when we have a few like significant customers. I do, I'm not happy about that generally speaking because it's fine that we have them, but I will not allow anybody to like cater to them in the sense of what we do has to operate on the many. And that's a very difficult problem.
David Cramer: But I think one, it's a lot easier to build something when you ask somebody what they want and you just give them that result. And that's what most successful companies do. hard to build something when you're kind of going after like a long tail market share kind of play where ACVs have to be lower, like more consumer apps.
David Cramer: Right. And I think it's harder because you often don't have the patience or you don't have the financial capability to be patient, you know. And I always tell people, cause like one, I think the greatest strengths we had when we started the everything and still today is like, we're just like, we're going to do something that we believe in and we're going to do it whatever way we think we is the most interesting or relevant to us.
David Cramer: We're not going to do whatever way it seems easiest or status quo or anything like this. And so it was always like, we want to do market share. We want to have like, as part of that, like low price point, we want to have it like self hosted for free and open source, with a big asterisk on it and things like that.
David Cramer: And those are not the straightforward decisions of building a successful business. Those are like, why would you choose all these like weird branches to make things more difficult along the way? And I don't think people often, one, are willing to take those sort of, call it risks, if you will. I don't see it that way.
David Cramer: But, and two, I just don't think people have a strong enough opinion about what the problem is they're solving in many cases.
Kshitij Grover: Why, why do you think, or maybe what gave you the patience 10, 15 years ago to say, we're not going to, you know, go and try to get a few million dollar deals and build a roadmap on them.
Kshitij Grover: Like, is it that you just think you had a very clear idea of what error monitoring or just business logic monitoring, let's call it should look like? Or is it just like, you know, you weren't interested in it for the monetization business aspect. And, you know, generally, genuinely is a side project that you wanted to productionize and gift to other developers?
David Cramer: I think it's interesting because by the time we were venture funded, we had a couple thousand paid accounts, making money. Not a lot. Our ACV was, must've been like 300 or something, terrible. Right. And there's a real challenge. You cannot do enterprise sales. And do bottoms up at the same time.
David Cramer: I just, I don't think it's possible. And you often see enterprise companies try to like now switch and add bottoms up and they all fail at it. Like nobody's ever pulled it off as far as I understand. And it's because you've got all these million dollar plus deals over here. How are you going to justify optimizing for these, you know, 30 a month, a hundred dollar a month things, right? And this is the same problem you see with cloud providers often. Right. And that's a, you've got to be, you've got to have like top down strategic belief in the model. Right. It's not a, you can't just react financially, otherwise it's not going to work. And so, we were always like, there was no way for us to sell a seven figure deal once upon a time.
David Cramer: It just didn't exist. Right. It's like, we can't charge enough money for errors. And so we knew we needed more products to raise ACV over a long period of time. And these days it's still hard for us to charge seven figures just for error monitoring. It's doable, but it's still very, rare and difficult.
David Cramer: And that's fine. You know, we just accept that's not the business we're in for the most part. And then we just focus on what the business is, you know? And I think it's fine, but it's a very different thing. And I think it's just, you got to decide one's probably easier than the other, I think seven figures is still hard to be fair, but like 10, 20, 30, 40K deals early on, like, here's a good analogy.
David Cramer: We had.. Delta airlines was, an early self serve customer. I don't know exactly what or how I just know they had an account. They're spending $29 a month. They would not expand ever. They didn't have the usage to expand. It was probably some internal R and D team doing who knows, right. That's okay. It's not fun when you're in sales and you see that because you're like, oh, that's an enterprise logo. I should be able to commercialize that. And the fact is you just can't because adoption creates commercialization and unless they're willing to, and have a need to adopt it somewhere else where there is just that usage of it, that is like consumer scale, we're just not going to monetize it well. And that's just, that's a hard thing to come to grips with because you can't manipulate growth then from a traditional field sales model. It's gotta be much more true product-led.
Kshitij Grover: And I mean, the other thing that is kind of complicating things, obviously is Sentry has this open source model which, which I think has been a mixed bag for the industry at large. But how has that played into this commercialization conversation and how do you think about, you know, big companies who maybe have a pretty big use case, just going and saying, we're going to use the open source software?
David Cramer: Yeah, I think the way I always talk about this is Sentry is open source or was open source. We're pretty close to it, but, just quick aside, Sentry has two models. A lot of our stuff's open source, true open source, free. And our core repo is open source after two years, but it's a license that allows you to non commercially utilize it effectively.
David Cramer: So you can just run it yourself, no strings attached, same code we run in prod. And so we used to be full open source. And again, this is only because we want to do business this way. We think it's a better path forward from, from an industry as a whole. And people would often ask like, well, how do you monetize open source?
David Cramer: Blah, blah, blah. I'm going to ignore all that because the answer is it's not related. Open source is effectively a distribution model. And so you got to think of that open source is marketing. And the way we always got value of open source is we would help these customers. They would contribute back once in a while, but not a lot. At the end of the day, it was mostly us building the software, right?
David Cramer: But the best story I have from early on is, Uber was a really big user of Sentry. They had a, the biggest installation bigger than our SaaS service. They had an installation inside of Uber. They're self hosting all this stuff. And when, when we venture funded, we're like, oh, this is gonna be easy. We're just going to convert Uber and Airbnb and all these people that were just like running like free open source. Right. Yeah, spoiler. We didn't convert any of them. But Uber is a good one because we actually did charge Uber at some point. A significant amount of money for the scale we were at. It was a six figure deal, but it was for like customer support and what might as well be, you know, consulting or professional services. Right. And very quickly, like this sucks. We don't want to do this. It's a distraction. And we did not renew the contract. And we were like, we just said, well, I don't think we can convert Uber.
David Cramer: And a couple of years later we realized it didn't matter because what we saw was all these companies like Uber, eventually they came to the SaaS service. But they didn't come from Uber in the sense of Uber never signed up, but all those employees, or a lot of employees, would move on to other, you know, roles at different companies and they'd be like, Oh yeah, I used Sentry at Uber, it was great.
David Cramer: Also, there's a SaaS service. It costs us like 30 bucks to set up. And that was phenomenal for us. And so that's where I go back to the distribution model. It's marketing at the end of the day. There's obviously other benefits of open source, but I think from a business point of view, that's such a big deal.
David Cramer: And I think it's very hard to believe that anybody is ever going to win again in the sort of IT infrastructure space against an open-source player that builds triple A product. And I wholeheartedly have that belief at this point. It may not be universally true yet. And again, it still matters.
David Cramer: It's gotta be like tomatoes to tomatoes kind of thing comparison. It can't be like, here's a mediocre, bad product that's open source, like a phenomenal product that's proprietary. But same, playing level, like open source is going to win because it just provides all these great benefits and beliefs.
David Cramer: And it's almost like, it's, a culture you're buying into, right? And I think people underappreciate that idea of open source. And that comes closely, like the last, like sort of, fork in that I want to go into is open source is, effectively for us, the way we think about it is brand marketing. It's not just marketing or distribution.
David Cramer: And what I mean by that is, I loved it, especially internally at Sentry these days. I talk a lot about brand marketing, brand awareness, and one of the examples I give is Liquid Death. Which if you're not familiar, it's a cool looking can of water. It's still water in a can. It's not a new idea, but they're not selling water. What they're selling is not beer. It's just a beer alternative, that just water, that looks cool. It looks like you're drinking beer. You go to clubs, they sell you Liquid Death. Now I was at a music concert. All they sold was Liquid Death, but they're selling not water. Right. And that's it. And it's like Sentry is not selling error monitoring. We're selling a philosophy and how we think about open source sustainability. Or we're selling a philosophy and how we think about privacy or product quality, you know, all these other things. And I think that is... that is not normal in tech for sure. But you see it a lot in consumer markets and things like that. And so that's the other part I think of open source. It really matters for us as it is a brand that we are really focused on.
Kshitij Grover: And I think to this point of it being a distribution strategy or brand marketing, it does feel like it's not the sort of thing that you can, you know, act on later in the company lifecycle. It really does feel like it has to be in the DNA of the company. Do you think that's true? Like, do you think it's, it presumably would have been very hard for you to espouse this philosophy years after Sentry started and maybe a related question. Like, do you think this is why there aren't more open source successful, large scale products? Just the founder DNA or the product DNA isn't there from the beginning?
David Cramer: So I think almost everything at a company has to be top down. In the sense of it has to be believed in at the top and enforced from the top. It might not come from the top, but for example, if it's like a bunch of engineers like open source and everybody in the executive team is like, whatever, we don't really understand or care about it.
David Cramer: It's not going to survive as a you know, as a core thing. So that, that matters. I think the reason it's less common is because it's very hard and it does have to be in your DNA. And it's not, it's, there's a bunch of these like attributes, right? Like Sentry is not successful because we're open source.
David Cramer: Sentry might be more successful because we were open source, but it's not successful because of that. And I think this is where the confusion comes in a lot. And so I think there's that aspect. But then there's a lot of this like fear and challenging, of like, well, what's the value of open source?
David Cramer: And we are open source just because like my entire career, I've just shipped open source software. I just didn't care, right? I'm like, yeah, it seems fine, you know, no, no problems here. But I do talk to a lot of people that like kind of go into open source and they don't, they maybe don't have a strong belief around why or what it is they think it's probably useful to do. Which is fine, you know, that's not a bad thing, but I think especially trying to go that direction after the fact, you can, don't get me wrong, you totally can.
David Cramer: Because Sentry, for example, where we have a bunch of this privacy stuff going on that we're going to talk about. We were not that early days. We're like, no, just give us all your data and trust us. We'll protect it. You know, it's useful if we have this data. Now we're like, you know what? We're actually going to give you a ton of controls and be like aggro pro privacy because we think like this is important now and it's changing.
David Cramer: So I think you can change first off. I think you've got to be thoughtful about what it is and what your risks are and when it does or doesn't matter. Right. And so Sentry was open source until one company decided they wanted to resell Sentry. One company that had never contributed to Sentry at all. And I'm like, this is annoying.
David Cramer: Like, why would we tolerate this kind of thing? And we were the IP holders, so it didn't matter. Now we could have been open source and that could have been fine. I don't know, but we chose to switch to this other licensing strategy where it's like, well, now we're protected. We can still monetize Sentry.
David Cramer: But I, think there's this interesting aspect where there's not been a company that's gotten so big and stayed true, free, open source and monetized extraordinarily well. Like there's no billion dollar revenue companies that are like exclusively open source, free open source software. And I think the free part is important because... what I mean by free is not free software foundation. I mean, free as in almost no strings attached in the sense of like not GPL. I mean, free as in permissive free, because GPL effectively has constraints that are attached to it. And one way some of these companies monetize is by scaring other companies from using their open source by putting these licenses that are really like high risk for certain organizations, right?
David Cramer: And so I want to believe somebody will do it at some point. They'll take that leap and it will work. We were not going to be the ones that quite do it because it was just frustrating for us. But I think it's doable. I don't think it's a limitation anymore.
Kshitij Grover: And I guess maybe last point on this open source topic, what do you think folks have to take away from, you know, things like Kafka or tools where cloud providers go and effectively take the software and resell it, in a way that, you know, presumably takes a decent chunk out of the, business of the company that created the software?
David Cramer: So one thing I was talking to a couple of friends the other day who are, considering open sourcing their core service and they're like, well, what kind of license do we use? Do we use the new same thing Sentry released recently?
David Cramer: Do we use a business source license, open core, blah, there's all these decisions. Right. And I told them, I'm like, the decision has to be inherently connected to how you're going to monetize because open source is part of the packaging models, part of the distribution model. It's the same way, like which features you get for 20 bucks are part of your packaging model. That's how you monetize everything. And so if it doesn't make sense, it's not attached to that, it's not successful. And so, you know, think about Kafka. If the only way to monetize Kafka is through cloud hosting and the whole goal of Kafka is to make it really easy to run well, I mean, newsflash, anybody that can sell it is going to win, you know, like whoever sells it best wins because there's nothing there. Like the packaging is just, let's make it dumb, easy to run. But if the whole thing with Kafka is say, like, you need all these tools on top or something else, right? Okay. Maybe it's fine all of a sudden. And it doesn't mean you'll have exclusive rights or anything. And that might be okay.
David Cramer: And I think there's this interesting thing that we've not seen enough data and you've got to take some risks to see of like, well... take, Sentry, for example, just cause it's, you can't really compete with us. If it was truly free open source, super easy to run and commercialize, so there could be Sentry alternatives in SaaS.
David Cramer: Would that harm our business or would it broaden the market for our business? It's really hard to say, you know, cause like you have MongoDB and Elastic who have these same kind of things with like AWS and whatnot, and they're still very successful companies. And so it's really hard to say that it hurt them at all. And to be fair, they've also always had proprietary components that they sort of shipped alongside.
David Cramer: Maybe in our case, those things were just not good enough. Like we had one product, which was the infrastructure component that had great product market fit. And then they had a bunch of products that never achieved product market fit. And that's why they couldn't monetize as well as they wanted. There's something along those lines, you know.
Kshitij Grover: Yeah, and it also feels like this weird tension of, you open source your core technology, and then you kind of actively work against that by trying to create closed source adjacencies that only you offer that someone else doesn't offer. It feels like.. It feels that tension with the original spirit of having open source, the thing in the first place.
Kshitij Grover: Right.
David Cramer: One, one point I want to make on that is this is one of the most challenging pieces and it's why Sentry's so...we have so high conviction around our model of like protect ourselves to sell the SaaS service to commercialize everything because we do not want to cripple the software to distribute it.
David Cramer: Right. And we're not a great case study. We're not infrastructure. We're just a product that we sell that's complicated to run. But it is like that business model. I think a lot of people, a lot of people desire to just be like, I just want to commercialize it via cloud services. But so do some of the biggest companies in the world.
David Cramer: And so that's sort of like where we're at with the tug of war, you know?
Kshitij Grover: I want to get back a little bit to the technical side of things. So Sentry obviously has to be reliable, has to ingest the data. And it also has to be really easy to set up with as little configurability as possible across a ton of different environments.
Kshitij Grover: How do you manage that product surface area? Like I'm assuming, you know, internally you can't possibly use every Sentry SDK. You can't test every framework. But on the other hand, you know, with that many companies and developers using Sentry, you're going to get bug reports across every framework and things that can go wrong will go wrong.
Kshitij Grover: That seems like a pretty challenging technical problem and scale. So, or maybe it's also an organizational staffing problem. How do you think about that?
David Cramer: So I'm going to use, I stole this from somebody else the other day. It's open source with a healthy, dose of capitalism. And what I mean by that is, Sentry for like, as long as it's had money has paid contractors to help us maintain those as our SDKs. And cause it is a staffing problem. It's just a sheer breadth of problems to solve and expertise. And those people we often pay, we're just contributors. We're like, hey, this is great. You want to keep contributing and just have money also. Which I would encourage more people to do.
David Cramer: It's a phenomenal model if it works. But it is a really hard challenge and I'll tell you it gets harder and harder because once upon a time, it's like, well, Django and Rails and Laravel, and they're, kind of, they're big targets are like a lot of people use one thing. You solve it once, you tweak it a little bit, every so often maintain it and you're fine.
David Cramer: That's become more and more problematic in recent years as the, like the change in user interfaces with the, like, basically with the JavaScript movement, you know. You have some of this on mobile too, where there's a lot of change in the, how we develop mobile apps and everybody wants like a new language or a new framework all the time.
David Cramer: And that shouldn't be such a problem, but the problem is most of these frameworks. Well, one, sometimes they're, built by people who have not really run like production operations and software. And so they're not always designed with... we need telemetry in mind, you know? So that's one challenge.
David Cramer: I think the other challenge is even when they are run by these people, they're so complicated sometimes that, and again, the same problem, they've not really thought about, hey, like people actually need to be able to diagnose all these problems in production. And that list gets endless, you know, and you see that a lot with the JavaScript frameworks these days where they've gotten so complicated to debug.
David Cramer: And the frameworks themselves, because there's so many moving parts and it's not like they often don't control the whole thing, it's very hard for them to even simplify that. And so in an ideal world, new framework comes out, it's built in some core hooks, you know? And we just, we shim those hooks, right?
David Cramer: And then for us, it should primarily be, do we support that language really well? And then do we support the framework by some hooks? But what I'll tell you in practice, it is, we spend lots and lots of engineering hours supporting every framework under the sun. And it's a never ending moving target. That is very, hard to do.
David Cramer: I don't know what our total staffing is behind it. But I do think if you include, we have SDKs, right? Which again, we either contract or full time engineering staff, pretty much any SDK that's run, and it's just more than one headcount per mobile, for example, I think we have nine people on, which is insane to me.
David Cramer: But, so that, that's one. And I think the other side you have to understand is it's not just like an SDK. There's all these like language sort of runtime specific challenges. You think about like, for me, SDK is like, it's next JS or it's Django. Well, you've got AWS Lambda serverless shenanigans, right?
David Cramer: And then you've got everybody else's flavor of serverless and they all work differently and they're all their own frameworks on top of everybody else's frameworks. And it's like, I don't know. I think this problem is unsolvable in its current shape. And I don't know what we're going to do about it in all honesty.
David Cramer: Like a lot of people want to believe like open telemetry and stuff's the solution. I will, tell you it's just not. Because it's just not, it doesn't solve it well enough in so many scenarios, especially in the front end where the problem is the biggest. And open telemetry is like ignore the logs and the metrics, but the tracing component, that's good, but it's only one little component of everything we need to do.
David Cramer: And Sentry is not going to like create some open errors spec, let alone cater to open telemetry trying to do it because one, it's big company process shenanigans. Two, we are so invested in our entire, you know, thing. We're not going to spend tens of millions of dollars just to make it easy for more copycats to show up, you know?
David Cramer: And so there's all these challenges, you know, and I just don't know what the solution is going to be. But I will tell you forever, it's always just like, we have to invest in it. We have to be really good at it. And when we're not, we have to recognize it and, double down on that investment, you know?
Kshitij Grover: But, that is interesting cause it sounds like, yeah, one shape of solution to that problem, which may or may not work is try to get an RFC out there. Try to make these frameworks move in some direction rather than you having to staff every incremental framework that comes out. And it sounds like you're saying that's probably not the right solution because it, yeah, in some sense lowers the barriers to entry to market, but also maybe it's just unrealistic for that to actually happen.
David Cramer: Yeah, I think it's unrealistic because what often happens with standard bodies is they develop a standard and it's fast right away because there's no adoption and then there's adoption and it stops.
David Cramer: And even when there's adoption, it's not universal adoption. There's some people that are like, I don't like this. I'm going to do it my way. And so even though I think open telemetry for the tracing specific stuff, so open tracing back in the day. There's a lot of great ideas in there because we should not have to instrument a bunch of third party libraries over and over and over.
David Cramer: Yeah, the adoption. This is like a many years old project at this point, and it's still got such minuscule adoption in the industry that I'm like, if that's the outcome, it's like, we're definitely not wasting time on trying to create a standard, you know. And so I don't know what the solution is. But one interesting analogy to that problem is recently we released something called Spotlight Essentiary, which is actually unrelated to Sentry, but all we did was we basically took our SDKs and we leveraged the data collection to build like a debug bar locally, like a better debugger in, in development environments.
David Cramer: And we did have to change the SDKs. Fortunately, we control them, so it's easy. But we were actually able to take something we had already built and leverage it into something that's entirely different. That just needed similar kinds of data, you know. There's this interesting concept in there, even though it's not like an open standard.
David Cramer: And we're certainly not going to change our SDKs to help a competitor. We were actually able to expand the market capabilities using what is our internal standard, you know? And so, I don't know, maybe, we'll just keep funding all this until the end of time and the number of engineers on SDKs will be like 500 in five years, I don't know.
David Cramer: But I, I like to believe some, it's becoming more of a mature industry. And I think there's these curves where it kind of expands in terms of like complexity and ideas, and then it contracts, as, things start to mature. We haven't seen a contract yet, unfortunately, in like the job control, especially.
David Cramer: But I would like to believe we're getting more mature about the technology these days. And there's a lot more people with, a lot more depth of experience. Building some of those projects versus, Oh, I just live in the PHP ecosystem. And all I do is build PHP. So I don't know what happens over here. We've got a lot of breath in the industry now.
Kshitij Grover: Maybe talking about a, I guess, relatively significantly more immature problem. I've noticed that you all have also started to build some AI features into Sentry, but, in a very like Sentry like way where, in some sense, it's very transparent that these things are guesses that you're not trying to pin down the problem, but you're trying to directionally suggest what might be the issue.
Kshitij Grover: We can talk about the future, but what I'm wondering about is more philosophically, do you think that's here to stay? Do you think that right now it's kind of a toe dip into AI just to see if it lands with developers and you're really going to solidly focus headcount and technical investment into actual proven problems with maybe an opinionated stance rather than let's just do AI because AI is the thing.
David Cramer: Yeah, I think a lot of people are doing AI because AI is the thing and I will tell you. I've I personally have very little confidence. A lot of the spaces make sense for it I think there are some interesting things that the language models can work on today. And I'm not an expert by any means but I have I've been playing with them. And then there are things they are just absolutely horrendous at, and I'm not a, I'm not a believer of like, oh, shiny new technology, let's force its way into something, you know. And that feels like what's happening in a lot of ways versus like, oh, obviously this is powerful here. And I think it's fair that they don't know when they're testing it. But every company in the world is like, look at our AI chat bot thing. I'm like, I don't need an AI chat bot, you know? And so ours is we actually just spun up a dedicated ML AI investment, but it's not a dedicated LLM ChatGPT investment. We are interested to see what makes sense there, but we're thoughtful about it from a point of view where we're realistic. And we are not just going to build something that anybody else can build.
David Cramer: Because frankly, a smaller company is going to be better at building something quickly than us. So it's like, well, what could we uniquely do that others can't? And that's like, well, what can we uniquely do with the amount of interesting telemetry we have? And things like that. And so a lot of the near term investment for us is kind of like, well,it's not so much that the LLM tech is interesting. It's that we should run more experiments. We are to scale where the money is irrelevant to us, the staff, like a small team, you know. And if something ends up being very valuable, well, that the ROI is really, large. And so I think we'll see, as an example, CodeCup, which is a company we acquired a while back, who kind of does, I like to think of it as like Sentry pre-production.
David Cramer: And so can we stop bugs from happening? Even though it's mostly focused on test coverage right now, but they actually have, it's like not, documented or announced, but there's something we built internally that is not good at all yet. Maybe it will be good, but it's just a code review buddy. And I'm like, can somebody just do this and have it evaluate the code and tell me what's broken?
David Cramer: Because again, the language models are not perfect. But I have seen in my own code where I've given it a complex function and it has identified something that I could not identify myself. And if it can do that, can it make code review more efficient or something? Now that's not a uniquely us problem, of course, but I'm like, there are areas where it does seem interesting, I guess, but I think a lot of the technology: the language models are either too expensive or they're just not capable enough yet to really pull something off that's great.
Kshitij Grover: But it sounds like basically they've reduced the barrier to entry enough where a company like Sentry, it makes sense to run more experiments involving them, even if it doesn't mean releasing full-fledged features that use them just yet.
David Cramer: Yeah, I will say that the complexity for Sentry is we're not going to funnel data to OpenAI. And so we have to run models, behind our firewall and like, GCP, we just, have this intense policy around data processors. And so we do have like an OpenAI feature, but it's opt in for this one single feature, meaning we couldn't all of a sudden make like a network-wide behavior.
David Cramer: So we're figuring out some of that, but the technology I think is exciting. But just like every new piece of technology, it's a little bit overhyped, I think. And you saw this with Bitcoin and shit, even though that was more like, how do we launder money? It was like, oh, can we apply like a distributed hash blockchain to every problem in the world where most of the time it makes no sense.
Kshitij Grover: Yeah, awesome. So maybe last question I'll ask is less on the technical side, but I'm curious about this. Sentry's brand is very unique. And as you were saying, a lot of these things I imagine do come top down, but you know, you all have billboards around Sentry can't fix this with this like phone falling on you. The text on the marketing site, in the product is really silly. What's driving that? Is that just like, again, your ethos and like kind of leadership ethos? Or do you think it's something to do with, you know, people who use Sentry are presumably stressed at times, right? And is there something around the actual product surface?
David Cramer: This is us just a strong belief in brand marketing. So part of our brand is be authentic, you know, and it's okay to be tongue in cheek. There's a, balance there, of course it's not, our brand is not "talk shit." My brand is "talk shit"to be clear. But our company brand is not. It's just like, let's tell it how it is.
David Cramer: So my favorite analogy here, or my favorite example of this rather, is, in our, in, Sentry's core, we have a single page applications. It's, very large these days. And when we built it, which it must be 2015, react SPA, we put it in a loading indicator we thought was funny. It looks cool.
David Cramer: And it's just like, please wait while we load an obnoxious amount of JavaScript. Back in the day, you never saw it. Now you see it all the time and it's very true. It actually wasn't that true back in the day. It still kind of felt like, and that's phenomenal. Like people love it. It's like tongue in cheek.
David Cramer: It's just us like sort of like demeaning ourselves, you know? But we actually have brand voice policies internally that are all centered around this. And so I think it did come from the early team. But I think great ideas are not just coming from a person or a set of people. There are things that people value and then latch onto.
David Cramer: And so that one's like held pretty well for us. And so I'm partially responsible for the billboard campaign and my voice is very strong, but other people took it and ran with it and then made it better or something. Cause they took...the idea was like Sentry can't fix this. First off Sentry doesn't fix anything.
David Cramer: And secondly, the things that we depict have nothing to do with anything Sentry does. So like the first one or the main one was, it was a minotaur, like the magical creature playing Beat Saber and knocking over some pots. And it's like, what the? What even is that? You know, that's the whole point.
David Cramer: It's like, a billboard. It's like, you're looking at it and you're like, what is that? And I'm like, what the hell is Sentry? And that's kind of the thesis behind it. But a lot of it just like, yeah, that's like, there's a lot of stiff corporate marketing and there's no reason to do it again. It's just a belief we have, you know?
Kshitij Grover: Awesome. Well, here's to more billboards and in the coming year. And thank you so much for taking the time and coming on the show.
David Cramer: Yeah, absolutely. And thanks again for having me. This was fun. Awesome.
Kshitij Grover: So really excited to dive in with David. David, welcome to the podcast.
David Cramer: Cool, yeah. Thanks for having me.
Kshitij Grover: Before we dive into the technical details of Sentry, I'd love to just hear a little bit about the origin story of Sentry in your words. Obviously, you've been working on Sentry for a while.
Kshitij Grover: How did it come to be? and, give us the kind of playback of the last decade or so.
David Cramer: So, I mean, Sentry is super old at this point. We started it in, or rather I started it, I think 2008. Open source project dashboard for errors is basically how you can think about it. And from there, I want to say it took on a little bit of a life of its own.
David Cramer: I think useful things get adoption and we saw some of that where people wanted what it meant built and I'm the kind of persona that if there's adoption, I actually want to work on it. It's like a motivating factor for me. And so hacked on it on the side for a few years and then eventually started the business.
David Cramer: We bootstrapped that. Raised some money, hired a bunch of people, and have been, continuing to kind of grow it out from there. But fundamentally it's very similar to what it was when we started the project. It's still a dashboard of your errors. We have some other products now, of course, but, it's a really simplistic idea, I guess.
Kshitij Grover: Yeah, that's, it's interesting that it's very easy to describe the product, especially to a developer who has to deal with errors. Do you think there's anything about the positioning of the product or the meat of the product that has really changed in that time or anything where you feel like "Oh, I'm surprised that we started going into this territory," whereas before I was just thinking it would, you know, maybe be a simpler product or just be dashboard of errors?
David Cramer: Yeah, I think when we raised very different industry when we raised our funding, which was 2015. And I didn't have a lot of conviction that errors like error monitoring as a product category was enough to build like a hundred million dollar sort of recurring revenue business, which is how you need to think about venture.
David Cramer: Right. and so we're like immediately like, well, we have to build more with, it's going to be this and then something else, which is probably performance monitoring. And we basically just took the idea of like, we messed things up a lot in production. Let's just build more tools for us to help us resolve those issues.
David Cramer: Kind of in the same guise as Sentry did because the bread and butter of the core product experience was just, there's an error, it's in your inbox, all the details there. It kind of took away this like frustration of you didn't know what's going on, you didn't know how to debug it, and we're like, well, there's a lot more problems that we could probably apply that same formula to. And I think the surprising thing over the years was one, the core product and what most people pay for is actually still.. it's obviously different, but it's mostly the same thing at like a surface level.
David Cramer: We expanded to like new audiences. So like new technology frameworks and things like that. But I think something like 80 percent of our revenue comes from error monitoring, which I would not have believed, if you told me, you know, in 2015, that was possible. So I guess that's cool. But yeah, nothing overly surprising.
David Cramer: I think we're still figuring out product two, three, four, five, and really getting us to something that we're proud of. But otherwise the core has done really well.
Kshitij Grover: So you, you've obviously had a large part in building the core of Sentry. One thing I was curious to dig into is, you know, it's easy to describe, but I imagine there are a lot of very tricky technical challenges, especially as you go from, you know, adopting it from a few teams to tens of thousands of teams.
Kshitij Grover: So, so what do you think are the kind of trickiest technical challenges you've had to architect around or, kind of push on over the years?
David Cramer: I think, you know, if you look at the bread and butter of Sentry, the goal is instrument every application, whenever there's an error, sort of process that, make it human readable, do whatever needs to happen to make it so you can consume that really quickly, and then decide if you need to notify somebody or not, right?
David Cramer: And, the interesting aspect.. I mean there's two fundamental things that are hard about that problem. One, how do you do this in every technology? But that's like, it's not a hard problem. It's just a tedious problem for the most part. And so it's just like constant, you know, write more code, write new language, deal with new framework and over So it's not a fun problem, but it's an easy problem, right? The other problem, which I wouldn't say is a hard problem, but maybe you wouldn't expect it early on, is one of the biggest challenges of Sentry is dealing with massive spikes of inbound errors and things like that. And this actually gets worse as we add more products.
David Cramer: And so for a long period of time, Sentry's fundamental goal was how do you just handle like denial of service nonstop? Like somebody's gonna have a big outage. They're going to send us all the errors. We can throw them away. That's great. Like the data loss is not a problem, but we need to stay online. And ideally it only impacts that customer that say we're delaying their data or something like that.
David Cramer: And so I, a good analogy, actually, if, I assume Gmail still does this, but back in the day, actually with Sentry, we flooded our inboxes because we took down the site and we were sending ourselves. This might not have been Sentry. This might've been something else, but we were flooding ourselves with error emails, basically it was like an email per error or something. And what Gmail does is it basically delays all inbound at that point. And it basically makes your email unusable. It's not fun. But no other customer gets affected, like it doesn't delay your inbox because I flooded my inbox. Right. Right. So Sentry has that same problem and that spent, or that took us a lot of time for, how do we really optimize things?
David Cramer: It really handles a lot of traffic pretty smoothly. How do we optimize things so when, especially when there's a repetitive concern, that's pretty tight to manage. And like I said, it's not as hard as it might sound. There's actually a lot of easy technologies to approach these problems with. But what it meant over time was like, we just built up many layers of a firewall.
David Cramer: So there's like an IP level, like, like pure IP tables, firewall level. That's like the cheapest version to remove data from the system. But also at least, flexible and accurate and things like that. And then we just have a few more of those. And so it's just a lot of dealing with things like that, which let me tell you is not fun when it's your project, because you're just on call 24/7 and you're also getting paged all the time, this kind of thing.
Kshitij Grover: Well, one question I have about that is. how do you think about the different use cases of error monitoring? So, you know, you could imagine some people are totally okay with losing a bunch of errors because they know there's a big outage and, you know, they don't need to see all those errors and in Sentry, whereas some people it's possible want like a pretty precise count of how many occurrences of this error were there, even in the presence of a big spike. So, you know, is that like an architecture level trade off, versus, you know, a customer by customer sort of give people as much configurability as they want so that they can, you know, in, in some sense, get the experience they want out of the product.
David Cramer: Turns out configurability is like one of the worst things that can exist. And so early days of Sentry, what we'd do is like, you have errors come in, we would fingerprint them, basically like hash the stack trace. And then, if we had seen it a lot, we would just throw them in the trash can. Right. So analogy here is like, maybe the first 100 errors, we captured all of them for the first 100, like instances of an error.
David Cramer: Right. And then the next 100, maybe we only captured 10 of them or something like that. And it, amounted to something like if there were a million copies of the same error, we might only store a thousand, which is awesome, right? When you think about, flexibility, there's some challenges with that because what you don't care about is the duplicate information in error, but you do care about all the unique information.
David Cramer: So you care about things like which browser was it happening on? If there's like a large variance of browsers, you actually need that metric effectively. And so we had to build abstractions to kind of make that cheap. And then over the years, we found there were more and more versions of that.
David Cramer: Well, I want to answer this kind of specific question about this error that would require me to have a lot more data than what's been sort of denormalized already. And so over time, we went from this sort of sampling algorithm to, we're just going to store everything. We're kind of charging for that anyways, which is where configurability can't work.
David Cramer: And it was kind of fine. And, what I mean by that is like, if you imagine we're trying to like operate on margins or something, because Sentry doesn't do value based pricing. We effectively do cost based pricing. And so it costs us a dollar for something. We charge you, 9 dollars. I don't know. I'm not sure the math on hand.
David Cramer: That's the goal though, is something like significant from a margin point of view, because if people don't know how this works, you got to pay for engineers. And what you're paying for is the continued development, not just the maintainability of software. Right. Anyway, so if I'm telling you it's a dollar per error and you're like, well, I want you to store less of those, but I still want you to process all of them.
David Cramer: I'm like, well, how much does that cost us? But all of a sudden you're in this complicated scenario where it's like, I don't want to deal with these problems, you know? And so we just decided to make it all the same. And here we are storing lots and lots of data.
Kshitij Grover: And it's interesting. You say you don't do value based pricing. Instead, you kind of do cost based pricing. I imagine it is very very hard to build up the same level of product experience that centering has built, even if you could store all the errors, right? I mean, you all have an alerting product, even just the ability to group and filter by dimension, like, all of those product capabilities, you know, have taken years of R and D. So is that a conversation you have with folks who maybe you're on the phone with trying to get to adopt Sentry? Or is it just like pretty obvious because so much of the product has already been used by developers that people know, like, yeah, there's no way I want to build this. And that category has just formed, especially over the last, I'm guessing, five, seven years.
David Cramer: Yeah, I don't think we have as much of a problem these days about like, should I build my own? There's a question on do I need something like Sentry versus just logs? And that question's persisted forever.
David Cramer: It's become less and less common because it's like, yeah, if you have a problem in production, it's like, does it matter to you? What matters to us because we ship changes all the time. So we're breaking things all the time. So we need to be able to fix it quickly. Right. And that's, the status quo of tech, but not everybody operates that way.
David Cramer: So they're like, ah, if there's a problem, we'll just figure it out. Or we'll just go dig in the logs and it's fine if it's harder or something. And that's fine for some people. Our sort of pitch has always been, and I actually still believe this to be the truth. Like I have a side project. I just run Sentry on it.
David Cramer: I actually, I don't have Datadog or metrics or any of these random things. And honestly, and I run on GCP often and I'm like, if I have to go dig in GCP's logs. And I'm not saying GCP is good at this, to be fair, they're not. But if I had to go dig in logs or something, I'm like, what am I doing? Like, all of my problems are errors or something, they should show up in Sentry.
David Cramer: I'm not at a scale where I should have other, like, contrived problems that I need something else. And that's a very big value prop right at the end of the day and, sort of Sentry's whole belief over a long period of time was we will be able to solve more and more needs of businesses. And so over time we'll be able to actually grow sort of more alongside a business versus trying to play catch up to what their needs might be. And that, that works out pretty okay, I would say.
Kshitij Grover: And that philosophy, I think, is actually very, different. You know, if you look at something like Datadog, they have a huge product suite. So, you know, obviously Sentry has multiple products, but Datadog has 30 products or 40 products, right? Do you think that is It's just a reflection of them being dragged into different markets and, you know, some business fundamental thing? Or do you think that is a different stance on, where they think instrumentation should go and like some technical philosophy stance?
David Cramer: I mean, I obviously can't really speak for them, but I think there's some mechanics that are kind of standard in businesses, right? There are different modes of operation.
David Cramer: First off, like some people choose, they want to build a thing. Right. And then they'll figure out how to get an adoption. Other people are like, tell me what you want customer. I'm just going to build that and sell it to you. That latter thing I think is what almost everybody does. And it's probably significantly easier as a business.
David Cramer: We don't do that. We say, what do we think is interesting to build? We're going to build that. We're going to have a thesis behind like, yeah, there are customers that want it and it's valuable. And usually that starts with, it's valuable to us. But that latter point, I think also becomes as you get bigger and bigger.
David Cramer: It's hard to just do like pure R and D, right? Like you are, dealing with demands of the market at the end of the day. And Datadog is very, big. And you look a lot at what they build and it's not very exciting for the most part just like AWS or GCP or any of these big enterprises, they build a lot of really boring check boxes that are not innovative.
David Cramer: They're not doing anything different than they ever did before. They're just like helping pick the most unexciting corporate logo you can think of. They're helping those companies consolidate on a mediocre product experience. Right? And that's fine. It is what it is. We try to avoid anything that looks like a checkbox.
David Cramer: And I, think the other thing is, at least in our market, there's this, there's this like confusing situation that's always existed where monitoring is infrastructure. It's like the I.T. Team that's running the production infrastructures in charge of monitoring and everything. And if you look at every tool that's kind of existed outside of Sentry or sort of some of these newer ideas, they're always just like, here's my logs, here's my metrics, here's my dashboards.
David Cramer: It's always the same damn thing over and they're never actually connected to like what we do, which is just change the code over and over. And so we kind of took the stance of like, that is systems monitoring and that's an important industry. But that industry has plateaued because at the end of the day, there are more and more people writing the business logic. Less and less net new people writing sort of the platform layer. Like I don't monitor disk failures, right? GCP monitors disk failures, I don't want to have those problems, right? And so our sort of belief is that wave continues and we continue to sort of abstract these low level problems. But if that's true, then there needs to be technology that helps you actually monitor more of the business logic, the application code and focus on those problems, third party vendors that you're using those problems rather than underlying hardware problems or something like that. And so, so that's kind of been our approach and that's why our error monitoring is so coupled to like the stack trace has to be really enriched.
David Cramer: It has to be service oriented. We don't care about CPU and memory. They're just not good tools in this kind of problem space. And that's almost how we think about the differentiation of things.
Kshitij Grover: And so would it be accurate to say that Sentry is much more closely coupled to the product engineer, whereas, you know, an infrastructure monitoring tool more generically would be for, as you were saying, the platform team? Or maybe even there's a stronger thesis there of that's going to just be less and less required as AWS or GCP build higher order abstractions where, you know, you don't have to have like infrastructure instance monitoring if you're using lambda, right? because that's kind of just been abstracted away from you.
David Cramer: And that's the fundamental belief. Whether you agree or not with serverless, the concept of outsourcing the platform has always like Heroku started the big wave here, in my opinion. Maybe not Heroku started it, but they were the big name at the time. We still want the same thing. Like, I don't want to run any of this stuff.
David Cramer: I spent the last two days fighting with GCP for my side project. Just like make it not cost a couple hundred dollars a month. You know? All I want to do is deploy some application images, have a database that exists, have it all securely managed for me, and that's it. Right. And that should get cheaper and more efficient over time.
David Cramer: I don't think cloud vendors are doing that, of course, but you see all the past investment these days, like Vercel, Render, Fly, like Railway. There, there's so many of them. and so the demand is still there and you, we bet on that demand. We bet that exactly what we want is the truth. Nobody wants to run that shit.
Kshitij Grover: Yeah, why do you think it is? And maybe this is a tangent from Sentry specifically, but why do you think it is that the experience of deploying code doesn't feel like it's gotten simpler or maybe you disagree, there is clearly demand for things to be simple. Obviously, Sentry has a very simple instrumentation experience in large part across lots of different environments, but you're still having to deal with deploying containers and you're still having to deal with observability issues. And yeah, it doesn't feel like we've made that much headway.
David Cramer: I think.. I don't know. there's like two sides of this, right? Big tech has gotten more complicated. And so Sentry needs something like GCP to fully outsource the systems layer, but we're building our own platform there on top.
David Cramer: That's fine. And to do that and to deal with customers like Sentry, you do actually need at least some degree of complexity or available complexity, I guess. Yeah, I think that, well, the reality is that's where the money is, right? It's servicing customers like Sentry, not servicing my side project.
David Cramer: Yeah, at least for cloud vendors. And so I just don't think they're incentivized and or they're just not good at simplicity. Because they've tried, of course, like you've seen, at least I'm not familiar as much with AWS, but like you had App Engine and all this stuff from Google that tried. And so I think that's like one challenge of it.
David Cramer: I think there is, a truth that technology has gotten more complex for two reasons. I think one, we're trying to do things that we didn't do before. And so I think that's richer, sort of more native feeling, responsive applications, and that there's inherent complexity to do that versus I've got a web server and it just returns HTML.
David Cramer: And so, okay, a little bit of complexity there. I think there's also just a lot of complexity in the scale of the internet and all this stuff has got, it's different, you know. That actually doesn't impact most people, which is odd because the complexity is still there no matter what. If I just had to put my finger on it, in my opinion, it's like everything's more complex because some things are more complex, which is not an outcome to be fair, but it is, yeah, it is difficult these days.
Marker
---
Kshitij Grover: Do you think it's possible that part of it is, and I think you were getting at this, you know, Sentry has, say 90,000 whereas most of these infrastructure providers or infrastructure tools, I'm guessing a lot of them start with some very big customers who have very complex requirements because that's where the money is to begin.
Kshitij Grover: And so you end up with a roadmap that's largely driven by that complexity rather than a very opinionated kind of product forward, almost consumer looking product because I imagine Sentry is not really beholden to one big customer that's driving a checkbox set of features, right?
David Cramer: Yeah, and that's very much true And I hate when we have a few like significant customers. I do, I'm not happy about that generally speaking because it's fine that we have them, but I will not allow anybody to like cater to them in the sense of what we do has to operate on the many. And that's a very difficult problem.
David Cramer: But I think one, it's a lot easier to build something when you ask somebody what they want and you just give them that result. And that's what most successful companies do. hard to build something when you're kind of going after like a long tail market share kind of play where ACVs have to be lower, like more consumer apps.
David Cramer: Right. And I think it's harder because you often don't have the patience or you don't have the financial capability to be patient, you know. And I always tell people, cause like one, I think the greatest strengths we had when we started the everything and still today is like, we're just like, we're going to do something that we believe in and we're going to do it whatever way we think we is the most interesting or relevant to us.
David Cramer: We're not going to do whatever way it seems easiest or status quo or anything like this. And so it was always like, we want to do market share. We want to have like, as part of that, like low price point, we want to have it like self hosted for free and open source, with a big asterisk on it and things like that.
David Cramer: And those are not the straightforward decisions of building a successful business. Those are like, why would you choose all these like weird branches to make things more difficult along the way? And I don't think people often, one, are willing to take those sort of, call it risks, if you will. I don't see it that way.
David Cramer: But, and two, I just don't think people have a strong enough opinion about what the problem is they're solving in many cases.
Kshitij Grover: Why, why do you think, or maybe what gave you the patience 10, 15 years ago to say, we're not going to, you know, go and try to get a few million dollar deals and build a roadmap on them.
Kshitij Grover: Like, is it that you just think you had a very clear idea of what error monitoring or just business logic monitoring, let's call it should look like? Or is it just like, you know, you weren't interested in it for the monetization business aspect. And, you know, generally, genuinely is a side project that you wanted to productionize and gift to other developers?
David Cramer: I think it's interesting because by the time we were venture funded, we had a couple thousand paid accounts, making money. Not a lot. Our ACV was, must've been like 300 or something, terrible. Right. And there's a real challenge. You cannot do enterprise sales. And do bottoms up at the same time.
David Cramer: I just, I don't think it's possible. And you often see enterprise companies try to like now switch and add bottoms up and they all fail at it. Like nobody's ever pulled it off as far as I understand. And it's because you've got all these million dollar plus deals over here. How are you going to justify optimizing for these, you know, 30 a month, a hundred dollar a month things, right? And this is the same problem you see with cloud providers often. Right. And that's a, you've got to be, you've got to have like top down strategic belief in the model. Right. It's not a, you can't just react financially, otherwise it's not going to work. And so, we were always like, there was no way for us to sell a seven figure deal once upon a time.
David Cramer: It just didn't exist. Right. It's like, we can't charge enough money for errors. And so we knew we needed more products to raise ACV over a long period of time. And these days it's still hard for us to charge seven figures just for error monitoring. It's doable, but it's still very, rare and difficult.
David Cramer: And that's fine. You know, we just accept that's not the business we're in for the most part. And then we just focus on what the business is, you know? And I think it's fine, but it's a very different thing. And I think it's just, you got to decide one's probably easier than the other, I think seven figures is still hard to be fair, but like 10, 20, 30, 40K deals early on, like, here's a good analogy.
David Cramer: We had.. Delta airlines was, an early self serve customer. I don't know exactly what or how I just know they had an account. They're spending $29 a month. They would not expand ever. They didn't have the usage to expand. It was probably some internal R and D team doing who knows, right. That's okay. It's not fun when you're in sales and you see that because you're like, oh, that's an enterprise logo. I should be able to commercialize that. And the fact is you just can't because adoption creates commercialization and unless they're willing to, and have a need to adopt it somewhere else where there is just that usage of it, that is like consumer scale, we're just not going to monetize it well. And that's just, that's a hard thing to come to grips with because you can't manipulate growth then from a traditional field sales model. It's gotta be much more true product-led.
Kshitij Grover: And I mean, the other thing that is kind of complicating things, obviously is Sentry has this open source model which, which I think has been a mixed bag for the industry at large. But how has that played into this commercialization conversation and how do you think about, you know, big companies who maybe have a pretty big use case, just going and saying, we're going to use the open source software?
David Cramer: Yeah, I think the way I always talk about this is Sentry is open source or was open source. We're pretty close to it, but, just quick aside, Sentry has two models. A lot of our stuff's open source, true open source, free. And our core repo is open source after two years, but it's a license that allows you to non commercially utilize it effectively.
David Cramer: So you can just run it yourself, no strings attached, same code we run in prod. And so we used to be full open source. And again, this is only because we want to do business this way. We think it's a better path forward from, from an industry as a whole. And people would often ask like, well, how do you monetize open source?
David Cramer: Blah, blah, blah. I'm going to ignore all that because the answer is it's not related. Open source is effectively a distribution model. And so you got to think of that open source is marketing. And the way we always got value of open source is we would help these customers. They would contribute back once in a while, but not a lot. At the end of the day, it was mostly us building the software, right?
David Cramer: But the best story I have from early on is, Uber was a really big user of Sentry. They had a, the biggest installation bigger than our SaaS service. They had an installation inside of Uber. They're self hosting all this stuff. And when, when we venture funded, we're like, oh, this is gonna be easy. We're just going to convert Uber and Airbnb and all these people that were just like running like free open source. Right. Yeah, spoiler. We didn't convert any of them. But Uber is a good one because we actually did charge Uber at some point. A significant amount of money for the scale we were at. It was a six figure deal, but it was for like customer support and what might as well be, you know, consulting or professional services. Right. And very quickly, like this sucks. We don't want to do this. It's a distraction. And we did not renew the contract. And we were like, we just said, well, I don't think we can convert Uber.
David Cramer: And a couple of years later we realized it didn't matter because what we saw was all these companies like Uber, eventually they came to the SaaS service. But they didn't come from Uber in the sense of Uber never signed up, but all those employees, or a lot of employees, would move on to other, you know, roles at different companies and they'd be like, Oh yeah, I used Sentry at Uber, it was great.
David Cramer: Also, there's a SaaS service. It costs us like 30 bucks to set up. And that was phenomenal for us. And so that's where I go back to the distribution model. It's marketing at the end of the day. There's obviously other benefits of open source, but I think from a business point of view, that's such a big deal.
David Cramer: And I think it's very hard to believe that anybody is ever going to win again in the sort of IT infrastructure space against an open-source player that builds triple A product. And I wholeheartedly have that belief at this point. It may not be universally true yet. And again, it still matters.
David Cramer: It's gotta be like tomatoes to tomatoes kind of thing comparison. It can't be like, here's a mediocre, bad product that's open source, like a phenomenal product that's proprietary. But same, playing level, like open source is going to win because it just provides all these great benefits and beliefs.
David Cramer: And it's almost like, it's, a culture you're buying into, right? And I think people underappreciate that idea of open source. And that comes closely, like the last, like sort of, fork in that I want to go into is open source is, effectively for us, the way we think about it is brand marketing. It's not just marketing or distribution.
David Cramer: And what I mean by that is, I loved it, especially internally at Sentry these days. I talk a lot about brand marketing, brand awareness, and one of the examples I give is Liquid Death. Which if you're not familiar, it's a cool looking can of water. It's still water in a can. It's not a new idea, but they're not selling water. What they're selling is not beer. It's just a beer alternative, that just water, that looks cool. It looks like you're drinking beer. You go to clubs, they sell you Liquid Death. Now I was at a music concert. All they sold was Liquid Death, but they're selling not water. Right. And that's it. And it's like Sentry is not selling error monitoring. We're selling a philosophy and how we think about open source sustainability. Or we're selling a philosophy and how we think about privacy or product quality, you know, all these other things. And I think that is... that is not normal in tech for sure. But you see it a lot in consumer markets and things like that. And so that's the other part I think of open source. It really matters for us as it is a brand that we are really focused on.
Kshitij Grover: And I think to this point of it being a distribution strategy or brand marketing, it does feel like it's not the sort of thing that you can, you know, act on later in the company lifecycle. It really does feel like it has to be in the DNA of the company. Do you think that's true? Like, do you think it's, it presumably would have been very hard for you to espouse this philosophy years after Sentry started and maybe a related question. Like, do you think this is why there aren't more open source successful, large scale products? Just the founder DNA or the product DNA isn't there from the beginning?
David Cramer: So I think almost everything at a company has to be top down. In the sense of it has to be believed in at the top and enforced from the top. It might not come from the top, but for example, if it's like a bunch of engineers like open source and everybody in the executive team is like, whatever, we don't really understand or care about it.
David Cramer: It's not going to survive as a you know, as a core thing. So that, that matters. I think the reason it's less common is because it's very hard and it does have to be in your DNA. And it's not, it's, there's a bunch of these like attributes, right? Like Sentry is not successful because we're open source.
David Cramer: Sentry might be more successful because we were open source, but it's not successful because of that. And I think this is where the confusion comes in a lot. And so I think there's that aspect. But then there's a lot of this like fear and challenging, of like, well, what's the value of open source?
David Cramer: And we are open source just because like my entire career, I've just shipped open source software. I just didn't care, right? I'm like, yeah, it seems fine, you know, no, no problems here. But I do talk to a lot of people that like kind of go into open source and they don't, they maybe don't have a strong belief around why or what it is they think it's probably useful to do. Which is fine, you know, that's not a bad thing, but I think especially trying to go that direction after the fact, you can, don't get me wrong, you totally can.
David Cramer: Because Sentry, for example, where we have a bunch of this privacy stuff going on that we're going to talk about. We were not that early days. We're like, no, just give us all your data and trust us. We'll protect it. You know, it's useful if we have this data. Now we're like, you know what? We're actually going to give you a ton of controls and be like aggro pro privacy because we think like this is important now and it's changing.
David Cramer: So I think you can change first off. I think you've got to be thoughtful about what it is and what your risks are and when it does or doesn't matter. Right. And so Sentry was open source until one company decided they wanted to resell Sentry. One company that had never contributed to Sentry at all. And I'm like, this is annoying.
David Cramer: Like, why would we tolerate this kind of thing? And we were the IP holders, so it didn't matter. Now we could have been open source and that could have been fine. I don't know, but we chose to switch to this other licensing strategy where it's like, well, now we're protected. We can still monetize Sentry.
David Cramer: But I, think there's this interesting aspect where there's not been a company that's gotten so big and stayed true, free, open source and monetized extraordinarily well. Like there's no billion dollar revenue companies that are like exclusively open source, free open source software. And I think the free part is important because... what I mean by free is not free software foundation. I mean, free as in almost no strings attached in the sense of like not GPL. I mean, free as in permissive free, because GPL effectively has constraints that are attached to it. And one way some of these companies monetize is by scaring other companies from using their open source by putting these licenses that are really like high risk for certain organizations, right?
David Cramer: And so I want to believe somebody will do it at some point. They'll take that leap and it will work. We were not going to be the ones that quite do it because it was just frustrating for us. But I think it's doable. I don't think it's a limitation anymore.
Kshitij Grover: And I guess maybe last point on this open source topic, what do you think folks have to take away from, you know, things like Kafka or tools where cloud providers go and effectively take the software and resell it, in a way that, you know, presumably takes a decent chunk out of the, business of the company that created the software?
David Cramer: So one thing I was talking to a couple of friends the other day who are, considering open sourcing their core service and they're like, well, what kind of license do we use? Do we use the new same thing Sentry released recently?
David Cramer: Do we use a business source license, open core, blah, there's all these decisions. Right. And I told them, I'm like, the decision has to be inherently connected to how you're going to monetize because open source is part of the packaging models, part of the distribution model. It's the same way, like which features you get for 20 bucks are part of your packaging model. That's how you monetize everything. And so if it doesn't make sense, it's not attached to that, it's not successful. And so, you know, think about Kafka. If the only way to monetize Kafka is through cloud hosting and the whole goal of Kafka is to make it really easy to run well, I mean, newsflash, anybody that can sell it is going to win, you know, like whoever sells it best wins because there's nothing there. Like the packaging is just, let's make it dumb, easy to run. But if the whole thing with Kafka is say, like, you need all these tools on top or something else, right? Okay. Maybe it's fine all of a sudden. And it doesn't mean you'll have exclusive rights or anything. And that might be okay.
David Cramer: And I think there's this interesting thing that we've not seen enough data and you've got to take some risks to see of like, well... take, Sentry, for example, just cause it's, you can't really compete with us. If it was truly free open source, super easy to run and commercialize, so there could be Sentry alternatives in SaaS.
David Cramer: Would that harm our business or would it broaden the market for our business? It's really hard to say, you know, cause like you have MongoDB and Elastic who have these same kind of things with like AWS and whatnot, and they're still very successful companies. And so it's really hard to say that it hurt them at all. And to be fair, they've also always had proprietary components that they sort of shipped alongside.
David Cramer: Maybe in our case, those things were just not good enough. Like we had one product, which was the infrastructure component that had great product market fit. And then they had a bunch of products that never achieved product market fit. And that's why they couldn't monetize as well as they wanted. There's something along those lines, you know.
Kshitij Grover: Yeah, and it also feels like this weird tension of, you open source your core technology, and then you kind of actively work against that by trying to create closed source adjacencies that only you offer that someone else doesn't offer. It feels like.. It feels that tension with the original spirit of having open source, the thing in the first place.
Kshitij Grover: Right.
David Cramer: One, one point I want to make on that is this is one of the most challenging pieces and it's why Sentry's so...we have so high conviction around our model of like protect ourselves to sell the SaaS service to commercialize everything because we do not want to cripple the software to distribute it.
David Cramer: Right. And we're not a great case study. We're not infrastructure. We're just a product that we sell that's complicated to run. But it is like that business model. I think a lot of people, a lot of people desire to just be like, I just want to commercialize it via cloud services. But so do some of the biggest companies in the world.
David Cramer: And so that's sort of like where we're at with the tug of war, you know?
Kshitij Grover: I want to get back a little bit to the technical side of things. So Sentry obviously has to be reliable, has to ingest the data. And it also has to be really easy to set up with as little configurability as possible across a ton of different environments.
Kshitij Grover: How do you manage that product surface area? Like I'm assuming, you know, internally you can't possibly use every Sentry SDK. You can't test every framework. But on the other hand, you know, with that many companies and developers using Sentry, you're going to get bug reports across every framework and things that can go wrong will go wrong.
Kshitij Grover: That seems like a pretty challenging technical problem and scale. So, or maybe it's also an organizational staffing problem. How do you think about that?
David Cramer: So I'm going to use, I stole this from somebody else the other day. It's open source with a healthy, dose of capitalism. And what I mean by that is, Sentry for like, as long as it's had money has paid contractors to help us maintain those as our SDKs. And cause it is a staffing problem. It's just a sheer breadth of problems to solve and expertise. And those people we often pay, we're just contributors. We're like, hey, this is great. You want to keep contributing and just have money also. Which I would encourage more people to do.
David Cramer: It's a phenomenal model if it works. But it is a really hard challenge and I'll tell you it gets harder and harder because once upon a time, it's like, well, Django and Rails and Laravel, and they're, kind of, they're big targets are like a lot of people use one thing. You solve it once, you tweak it a little bit, every so often maintain it and you're fine.
David Cramer: That's become more and more problematic in recent years as the, like the change in user interfaces with the, like, basically with the JavaScript movement, you know. You have some of this on mobile too, where there's a lot of change in the, how we develop mobile apps and everybody wants like a new language or a new framework all the time.
David Cramer: And that shouldn't be such a problem, but the problem is most of these frameworks. Well, one, sometimes they're, built by people who have not really run like production operations and software. And so they're not always designed with... we need telemetry in mind, you know? So that's one challenge.
David Cramer: I think the other challenge is even when they are run by these people, they're so complicated sometimes that, and again, the same problem, they've not really thought about, hey, like people actually need to be able to diagnose all these problems in production. And that list gets endless, you know, and you see that a lot with the JavaScript frameworks these days where they've gotten so complicated to debug.
David Cramer: And the frameworks themselves, because there's so many moving parts and it's not like they often don't control the whole thing, it's very hard for them to even simplify that. And so in an ideal world, new framework comes out, it's built in some core hooks, you know? And we just, we shim those hooks, right?
David Cramer: And then for us, it should primarily be, do we support that language really well? And then do we support the framework by some hooks? But what I'll tell you in practice, it is, we spend lots and lots of engineering hours supporting every framework under the sun. And it's a never ending moving target. That is very, hard to do.
David Cramer: I don't know what our total staffing is behind it. But I do think if you include, we have SDKs, right? Which again, we either contract or full time engineering staff, pretty much any SDK that's run, and it's just more than one headcount per mobile, for example, I think we have nine people on, which is insane to me.
David Cramer: But, so that, that's one. And I think the other side you have to understand is it's not just like an SDK. There's all these like language sort of runtime specific challenges. You think about like, for me, SDK is like, it's next JS or it's Django. Well, you've got AWS Lambda serverless shenanigans, right?
David Cramer: And then you've got everybody else's flavor of serverless and they all work differently and they're all their own frameworks on top of everybody else's frameworks. And it's like, I don't know. I think this problem is unsolvable in its current shape. And I don't know what we're going to do about it in all honesty.
David Cramer: Like a lot of people want to believe like open telemetry and stuff's the solution. I will, tell you it's just not. Because it's just not, it doesn't solve it well enough in so many scenarios, especially in the front end where the problem is the biggest. And open telemetry is like ignore the logs and the metrics, but the tracing component, that's good, but it's only one little component of everything we need to do.
David Cramer: And Sentry is not going to like create some open errors spec, let alone cater to open telemetry trying to do it because one, it's big company process shenanigans. Two, we are so invested in our entire, you know, thing. We're not going to spend tens of millions of dollars just to make it easy for more copycats to show up, you know?
David Cramer: And so there's all these challenges, you know, and I just don't know what the solution is going to be. But I will tell you forever, it's always just like, we have to invest in it. We have to be really good at it. And when we're not, we have to recognize it and, double down on that investment, you know?
Kshitij Grover: But, that is interesting cause it sounds like, yeah, one shape of solution to that problem, which may or may not work is try to get an RFC out there. Try to make these frameworks move in some direction rather than you having to staff every incremental framework that comes out. And it sounds like you're saying that's probably not the right solution because it, yeah, in some sense lowers the barriers to entry to market, but also maybe it's just unrealistic for that to actually happen.
David Cramer: Yeah, I think it's unrealistic because what often happens with standard bodies is they develop a standard and it's fast right away because there's no adoption and then there's adoption and it stops.
David Cramer: And even when there's adoption, it's not universal adoption. There's some people that are like, I don't like this. I'm going to do it my way. And so even though I think open telemetry for the tracing specific stuff, so open tracing back in the day. There's a lot of great ideas in there because we should not have to instrument a bunch of third party libraries over and over and over.
David Cramer: Yeah, the adoption. This is like a many years old project at this point, and it's still got such minuscule adoption in the industry that I'm like, if that's the outcome, it's like, we're definitely not wasting time on trying to create a standard, you know. And so I don't know what the solution is. But one interesting analogy to that problem is recently we released something called Spotlight Essentiary, which is actually unrelated to Sentry, but all we did was we basically took our SDKs and we leveraged the data collection to build like a debug bar locally, like a better debugger in, in development environments.
David Cramer: And we did have to change the SDKs. Fortunately, we control them, so it's easy. But we were actually able to take something we had already built and leverage it into something that's entirely different. That just needed similar kinds of data, you know. There's this interesting concept in there, even though it's not like an open standard.
David Cramer: And we're certainly not going to change our SDKs to help a competitor. We were actually able to expand the market capabilities using what is our internal standard, you know? And so, I don't know, maybe, we'll just keep funding all this until the end of time and the number of engineers on SDKs will be like 500 in five years, I don't know.
David Cramer: But I, I like to believe some, it's becoming more of a mature industry. And I think there's these curves where it kind of expands in terms of like complexity and ideas, and then it contracts, as, things start to mature. We haven't seen a contract yet, unfortunately, in like the job control, especially.
David Cramer: But I would like to believe we're getting more mature about the technology these days. And there's a lot more people with, a lot more depth of experience. Building some of those projects versus, Oh, I just live in the PHP ecosystem. And all I do is build PHP. So I don't know what happens over here. We've got a lot of breath in the industry now.
Kshitij Grover: Maybe talking about a, I guess, relatively significantly more immature problem. I've noticed that you all have also started to build some AI features into Sentry, but, in a very like Sentry like way where, in some sense, it's very transparent that these things are guesses that you're not trying to pin down the problem, but you're trying to directionally suggest what might be the issue.
Kshitij Grover: We can talk about the future, but what I'm wondering about is more philosophically, do you think that's here to stay? Do you think that right now it's kind of a toe dip into AI just to see if it lands with developers and you're really going to solidly focus headcount and technical investment into actual proven problems with maybe an opinionated stance rather than let's just do AI because AI is the thing.
David Cramer: Yeah, I think a lot of people are doing AI because AI is the thing and I will tell you. I've I personally have very little confidence. A lot of the spaces make sense for it I think there are some interesting things that the language models can work on today. And I'm not an expert by any means but I have I've been playing with them. And then there are things they are just absolutely horrendous at, and I'm not a, I'm not a believer of like, oh, shiny new technology, let's force its way into something, you know. And that feels like what's happening in a lot of ways versus like, oh, obviously this is powerful here. And I think it's fair that they don't know when they're testing it. But every company in the world is like, look at our AI chat bot thing. I'm like, I don't need an AI chat bot, you know? And so ours is we actually just spun up a dedicated ML AI investment, but it's not a dedicated LLM ChatGPT investment. We are interested to see what makes sense there, but we're thoughtful about it from a point of view where we're realistic. And we are not just going to build something that anybody else can build.
David Cramer: Because frankly, a smaller company is going to be better at building something quickly than us. So it's like, well, what could we uniquely do that others can't? And that's like, well, what can we uniquely do with the amount of interesting telemetry we have? And things like that. And so a lot of the near term investment for us is kind of like, well,it's not so much that the LLM tech is interesting. It's that we should run more experiments. We are to scale where the money is irrelevant to us, the staff, like a small team, you know. And if something ends up being very valuable, well, that the ROI is really, large. And so I think we'll see, as an example, CodeCup, which is a company we acquired a while back, who kind of does, I like to think of it as like Sentry pre-production.
David Cramer: And so can we stop bugs from happening? Even though it's mostly focused on test coverage right now, but they actually have, it's like not, documented or announced, but there's something we built internally that is not good at all yet. Maybe it will be good, but it's just a code review buddy. And I'm like, can somebody just do this and have it evaluate the code and tell me what's broken?
David Cramer: Because again, the language models are not perfect. But I have seen in my own code where I've given it a complex function and it has identified something that I could not identify myself. And if it can do that, can it make code review more efficient or something? Now that's not a uniquely us problem, of course, but I'm like, there are areas where it does seem interesting, I guess, but I think a lot of the technology: the language models are either too expensive or they're just not capable enough yet to really pull something off that's great.
Kshitij Grover: But it sounds like basically they've reduced the barrier to entry enough where a company like Sentry, it makes sense to run more experiments involving them, even if it doesn't mean releasing full-fledged features that use them just yet.
David Cramer: Yeah, I will say that the complexity for Sentry is we're not going to funnel data to OpenAI. And so we have to run models, behind our firewall and like, GCP, we just, have this intense policy around data processors. And so we do have like an OpenAI feature, but it's opt in for this one single feature, meaning we couldn't all of a sudden make like a network-wide behavior.
David Cramer: So we're figuring out some of that, but the technology I think is exciting. But just like every new piece of technology, it's a little bit overhyped, I think. And you saw this with Bitcoin and shit, even though that was more like, how do we launder money? It was like, oh, can we apply like a distributed hash blockchain to every problem in the world where most of the time it makes no sense.
Kshitij Grover: Yeah, awesome. So maybe last question I'll ask is less on the technical side, but I'm curious about this. Sentry's brand is very unique. And as you were saying, a lot of these things I imagine do come top down, but you know, you all have billboards around Sentry can't fix this with this like phone falling on you. The text on the marketing site, in the product is really silly. What's driving that? Is that just like, again, your ethos and like kind of leadership ethos? Or do you think it's something to do with, you know, people who use Sentry are presumably stressed at times, right? And is there something around the actual product surface?
David Cramer: This is us just a strong belief in brand marketing. So part of our brand is be authentic, you know, and it's okay to be tongue in cheek. There's a, balance there, of course it's not, our brand is not "talk shit." My brand is "talk shit"to be clear. But our company brand is not. It's just like, let's tell it how it is.
David Cramer: So my favorite analogy here, or my favorite example of this rather, is, in our, in, Sentry's core, we have a single page applications. It's, very large these days. And when we built it, which it must be 2015, react SPA, we put it in a loading indicator we thought was funny. It looks cool.
David Cramer: And it's just like, please wait while we load an obnoxious amount of JavaScript. Back in the day, you never saw it. Now you see it all the time and it's very true. It actually wasn't that true back in the day. It still kind of felt like, and that's phenomenal. Like people love it. It's like tongue in cheek.
David Cramer: It's just us like sort of like demeaning ourselves, you know? But we actually have brand voice policies internally that are all centered around this. And so I think it did come from the early team. But I think great ideas are not just coming from a person or a set of people. There are things that people value and then latch onto.
David Cramer: And so that one's like held pretty well for us. And so I'm partially responsible for the billboard campaign and my voice is very strong, but other people took it and ran with it and then made it better or something. Cause they took...the idea was like Sentry can't fix this. First off Sentry doesn't fix anything.
David Cramer: And secondly, the things that we depict have nothing to do with anything Sentry does. So like the first one or the main one was, it was a minotaur, like the magical creature playing Beat Saber and knocking over some pots. And it's like, what the? What even is that? You know, that's the whole point.
David Cramer: It's like, a billboard. It's like, you're looking at it and you're like, what is that? And I'm like, what the hell is Sentry? And that's kind of the thesis behind it. But a lot of it just like, yeah, that's like, there's a lot of stiff corporate marketing and there's no reason to do it again. It's just a belief we have, you know?
Kshitij Grover: Awesome. Well, here's to more billboards and in the coming year. And thank you so much for taking the time and coming on the show.
David Cramer: Yeah, absolutely. And thanks again for having me. This was fun. Awesome.
