[00:00:00] Sagar Batchu: Hi everyone. This is Sagar, CEO of Speakeasy.
[00:00:02] And today's second ever episode of Request Response. We had Robert Ross, AKA Bobby Tables, CEO of FireHydrant. We chatted about his journey to building FireHydrant, his experience building with gRPC, and why flying on airplanes is important to finding great developer experience.
[00:00:19] Lastly, will LLMs make API design easier than it is today, or will it make our lives more complicated? Join us to learn more.
[00:00:27] Hey, everyone. Welcome to the second ever episode of Request Response. I'm your host, Sagar Batchu, CEO and Co-Founder of Speakeasy. We are the modern tool chain for REST API development. I'm joined today by Robert Ross, CEO and Co-Founder of FireHydrant. Robert, it's great to have you here today.
[00:00:42] How's it going over there in New York?
[00:00:44] Robert Ross: Oh, thanks for having me. It's a rainy and snowy day, so it's going great.
[00:00:48] Sagar Batchu: Just to help our audience here, I'd love to learn a little bit more about what FireHydrant does, your role there today, and just how you got to this place.
[00:00:57] Robert Ross: Yeah, so FireHydrant is the end-to-end incident management platform. We've been around about six years now, just over six. And my background is I've been building software for a long time, was an on-call engineer for a number of years. And sometimes I was putting out the fires,
[00:01:15] sometimes I was the arsonist and what led to the creation of FireHydrant was the,
[00:01:21] you would get paged and it was kind of a, "now what" moment, and it was a lot of automation missing from my tool chain. So started going to a Starbucks early in the morning here in New York City, built the first kind of iterations of FireHydrant, and a few months later, accidentally started a company.
[00:01:36] So, here we are today.
[00:01:37] Sagar Batchu: That's an inspiring story. I mean, I can definitely relate. I haven't spent that many years as an on-call engineer, but those that I did, a lot of chaos every time something happens, something goes wrong and then suddenly you're like, what do I do? Who do I contact? How do I coordinate the situation?
[00:01:52] So. No, it's super cool. I've checked out your product and I wish I had it many years ago at different companies.
[00:01:59] Robert Ross: Not now? Maybe soon, maybe soon.
[00:02:01] Sagar Batchu: Yeah. Yeah. So, you know, shifting gears a little bit. I'd love to talk a little bit about the role APIs play for FireHydrant today, maybe just to kick it off, do you guys have a public API and is it important to your business model at all?
[00:02:16] Robert Ross: Oh yeah. So actually when the company first started, I had just been, this was like 2017, even 2016, my job at my last job was to build the core services of our application. And the way I approached it is I was using gRPC, we had gRPC gateways in front of everything. And what we started doing is we started moving the backend Rails application
[00:02:46] to a lot of that logic to an API and then making the front end React at that point in time basically making API calls to that internal API. And I saw the benefits of it, but I also saw how painful it was to move to that pattern given I was doing it. So when FireHydrant started, I actually said from day one, that's going to be the architecture. There's not going to be Rails controllers that are, you know, whatever your MVC framework of choice is, there's not going to be a backend rendering the dynamic content that then is sent through NGINX.
[00:03:21] It's always going to be a JavaScript front end making API requests to a backend. So I started that very early on, which might've felt like over-engineering at the time, but now I think we're well over 350 public API endpoints in FireHydrant, all of our front end, it uses those API endpoints.
[00:03:41] Which is nice because now our front end team can actually make adjustments without a backend deploy to the application. It's had a lot of downstream effect benefits since then.
[00:03:50] Sagar Batchu: You know, I have to ask an age-old question, why did you pick gRPC for that transition?
[00:03:56] Robert Ross: You're shiny. I mean, I joke, but that was actually the case back then. And we had some really forward-thinking engineers at the last job I had and we were using Go and we knew that we wanted like a contract-based API system for internal and external and gRPC, you know, had just come out, we started playing around with it, we built tools for it and it ended up kind of being the standard.
[00:04:25] So, you know, it's funny, that was what we were doing at the last job, but FireHydrant doesn't use that pattern. And we kind of have a raw JSON REST API backend built into our Rails application. If I were to do it again, though, I would a hundred percent start with protocol buffer definitions of that API and go about it that way.
[00:04:44] Sagar Batchu: Is that what you would suggest to people building a backend from scratch today? Like, is your advice a blanket Go with gRPC or is it something more nuanced?
[00:04:54] Robert Ross: Yeah. I don't know if it's like Go and gRPC. I think when I say gRPC, really what I'm talking about is the protocol buffer definition and the IDL of that - for the folks that don't know, interface definition language - of protocol buffers and the advantage of it is really it allows you to explain your API without implementing your API very quickly.
[00:05:19] And what I also like about it is you can have a debate with your team. You can have a pull request of just your protocol buffer definition and your team can see that and go, that's a good interface design for this API. And the other advantage of it now is,
[00:05:36] and this has existed for a while, but there's new tools out there that make this easier for generating stubs and the code generation surrounding the protocol buffer tool chain has improved dramatically in the last 10 years.
[00:05:48] And now I kind of look at it and go, there's not really a reason not to implement it this way. Because I mean, you get to explain your entire API and then you also for free get an OpenAPI 3.1 file on the other side of that. And then you can use a tool like Speakeasy to then generate API documentation and stubs for many other languages in a Terraform provider. I see very little pitfalls of doing it that way.
[00:06:17] Sagar Batchu: Yeah, that makes sense. I mean, I resonate with that so much. Like the, like you said, a big reason why we exist is we want to bring a lot of that native code generation experience that gRPC and protocol buffers has to the world of REST. And if I flip back a couple pages in my book, I think there would have been so many situations where, you know, that idea of
[00:06:40] a team being able to do true design-first development would have been enabled by protocol buffers. Like, I think I tried this at a previous company with a natively to an OpenAPI spec and just the verboseness and the kind of lack of focus on design made it actually a really hard you know, spec to work on for design-first experience.
[00:06:59] So that, you know, a lot of people, when I asked them, this will kind of focus on the tech and the code gen and all of those pieces. But I think you brought up a really important point around the collaboration experience that I think, you know, especially startup engineers, we forget because as a company gets bigger, that's kind of probably the limiting factor for development.
[00:07:19] Robert Ross: Yeah. Yeah. We use protocol buffers now in our systems that talk to each other. So we have a Rails application and we have a Go application. We have a planet-sized Rails monolith with like many moons of Go microservices surrounding it. And the interfaces between those applications is protocol buffers and gRPC and we like the code generation of it.
[00:07:42] Like the protocol buffer definition only gets you so far, though. It's going to give you a pretty myopic client, I should say. So like, there's other things that you have to consider such as retry policies, error handling. You have to consider like, if you want to go further with it, like I said earlier, like a Terraform provider from these things is possible with your folks' tool, which I think is amazing.
[00:08:05] So, you know, there's trade-offs, you just got to figure out which ones you're willing to take on.
[00:08:10] Sagar Batchu: Yeah, that makes a lot of sense. You mentioned, you know, over 350 public endpoints for FireHydrant today. Like you must see a lot of patterns of consumption by enterprise with your API. What's shifting in the last few years?
[00:08:23] Robert Ross: Oh, what's shifting. You know, I think more and more people are willing to pull data from your API. There used to be kind of a requirement many years ago of like, you need to push data to me, into my data lake. And the problem with that is, sure, we can push data all day long, but it may not be in the format you want.
[00:08:46] It may not be, you know, enough, it may not be, like, frequent enough. Like, you might have stale data if we're doing, like, a nightly push to our customers' data lakes. Like, that's not fast enough in some cases. So, a lot of our customers are scraping our API every five minutes in some cases to get their incident data, you know, we have our most popular endpoint is our get incidents endpoint.
[00:09:15] And what our customers tell us is that they have data analyst teams. They have data analytics engineers that their job is to write scripts that pull the data into their internal data lake from all of their vendors. And I think that's, I don't know if it's a new trend or the new normal, I'm not quite sure, but it's definitely something I've noticed is that enterprises are spending more time building the teams to get the data in the exact format, the exact time that they want it, as opposed to having the requirement of push the data to me.
[00:09:47] Sagar Batchu: That's super interesting. So it's like they're taking data from API, populating some kind of data warehouse BI system for their own work. Have you guys at all thought about like exposing the data directly in a more native way, like to a data stream or to just you know, tables that they could materialize themselves?
[00:10:05] I'm curious if that's changed your viewpoint on actually just having a traditional REST API though.
[00:10:11] Robert Ross: Yeah, a few things that I've considered is webhook, you know, fire hose out of FireHydrant. Actually, that would be a really perfect name for that, wouldn't it?
[00:10:23] Sagar Batchu: Really.
[00:10:25] Robert Ross: Hook up the Firehose to FireHydrant. We do webhook delivery today. Internally, our architecture is all PubSub events. So anytime you create an event in FireHydrant
[00:10:35] such as like a note, a chat event, updating an incident, that goes to a published subscribe model. And then if you want, you can send all of that data via a webhook. We do that today. And I think that you can go further with it. I think that you can consume from a stream especially if you have like an internal audit logging system, can you publish events and things like that.
[00:10:58] And you kind of let your customers materialize the data the way that they want it. I'm seeing that pattern more and more. There's a company called Offset that does this very well with their materialize RBAC system. I think up there, it's actually a pretty good model for where maybe the industry is going to go with materializing and sending webhooks.
[00:11:19] Sagar Batchu: That's really interesting. Yeah, we're seeing some interesting shifts there as well. Like on one side some people are considering opening up their, you know, Kafka data streams directly to customers.
[00:11:29] Robert Ross: That's what I had in mind. Yeah.
[00:11:31] Sagar Batchu: Yeah, yeah. I love the name firehose for FireHydrant by the way, I feel like now you have to launch a product.
[00:11:36] Robert Ross: I, yeah, I actually, yeah, my engineers are going to love that spur of a moment idea on a podcast that they haven't built.
[00:11:42] Sagar Batchu: Right, right. Well, drive-by development. Yeah, that's really cool. You know, I mean, to make shifts like that, something we think about a lot is like what metrics should companies actually measure for API success? Because, you know, if you're going to go do any kind of big technical change or, you know, anything that actually impacts the experience, you have to know what you're measuring.
[00:12:02] And I think in the public API world is actually quite a differing kind of set of viewpoints on what you should look at. I'm curious if you've ever thought about like what KPIs are important for the FireHydrant API.
[00:12:13] Robert Ross: Yeah. I mean, latency is, time to first byte is one of the most important ones for us. We have a couple of pretty slow endpoints. I mean, we just represent so much data from that endpoint that we made some mistakes that we can't go back on now, which is an issue with API design that you have to be considerate of.
[00:12:34] But I digress. I mean, definitely time to first byte. How long is it going to take when you get the request from me, how long do I have to expect until I start getting that request back? And I think that's one of the most important measurements, especially if you're using your own API on your front end.
[00:12:51] You get a little bit more leniency if this API is just for you know, programmatic client calls. Right? Should it take several seconds? No. That's probably bad, more worse for you than it is for them, if you're taking a few seconds. Couple of other KPIs. I, you know, I think obviously like, do people have to retry requests to get data?
[00:13:16] That's a KPI that we measure because you can get into a situation where you have a thundering herd issue and what are the leading indicators of, well, if you're slow and the client call times out and it doesn't automatically retry and you have thousands of customers that hit that just because your API tips over three seconds, all of a sudden your API is going to get smoked even harder.
[00:13:38] So you kind of have to set your KPIs a little bit lower down the chain.
[00:13:43] Sagar Batchu: Hmm.
[00:13:44] Robert Ross: So if you have an SDK, for example, that is Ruby or Go SDK for your API, your read timeouts should be less than the API's timeouts. That way you kind of have a buffer in between, you know, the client code. So that's one thing I definitely think people should be measuring and being conscious of.
[00:14:02] I can't honestly think of another one. I mean, like, usability of the API is really hard to programmatically measure, other than going to the people using your API and asking, how's it feel? I think that we have just some very old decisions in our API. We're still on V1. Like, we have not done an API update, like, structurally
[00:14:24] in six years of starting. It really is the same API endpoint you got six years ago. It's the same one. Maybe it has more data in it, but it's the same exact structure. And maybe that's a KPI. How often do you have to change your structure?
[00:14:40] Sagar Batchu: Yeah, these are definitely interesting ones. I mean that like, so, you know, time to first byte absolutely resonates with me. We call it internally time to first 200 or time to 200 for
[00:14:49] Robert Ross: I like that.
[00:14:50] Sagar Batchu: Yeah. I've always thought maybe we should, if we ever get into the request tracking game, we should
[00:14:55] throw up a metric in our dashboard that tells the user what the, you know, current time to 200 on the SDKs is.
[00:15:01] Robert Ross: Like that.
[00:15:02] Sagar Batchu: Yeah, I think another thing that's interesting is the API design problem you brought up. Like that is a really tough KPI, I think, to measure in any meaningful way. We've, you know, a lot of customers come to us with OpenAPI specs and a variety of quality and state of kind of maturity,
[00:15:18] and today, you know, we consume all of them. We give them, you know, linting warnings and errors, but I've thought a lot about whether we should give them some kind of score, right? Something to tell them that you know, the design of API today is flexible or it's not going to like block future decisions.
[00:15:34] It's a really hard thing to do because I think, you know, industry to industry, there's also, you know, different requirements like the heavy requirement around pulling data from your API, like that might not exist for another API. So I think where we've gone to is we definitely tell our customers and people who work with, like, do you have someone just thinking about the spec design as it changes over time?
[00:15:57] Robert Ross: Yeah. I mean, the ergonomics of an API are almost
[00:16:01] more important. I mean, you know, when you go to the furniture store and you're trying out sofas, which one do you buy first? You buy the one that's the most comfortable. API is the same thing. If you have a really uncomfortable API and that's important, like people aren't going to buy that product if that's a big part of their purchasing decision.
[00:16:20] And you know, like, for example, if you're comparing Jira to, I mean, I think Jira has the one of the worst APIs in like human history. And I can say that with confidence because we have an extremely intricate Jira integration and the amount of like backflips that we've had to do to make very simple things work with the Jira API is astounding. We also have a Linear API integration when, you know, basically it's the same thing. Create an incident ticket in Jira, create an incident ticket in Linear. The Linear code in the Linear API made it so that integration we built it in a matter of days. Like it was drop-dead simple because the API was so good.
[00:17:12] And I think we're going to enter into a pretty interesting reality where the companies that have very, very simple APIs, maybe not simple, but easy to integrate. Like the ergonomics of it are really good. Those companies are going to see longer-term success because more people are going to want to integrate with that tool. If I'm asked right now, if I had a gun to my head that said, like, pick a tool to integrate with, I'm going to pick like Linear right now. Well, after FireHydrant, but I'm going to pick Linear because they've just done like an astounding job with their documentation, with their API design, they use GraphQL. That was their choice, not mine, but it didn't matter because the client SDKs were so good.
[00:17:53] Sagar Batchu: That's really interesting. Yeah. Especially in these competitive markets where, you know, over time the product differentiation kind of deteriorates, like these kinds of things really start to stand out for sure. Switching gears a little bit, you know, we're just in the world of dev tools, developer platforms, a lot of paradigm shifts happening with LLMs and AI.
[00:18:14] I'm curious how, like what your point of view on this is when it comes to the APIs that your customers use or the APIs that you guys design. Has it like changed customer demand in any way or people asking to integrate in new ways?
[00:18:27] Robert Ross: I would say our approach to APIs has changed, which I think will have downstream effects. I mean, now when I've been designing new APIs, I actually will kind of vet the design of it. And yes, this is me admitting I'm still writing code six years in as a CEO of FireHydrant. I've talked publicly about that though, but I kind of vet the ideas of a new API against LLMs now, honestly, they have pretty good opinions
[00:18:55] on API design, especially if you prompt it with, I need my API to be it's in, it's a gRPC service definition. And you know, there should always be a verb before the request message or something like that. Like you can kind of create your linting of an API, your design principles, and you can ask LLMs, like, what do you think about this?
[00:19:22] You can kind of have a conversation with it. And eventually, honestly, I've actually had some pretty good, at least in my opinion, internal API decisions come out of that tooling.
[00:19:32] I don't think that LLMs are going to fundamentally change how we interact with APIs. At least not in the short term.
[00:19:39] I think that they'll help us build better clients and I think they'll help us build better APIs, but the moment we start using them, I mean, that's just going to be code calling code. I just don't think that LLMs are going to get too integrated at that point.
[00:19:53] Sagar Batchu: Yeah, that's really interesting. I, then the design part you know, we probably see like a couple, I don't know, a dozen hundred specs a week and a lot of them, I, you know, we have our own tools, but I've also been kind of just copy pasting them into, you know, those that are public specs into an LLM and just
[00:20:12] talking about it. Right. Like kind of semantically trying to work through what the spec is, what the state of the spec is. Yeah, it's super, super interesting just in terms of like being a guide right for you. Just the way we're seeing Copilots become a guide in IDE.
[00:20:27] Robert Ross: I can see like one opportunity where like a good API design with LLMs would play out very nicely. So Agentic systems is the next probably big topic in the realm of AI, I think. If not, you know, it's going to be a big part of the pie chart of discussion. And if you think about the way that these Agentic systems work, you have to give them data to then keep having a conversation until it ends up in like some result, either through tool usage or, you know, some pattern.
[00:21:01] And a big part of the tool usage that these LLMs are performing. We use Anthropic and Claude, which has really, really good tool usage APIs. We basically give it like raw JSON responses in some cases. And if that JSON response is bad, then it's going to have a bad result. And it's kind of making me rethink, even just the key names of these JSON responses.
[00:21:28] So for example, for an incident, the resource, the URL is slash incidents, right? But the payload is name. And if I give that to an LLM, all it gets is name. So are we going to have to start prefixing things with incident_name so that these LLMs actually have a little bit more context, a little bit more, you know, data to go off of, are we going to have to go back to the world of where everything is namespace under a key incident with an object of name description, right?
[00:22:02] Like our API doesn't do that. If we were to change that, would that make our LLM responses for either us or our customers be better? I don't know. I think we're going to have to see, but those are the types of questions I think API design is going to start driving Agentic systems and, you know, catch trying to buy vice versa.
[00:22:19] Sagar Batchu: Yeah, totally. I mean, like on that bit about just the names, right? Like API naming is already hard enough and I think something without giving too much away on like an upcoming release, we're planning to do in this space, when you think about tools, like if these tools are going to be used mostly by agents, does the verboseness of the name matter?
[00:22:38] If you can stick in a bunch more context by prefixing or suffixing a name, like maybe you should do it. And as long as it's not a tool that's going to be used by a human, it doesn't really matter, right? Like kind of the conciseness and all of the, I think you know, really strong points of view people have on ergonomics today on design is kind of human-led, right?
[00:22:57] It's to help a human looking at the docs of the SDK methods in for what a method is about. But that doesn't necessarily, it's not the same for an LLM, right? Maybe you do want to stick as much information in whatever, you know, token window you have that you can give it. So, yeah, no, really, really interesting.
[00:23:14] I think it also means from an API design perspective, like I wonder if we get to the place where there are APIs for human integrators and APIs for agent integrators in terms of just ergonomics. Like there's probably going to be a split the same way. I think they're going to be like code bases that are fully human administered versus code bases completely driven through bots and copilots because the quality and the verboseness of the code is - those metrics change.
[00:23:40] Like it doesn't, you don't need to do the same thing in both of those cases.
[00:23:44] Robert Ross: Yeah, no, I agree with that.
[00:23:47] Sagar Batchu: Yeah. You know, just to round this out I'm, you know, with all this stuff going on, like, what does, you know, does great developer experience still mean the same thing to you today? Like, has that, has your point of view changed on that over the last few years?
[00:24:02] Robert Ross: When it comes to developer experience, I think that my guiding principle is like,
[00:24:06] is it easy? Like, how far can I get before I have to read documentation? And I think that includes like great error messages, like, if I can do something on an airplane with really crappy wifi and like be productive, that's great developer experience.
[00:24:22] As someone that flies a lot, you know, that's when I build things honestly, is like when I get on a plane and that's when you really feel bad developer experience because things are slower or you don't have internet, you get an error and you're like, this is the most cryptic error I've ever seen.
[00:24:39] You go to Google it and Google takes five seconds to load, like that's when it reveals itself. So, and I think it ties into, you know, API design in the same way. Like, do I have to look up your API? If I do like, then that's frustrating. If I'm using a Go SDK to use your API and that SDK has a method that doesn't explain the parameters, like I'm going to be mad at you. That's really bad developer experience. So, that's kind of my litmus test, how much can I build on a plane? And if I, you know, it's gotten to the point now where I just know that I have to run Docker build before I get on a plane. Like that's the habit I've had to build lately. Cause the moment I get on a plane and I start my, I go into dev mode and it boots and it's like downloading layers.
[00:25:30] I'm like, I'm not building anything on this plane today.
[00:25:33] Sagar Batchu: That's a pretty good soundbite. Airplane mode is the true test of developer experience.
[00:25:38] Robert Ross: I think so, yeah. I've never said it that way, but yeah, I think just coined my term for it.
[00:25:44] Sagar Batchu: Alright, I'm gonna take that one away with me. You know, on that note, thank you so much for joining us here today. You know, this is our second ever episode of this podcast. You know, thoroughly enjoyed talking to you here. Yeah, I'm looking forward to chatting more in the future.
[00:25:57] If our listeners want to learn a little bit more about FireHydrant or you and the stuff you're working on, where can they go?
[00:26:01] Robert Ross: You can go to FireHydrant.com. That's our domain. I'm Robert Ross on LinkedIn. If you have any questions or @Bobbytables on X.com, formerly Twitter, and Bobby tables on GitHub. If you're curious, if I'm full of it.
[00:26:16] Sagar Batchu: I think there's a backstory to that name, but let's save that one for next time you're on.
[00:26:21] Robert Ross: Yeah. Thanks.
[00:26:22] Sagar Batchu: Alright. Thank you so much, Robert.
[00:00:02] And today's second ever episode of Request Response. We had Robert Ross, AKA Bobby Tables, CEO of FireHydrant. We chatted about his journey to building FireHydrant, his experience building with gRPC, and why flying on airplanes is important to finding great developer experience.
[00:00:19] Lastly, will LLMs make API design easier than it is today, or will it make our lives more complicated? Join us to learn more.
[00:00:27] Hey, everyone. Welcome to the second ever episode of Request Response. I'm your host, Sagar Batchu, CEO and Co-Founder of Speakeasy. We are the modern tool chain for REST API development. I'm joined today by Robert Ross, CEO and Co-Founder of FireHydrant. Robert, it's great to have you here today.
[00:00:42] How's it going over there in New York?
[00:00:44] Robert Ross: Oh, thanks for having me. It's a rainy and snowy day, so it's going great.
[00:00:48] Sagar Batchu: Just to help our audience here, I'd love to learn a little bit more about what FireHydrant does, your role there today, and just how you got to this place.
[00:00:57] Robert Ross: Yeah, so FireHydrant is the end-to-end incident management platform. We've been around about six years now, just over six. And my background is I've been building software for a long time, was an on-call engineer for a number of years. And sometimes I was putting out the fires,
[00:01:15] sometimes I was the arsonist and what led to the creation of FireHydrant was the,
[00:01:21] you would get paged and it was kind of a, "now what" moment, and it was a lot of automation missing from my tool chain. So started going to a Starbucks early in the morning here in New York City, built the first kind of iterations of FireHydrant, and a few months later, accidentally started a company.
[00:01:36] So, here we are today.
[00:01:37] Sagar Batchu: That's an inspiring story. I mean, I can definitely relate. I haven't spent that many years as an on-call engineer, but those that I did, a lot of chaos every time something happens, something goes wrong and then suddenly you're like, what do I do? Who do I contact? How do I coordinate the situation?
[00:01:52] So. No, it's super cool. I've checked out your product and I wish I had it many years ago at different companies.
[00:01:59] Robert Ross: Not now? Maybe soon, maybe soon.
[00:02:01] Sagar Batchu: Yeah. Yeah. So, you know, shifting gears a little bit. I'd love to talk a little bit about the role APIs play for FireHydrant today, maybe just to kick it off, do you guys have a public API and is it important to your business model at all?
[00:02:16] Robert Ross: Oh yeah. So actually when the company first started, I had just been, this was like 2017, even 2016, my job at my last job was to build the core services of our application. And the way I approached it is I was using gRPC, we had gRPC gateways in front of everything. And what we started doing is we started moving the backend Rails application
[00:02:46] to a lot of that logic to an API and then making the front end React at that point in time basically making API calls to that internal API. And I saw the benefits of it, but I also saw how painful it was to move to that pattern given I was doing it. So when FireHydrant started, I actually said from day one, that's going to be the architecture. There's not going to be Rails controllers that are, you know, whatever your MVC framework of choice is, there's not going to be a backend rendering the dynamic content that then is sent through NGINX.
[00:03:21] It's always going to be a JavaScript front end making API requests to a backend. So I started that very early on, which might've felt like over-engineering at the time, but now I think we're well over 350 public API endpoints in FireHydrant, all of our front end, it uses those API endpoints.
[00:03:41] Which is nice because now our front end team can actually make adjustments without a backend deploy to the application. It's had a lot of downstream effect benefits since then.
[00:03:50] Sagar Batchu: You know, I have to ask an age-old question, why did you pick gRPC for that transition?
[00:03:56] Robert Ross: You're shiny. I mean, I joke, but that was actually the case back then. And we had some really forward-thinking engineers at the last job I had and we were using Go and we knew that we wanted like a contract-based API system for internal and external and gRPC, you know, had just come out, we started playing around with it, we built tools for it and it ended up kind of being the standard.
[00:04:25] So, you know, it's funny, that was what we were doing at the last job, but FireHydrant doesn't use that pattern. And we kind of have a raw JSON REST API backend built into our Rails application. If I were to do it again, though, I would a hundred percent start with protocol buffer definitions of that API and go about it that way.
[00:04:44] Sagar Batchu: Is that what you would suggest to people building a backend from scratch today? Like, is your advice a blanket Go with gRPC or is it something more nuanced?
[00:04:54] Robert Ross: Yeah. I don't know if it's like Go and gRPC. I think when I say gRPC, really what I'm talking about is the protocol buffer definition and the IDL of that - for the folks that don't know, interface definition language - of protocol buffers and the advantage of it is really it allows you to explain your API without implementing your API very quickly.
[00:05:19] And what I also like about it is you can have a debate with your team. You can have a pull request of just your protocol buffer definition and your team can see that and go, that's a good interface design for this API. And the other advantage of it now is,
[00:05:36] and this has existed for a while, but there's new tools out there that make this easier for generating stubs and the code generation surrounding the protocol buffer tool chain has improved dramatically in the last 10 years.
[00:05:48] And now I kind of look at it and go, there's not really a reason not to implement it this way. Because I mean, you get to explain your entire API and then you also for free get an OpenAPI 3.1 file on the other side of that. And then you can use a tool like Speakeasy to then generate API documentation and stubs for many other languages in a Terraform provider. I see very little pitfalls of doing it that way.
[00:06:17] Sagar Batchu: Yeah, that makes sense. I mean, I resonate with that so much. Like the, like you said, a big reason why we exist is we want to bring a lot of that native code generation experience that gRPC and protocol buffers has to the world of REST. And if I flip back a couple pages in my book, I think there would have been so many situations where, you know, that idea of
[00:06:40] a team being able to do true design-first development would have been enabled by protocol buffers. Like, I think I tried this at a previous company with a natively to an OpenAPI spec and just the verboseness and the kind of lack of focus on design made it actually a really hard you know, spec to work on for design-first experience.
[00:06:59] So that, you know, a lot of people, when I asked them, this will kind of focus on the tech and the code gen and all of those pieces. But I think you brought up a really important point around the collaboration experience that I think, you know, especially startup engineers, we forget because as a company gets bigger, that's kind of probably the limiting factor for development.
[00:07:19] Robert Ross: Yeah. Yeah. We use protocol buffers now in our systems that talk to each other. So we have a Rails application and we have a Go application. We have a planet-sized Rails monolith with like many moons of Go microservices surrounding it. And the interfaces between those applications is protocol buffers and gRPC and we like the code generation of it.
[00:07:42] Like the protocol buffer definition only gets you so far, though. It's going to give you a pretty myopic client, I should say. So like, there's other things that you have to consider such as retry policies, error handling. You have to consider like, if you want to go further with it, like I said earlier, like a Terraform provider from these things is possible with your folks' tool, which I think is amazing.
[00:08:05] So, you know, there's trade-offs, you just got to figure out which ones you're willing to take on.
[00:08:10] Sagar Batchu: Yeah, that makes a lot of sense. You mentioned, you know, over 350 public endpoints for FireHydrant today. Like you must see a lot of patterns of consumption by enterprise with your API. What's shifting in the last few years?
[00:08:23] Robert Ross: Oh, what's shifting. You know, I think more and more people are willing to pull data from your API. There used to be kind of a requirement many years ago of like, you need to push data to me, into my data lake. And the problem with that is, sure, we can push data all day long, but it may not be in the format you want.
[00:08:46] It may not be, you know, enough, it may not be, like, frequent enough. Like, you might have stale data if we're doing, like, a nightly push to our customers' data lakes. Like, that's not fast enough in some cases. So, a lot of our customers are scraping our API every five minutes in some cases to get their incident data, you know, we have our most popular endpoint is our get incidents endpoint.
[00:09:15] And what our customers tell us is that they have data analyst teams. They have data analytics engineers that their job is to write scripts that pull the data into their internal data lake from all of their vendors. And I think that's, I don't know if it's a new trend or the new normal, I'm not quite sure, but it's definitely something I've noticed is that enterprises are spending more time building the teams to get the data in the exact format, the exact time that they want it, as opposed to having the requirement of push the data to me.
[00:09:47] Sagar Batchu: That's super interesting. So it's like they're taking data from API, populating some kind of data warehouse BI system for their own work. Have you guys at all thought about like exposing the data directly in a more native way, like to a data stream or to just you know, tables that they could materialize themselves?
[00:10:05] I'm curious if that's changed your viewpoint on actually just having a traditional REST API though.
[00:10:11] Robert Ross: Yeah, a few things that I've considered is webhook, you know, fire hose out of FireHydrant. Actually, that would be a really perfect name for that, wouldn't it?
[00:10:23] Sagar Batchu: Really.
[00:10:25] Robert Ross: Hook up the Firehose to FireHydrant. We do webhook delivery today. Internally, our architecture is all PubSub events. So anytime you create an event in FireHydrant
[00:10:35] such as like a note, a chat event, updating an incident, that goes to a published subscribe model. And then if you want, you can send all of that data via a webhook. We do that today. And I think that you can go further with it. I think that you can consume from a stream especially if you have like an internal audit logging system, can you publish events and things like that.
[00:10:58] And you kind of let your customers materialize the data the way that they want it. I'm seeing that pattern more and more. There's a company called Offset that does this very well with their materialize RBAC system. I think up there, it's actually a pretty good model for where maybe the industry is going to go with materializing and sending webhooks.
[00:11:19] Sagar Batchu: That's really interesting. Yeah, we're seeing some interesting shifts there as well. Like on one side some people are considering opening up their, you know, Kafka data streams directly to customers.
[00:11:29] Robert Ross: That's what I had in mind. Yeah.
[00:11:31] Sagar Batchu: Yeah, yeah. I love the name firehose for FireHydrant by the way, I feel like now you have to launch a product.
[00:11:36] Robert Ross: I, yeah, I actually, yeah, my engineers are going to love that spur of a moment idea on a podcast that they haven't built.
[00:11:42] Sagar Batchu: Right, right. Well, drive-by development. Yeah, that's really cool. You know, I mean, to make shifts like that, something we think about a lot is like what metrics should companies actually measure for API success? Because, you know, if you're going to go do any kind of big technical change or, you know, anything that actually impacts the experience, you have to know what you're measuring.
[00:12:02] And I think in the public API world is actually quite a differing kind of set of viewpoints on what you should look at. I'm curious if you've ever thought about like what KPIs are important for the FireHydrant API.
[00:12:13] Robert Ross: Yeah. I mean, latency is, time to first byte is one of the most important ones for us. We have a couple of pretty slow endpoints. I mean, we just represent so much data from that endpoint that we made some mistakes that we can't go back on now, which is an issue with API design that you have to be considerate of.
[00:12:34] But I digress. I mean, definitely time to first byte. How long is it going to take when you get the request from me, how long do I have to expect until I start getting that request back? And I think that's one of the most important measurements, especially if you're using your own API on your front end.
[00:12:51] You get a little bit more leniency if this API is just for you know, programmatic client calls. Right? Should it take several seconds? No. That's probably bad, more worse for you than it is for them, if you're taking a few seconds. Couple of other KPIs. I, you know, I think obviously like, do people have to retry requests to get data?
[00:13:16] That's a KPI that we measure because you can get into a situation where you have a thundering herd issue and what are the leading indicators of, well, if you're slow and the client call times out and it doesn't automatically retry and you have thousands of customers that hit that just because your API tips over three seconds, all of a sudden your API is going to get smoked even harder.
[00:13:38] So you kind of have to set your KPIs a little bit lower down the chain.
[00:13:43] Sagar Batchu: Hmm.
[00:13:44] Robert Ross: So if you have an SDK, for example, that is Ruby or Go SDK for your API, your read timeouts should be less than the API's timeouts. That way you kind of have a buffer in between, you know, the client code. So that's one thing I definitely think people should be measuring and being conscious of.
[00:14:02] I can't honestly think of another one. I mean, like, usability of the API is really hard to programmatically measure, other than going to the people using your API and asking, how's it feel? I think that we have just some very old decisions in our API. We're still on V1. Like, we have not done an API update, like, structurally
[00:14:24] in six years of starting. It really is the same API endpoint you got six years ago. It's the same one. Maybe it has more data in it, but it's the same exact structure. And maybe that's a KPI. How often do you have to change your structure?
[00:14:40] Sagar Batchu: Yeah, these are definitely interesting ones. I mean that like, so, you know, time to first byte absolutely resonates with me. We call it internally time to first 200 or time to 200 for
[00:14:49] Robert Ross: I like that.
[00:14:50] Sagar Batchu: Yeah. I've always thought maybe we should, if we ever get into the request tracking game, we should
[00:14:55] throw up a metric in our dashboard that tells the user what the, you know, current time to 200 on the SDKs is.
[00:15:01] Robert Ross: Like that.
[00:15:02] Sagar Batchu: Yeah, I think another thing that's interesting is the API design problem you brought up. Like that is a really tough KPI, I think, to measure in any meaningful way. We've, you know, a lot of customers come to us with OpenAPI specs and a variety of quality and state of kind of maturity,
[00:15:18] and today, you know, we consume all of them. We give them, you know, linting warnings and errors, but I've thought a lot about whether we should give them some kind of score, right? Something to tell them that you know, the design of API today is flexible or it's not going to like block future decisions.
[00:15:34] It's a really hard thing to do because I think, you know, industry to industry, there's also, you know, different requirements like the heavy requirement around pulling data from your API, like that might not exist for another API. So I think where we've gone to is we definitely tell our customers and people who work with, like, do you have someone just thinking about the spec design as it changes over time?
[00:15:57] Robert Ross: Yeah. I mean, the ergonomics of an API are almost
[00:16:01] more important. I mean, you know, when you go to the furniture store and you're trying out sofas, which one do you buy first? You buy the one that's the most comfortable. API is the same thing. If you have a really uncomfortable API and that's important, like people aren't going to buy that product if that's a big part of their purchasing decision.
[00:16:20] And you know, like, for example, if you're comparing Jira to, I mean, I think Jira has the one of the worst APIs in like human history. And I can say that with confidence because we have an extremely intricate Jira integration and the amount of like backflips that we've had to do to make very simple things work with the Jira API is astounding. We also have a Linear API integration when, you know, basically it's the same thing. Create an incident ticket in Jira, create an incident ticket in Linear. The Linear code in the Linear API made it so that integration we built it in a matter of days. Like it was drop-dead simple because the API was so good.
[00:17:12] And I think we're going to enter into a pretty interesting reality where the companies that have very, very simple APIs, maybe not simple, but easy to integrate. Like the ergonomics of it are really good. Those companies are going to see longer-term success because more people are going to want to integrate with that tool. If I'm asked right now, if I had a gun to my head that said, like, pick a tool to integrate with, I'm going to pick like Linear right now. Well, after FireHydrant, but I'm going to pick Linear because they've just done like an astounding job with their documentation, with their API design, they use GraphQL. That was their choice, not mine, but it didn't matter because the client SDKs were so good.
[00:17:53] Sagar Batchu: That's really interesting. Yeah. Especially in these competitive markets where, you know, over time the product differentiation kind of deteriorates, like these kinds of things really start to stand out for sure. Switching gears a little bit, you know, we're just in the world of dev tools, developer platforms, a lot of paradigm shifts happening with LLMs and AI.
[00:18:14] I'm curious how, like what your point of view on this is when it comes to the APIs that your customers use or the APIs that you guys design. Has it like changed customer demand in any way or people asking to integrate in new ways?
[00:18:27] Robert Ross: I would say our approach to APIs has changed, which I think will have downstream effects. I mean, now when I've been designing new APIs, I actually will kind of vet the design of it. And yes, this is me admitting I'm still writing code six years in as a CEO of FireHydrant. I've talked publicly about that though, but I kind of vet the ideas of a new API against LLMs now, honestly, they have pretty good opinions
[00:18:55] on API design, especially if you prompt it with, I need my API to be it's in, it's a gRPC service definition. And you know, there should always be a verb before the request message or something like that. Like you can kind of create your linting of an API, your design principles, and you can ask LLMs, like, what do you think about this?
[00:19:22] You can kind of have a conversation with it. And eventually, honestly, I've actually had some pretty good, at least in my opinion, internal API decisions come out of that tooling.
[00:19:32] I don't think that LLMs are going to fundamentally change how we interact with APIs. At least not in the short term.
[00:19:39] I think that they'll help us build better clients and I think they'll help us build better APIs, but the moment we start using them, I mean, that's just going to be code calling code. I just don't think that LLMs are going to get too integrated at that point.
[00:19:53] Sagar Batchu: Yeah, that's really interesting. I, then the design part you know, we probably see like a couple, I don't know, a dozen hundred specs a week and a lot of them, I, you know, we have our own tools, but I've also been kind of just copy pasting them into, you know, those that are public specs into an LLM and just
[00:20:12] talking about it. Right. Like kind of semantically trying to work through what the spec is, what the state of the spec is. Yeah, it's super, super interesting just in terms of like being a guide right for you. Just the way we're seeing Copilots become a guide in IDE.
[00:20:27] Robert Ross: I can see like one opportunity where like a good API design with LLMs would play out very nicely. So Agentic systems is the next probably big topic in the realm of AI, I think. If not, you know, it's going to be a big part of the pie chart of discussion. And if you think about the way that these Agentic systems work, you have to give them data to then keep having a conversation until it ends up in like some result, either through tool usage or, you know, some pattern.
[00:21:01] And a big part of the tool usage that these LLMs are performing. We use Anthropic and Claude, which has really, really good tool usage APIs. We basically give it like raw JSON responses in some cases. And if that JSON response is bad, then it's going to have a bad result. And it's kind of making me rethink, even just the key names of these JSON responses.
[00:21:28] So for example, for an incident, the resource, the URL is slash incidents, right? But the payload is name. And if I give that to an LLM, all it gets is name. So are we going to have to start prefixing things with incident_name so that these LLMs actually have a little bit more context, a little bit more, you know, data to go off of, are we going to have to go back to the world of where everything is namespace under a key incident with an object of name description, right?
[00:22:02] Like our API doesn't do that. If we were to change that, would that make our LLM responses for either us or our customers be better? I don't know. I think we're going to have to see, but those are the types of questions I think API design is going to start driving Agentic systems and, you know, catch trying to buy vice versa.
[00:22:19] Sagar Batchu: Yeah, totally. I mean, like on that bit about just the names, right? Like API naming is already hard enough and I think something without giving too much away on like an upcoming release, we're planning to do in this space, when you think about tools, like if these tools are going to be used mostly by agents, does the verboseness of the name matter?
[00:22:38] If you can stick in a bunch more context by prefixing or suffixing a name, like maybe you should do it. And as long as it's not a tool that's going to be used by a human, it doesn't really matter, right? Like kind of the conciseness and all of the, I think you know, really strong points of view people have on ergonomics today on design is kind of human-led, right?
[00:22:57] It's to help a human looking at the docs of the SDK methods in for what a method is about. But that doesn't necessarily, it's not the same for an LLM, right? Maybe you do want to stick as much information in whatever, you know, token window you have that you can give it. So, yeah, no, really, really interesting.
[00:23:14] I think it also means from an API design perspective, like I wonder if we get to the place where there are APIs for human integrators and APIs for agent integrators in terms of just ergonomics. Like there's probably going to be a split the same way. I think they're going to be like code bases that are fully human administered versus code bases completely driven through bots and copilots because the quality and the verboseness of the code is - those metrics change.
[00:23:40] Like it doesn't, you don't need to do the same thing in both of those cases.
[00:23:44] Robert Ross: Yeah, no, I agree with that.
[00:23:47] Sagar Batchu: Yeah. You know, just to round this out I'm, you know, with all this stuff going on, like, what does, you know, does great developer experience still mean the same thing to you today? Like, has that, has your point of view changed on that over the last few years?
[00:24:02] Robert Ross: When it comes to developer experience, I think that my guiding principle is like,
[00:24:06] is it easy? Like, how far can I get before I have to read documentation? And I think that includes like great error messages, like, if I can do something on an airplane with really crappy wifi and like be productive, that's great developer experience.
[00:24:22] As someone that flies a lot, you know, that's when I build things honestly, is like when I get on a plane and that's when you really feel bad developer experience because things are slower or you don't have internet, you get an error and you're like, this is the most cryptic error I've ever seen.
[00:24:39] You go to Google it and Google takes five seconds to load, like that's when it reveals itself. So, and I think it ties into, you know, API design in the same way. Like, do I have to look up your API? If I do like, then that's frustrating. If I'm using a Go SDK to use your API and that SDK has a method that doesn't explain the parameters, like I'm going to be mad at you. That's really bad developer experience. So, that's kind of my litmus test, how much can I build on a plane? And if I, you know, it's gotten to the point now where I just know that I have to run Docker build before I get on a plane. Like that's the habit I've had to build lately. Cause the moment I get on a plane and I start my, I go into dev mode and it boots and it's like downloading layers.
[00:25:30] I'm like, I'm not building anything on this plane today.
[00:25:33] Sagar Batchu: That's a pretty good soundbite. Airplane mode is the true test of developer experience.
[00:25:38] Robert Ross: I think so, yeah. I've never said it that way, but yeah, I think just coined my term for it.
[00:25:44] Sagar Batchu: Alright, I'm gonna take that one away with me. You know, on that note, thank you so much for joining us here today. You know, this is our second ever episode of this podcast. You know, thoroughly enjoyed talking to you here. Yeah, I'm looking forward to chatting more in the future.
[00:25:57] If our listeners want to learn a little bit more about FireHydrant or you and the stuff you're working on, where can they go?
[00:26:01] Robert Ross: You can go to FireHydrant.com. That's our domain. I'm Robert Ross on LinkedIn. If you have any questions or @Bobbytables on X.com, formerly Twitter, and Bobby tables on GitHub. If you're curious, if I'm full of it.
[00:26:16] Sagar Batchu: I think there's a backstory to that name, but let's save that one for next time you're on.
[00:26:21] Robert Ross: Yeah. Thanks.
[00:26:22] Sagar Batchu: Alright. Thank you so much, Robert.
