Glen (Plabayo): Welcome to another week of Netstable FM. Today will be another protocol shorts and with me is Brecht. He also helps to maintain Rama. had him as guest already in episode 23 where we talked around HTTP and using it as an application network bus. Today will be a bit different. So welcome Brecht.
Elizabeth (Plabayo): Netstack.fm is brought to you by Rama, an open source framework for moving and transforming network packets. Rama is built and maintained by Plabayo a company focused on secure, open, and resilient infrastructure with rust, protocols, and purpose.
Elizabeth (Plabayo BV): This is netstack.fm, your weekly podcast about networking, Rust, and everything in between. You are listening to episode 38, our protocol short series, recorded on 20th May, 2026.
Brecht: Hey, good to be back.
Elizabeth (Plabayo): The theme music of this podcast was composed by DJ Mailbox.
Glen (Plabayo): Yeah, we were talking recently and it got to me that you are building all kinds of very interesting stuff with Rama. that, and also the fact that you have to maintain and evolve Rama, I came to realize ⁓ that are a couple of interesting things that we could discuss together and enlighten our listeners. So maybe before we dive into technical details.
Elizabeth (Plabayo): For more conversations like this, subscribe so you don't miss what's coming next. And if you know someone who could benefit from this episode, share it with them. They might appreciate
Elizabeth (Plabayo BV): In this episode, we talk with Brecht Stamper about how Rama helps developers build and test complex network systems. More Let's begin.
Glen (Plabayo): Can you talk a bit about the things you are building with Rama as well as within Rama?
Elizabeth (Plabayo): have experience in protocols, networking, or infrastructure, and want to share your work, your ideas, or experience, we would love to hear from you. Reach out at hello@netstack.fm.
Brecht: Yes I can. The I'm building with Rama are mostly high throughput, network moving ⁓ It's really because it's not a single use case, but the biggest use case is ⁓ a proxy gateway like you would expect it to be. But there's a lot of side projects I'm also doing with Rama, ⁓ either related my work or for work. And open source.
Elizabeth (Plabayo): Thank you for being here. See you next time for the next handshake.
Brecht: On the Rama aspect, I'm mostly trying to improve the developer experience and make everything less magical and explicit, which sounds boring, it helps in maintaining stuff and making easier to use for complex applications because magic was often times causing very subtle bugs which were hard to debug and slowed everything down. Glen is the same opinion. a little bit more expressive and avoid magic but everything clear and really scalable to however complex the application wants to be.
Glen (Plabayo): Yeah, totally agreed. that was indeed always the goal, but sometimes, of course, as you're trying to find abstractions ⁓ ways people can compose network services, with the framework, you at times have accidental magic and I'm very glad that you're helping to clean it up. Now, if we go back in time and think how you were introduced to Rama. remember that the concept of everything is the service was a bit overwhelming, I can say, or maybe a bit too generic. Can you maybe talk a bit about how you were looking at it and then how you slowly started to understand it is and how you understand it right now?
Brecht: Yes I can. So when I was introduced to Rama, I was everything is a service. Even most of file names at that point were called "service" in internal codebase which didn't help a lot. So to me, service was like trying to do everything with only one thing. And in the past, everything always clear. Like this is core storage, this is network layer, this is that. Suddenly everything is a service, which sounded very abstract and wrong. Service had a different meaning for me back then, but I've come to understand that service is not really an abstract idea, it's more of like a function signature that you lock in. But the in Rama is nothing more than this is the input I'm expecting and this is the output I'm gonna get. ⁓ Which allows you to stack it with other services and other layers which is magical at certain ⁓ as long as the input is what is expected and the output also matches you can just stack everything and anything together even things you should not stack which is incredibly useful for testing things but when you think of a service in Rama I really think of a contract between input and output signatures nothing more, nothing less, which is a very simple idea, but it allows you to stack until like crazy complex applications and it stays simple, magically simple. So yeah, that's kind of how it evolved for me.
Glen (Plabayo): do you remember a bit, when was the moment that you came to realize, okay, now do I understand it or did it just come very slowly and you have no idea?
Brecht: At one point I understood the idea but kept trying to fight it because I always thought "For this use case let's make a new trait for it that describes my use case better." Like doing a request you could call it do_my_special_request which is a little nicer to expose instead of just serve. But that's opened so many issues over time, like I want to unit test this, okay I have to implement this Well for service, everything and anything implements it, so as long as the contract agrees, it just plugs in the entire Rama codebase and everything around it. Every single time you do something custom, that doesn't happen automatically. need to do a lot of work yourself and in the beginning it's hard to con- vince people - myself - why that is so powerful. So for everyone exploring I encourage you to just don't commit to the Service idea, try to invent your own you're gonna very quickly realize why it's so nice and how it saves so much time.
Glen (Plabayo): Yeah. of course, reason why I initially liked it many years ago is because, okay, I can easily stack things because I compare it to functions where a function calls another function and similar with services, you can compose them. Meaning like first I do something here, then I pass the request on and eventually something will give me back a response or maybe something earlier gave a response because there was an error or whatever. But then another benefit I came to realize quite quickly was the fact that let's say I a web server and I have now my entire composition so like my middleware and my router my different endpoints so it's like fully the system is ready there normally what I would have done in the past is okay that point I need to spawn this ⁓ or let's say bind it to a socket and I need to test it with like a real HTTP client, which kind of means I'm also testing everything, even though really just want to test my business logic. And with the service concept, I could just send it, give it a request and it will give me back the response kind of like how I'm just using a client. I don't think that's immediately clear because it kind of ties in what you were saying that everything is a service and so it's hard to look past the abstraction, but beautiful the fact that a server and a client looks the same. But I wonder there ⁓ other examples like you can think of that like didn't immediately work clear to you, but The fact that everything is a service and the fact that how it composed so nicely and the fact that it's just a simple idea, like that it unlocked things that you couldn't otherwise have done.
Brecht: Maybe something I could have done in the past but that was always annoying in pretty much every other stack when you want to test stuff you have to mock it but mocking stuff always felt like Really hacky where you patch something you return something ugly. ⁓ Remember a few months ago gRPC support was added to Rama which was a huge PR and gRPC is in itself quite complex. It does a lot but you have a gRPC client and then you implement the service traits ⁓ the server side for the actual logic and it was crazy simple to just unit these things, the client and the server without needing HTTP, without needing TCP, without all of the stuff. If you do this in other languages, you need to spawn a server, you need to do all of that stuff. Here, you could just literally have the client ,and instead of giving it an HTTP client do the network stuff, you could just literally pass in the gRPC server, you just made, implementing the business logic, skipping of the network stuff and you could instantly do all of that logic. Just put the server inside the client as the network stack and works, which was again, like one of those moments I realized, wow, this is so clean, so simple. And it allows me to test these things I want to test here. Of course you ⁓ to test the entire system. It allows to focus on each module, really clear, really simple. Another use case is Orama we have a mock connector, which is basically a fake connector that you a stream. Which allows you to any connection that will need TCP otherwise, if you implement the test with TCP before, they're always tricky like either spawn them on port 0. need to wait until you get a port then combine everything which involves lot of Overhead is quite slow if you need it for a lot of tests it adds OS Complexity if you don't care about that like if you're implementing HTTP stuff. You don't care TCP you care about the TCP layer just get yourself a client a server and connect them with a fake mock connector and focus on what that layer needs to do, which allows you really test every layer on its own very simple without bringing in everything and ⁓ all things just for HTTP for example. Those are some of the examples they sound but to in the past doing these things how hacky it was or how much code you needed just works with Rama as long as input output match Everything just works, which is really nice.
Glen (Plabayo): Yeah, fully agreed. talking about gRPC, you've using it quite a lot in the recent months. I remember one point you had issues I believe it was timeouts, which was the end symptom. ⁓ And your solution was the fact that you needed a couple of low level configs on the socket, such as TCP, no delay. Can you talk a bit that? Because like, okay. First of all, what were you noticing? And then secondly how did you ⁓ that? Because could be quite interesting to learn about that. And then maybe finish off by how could a framework like Rama help guide the user better or even just help within its to prevent such issues. Because I don't think most people that will use gRPC will actually know that much about TCP to even think about those things.
Brecht: Okay, I noticed the issue because, on all of the gRPC stuff, I have really strict timeouts Which I advise everyone to if you expect it to be fast make sure it is fast ⁓ You can either crash slow or, at the very least, log it if it's slow so you know it's happening And for everything I do I set open telemetry which gives me metrics, traces, loss to debug these things And quite consistently, I was seeing above milliseconds for gRPC, got me investigating Like this is a really documented issue- it's all over the place - but ⁓ TCP, design, doesn't send very small packets because ⁓ it doesn't that ⁓ which means buffers them and there's some rules about when it does send them, but gRPC happens to be almost always small messages, which means you send a message, almost always trigger a timeout unless you're pushing more over the same connection. So you need to set TCP_NODELAY. Most gRPC libraries do this for you, always behind the scenes. In Rama when you use gRPC you provide your own client, so it's not set everywhere because ⁓ it's useful not set value for normal web traffic but for gRPC you do want it. that also brings an important thing with Rama. All the building blocks are there and we also provide abstractions but it's still important that you know about all of the stuff that's there if you want to do low level stuff. So for gRPC I imagine the thing we need to do and it's still on my to do is actually document this but we also have like the easy web client we could also provide a gRPC web client which does these tweaks by design or by default so other people don't need to find this the hard way like I did. But yeah it's really documented it's all over the internet so when I was searching: Why is my gRPC slow? Google gave me 50 pages with all the same answer: ⁓ Is set? No, it wasn't that opens the door but Rama is really powerful you really need to tweak all of the low level stuff as well. TCP stuff, Linux stuff, a lot there. But yeah, when you make a new stuff, definitely set up open telemetry or at least logging so you see what's happening under the hood.
Glen (Plabayo): Yeah, and of course both are integrated within the Rama And I agree, like you can do all the low level stuff, you can control whatever layer of the network you want. That said, as you also ended yourself, we do provide abstractions at various layers. And for most users, I don't think they will need to tweak this. And so I think next to documenting for sure. We need to also provide an abstraction there similar to how for most let's say users need to do simple HTTP, API calls. There is the easy web client, like you mentioned. I think it is right to also provide something like that, like a pre-built transport client either there or somehow provide that support within the existing client ⁓ to make it easier because yeah we only want them to have to worry about the network layer if if they really need to of course otherwise doesn't any sense That said, this is only like a client issue or that's also like on the server. And then next to that you mentioned that most gRPC implementations ship the support built-in like they have like just a single API and you get your client or whatever So then how come they still have the issue? as well or least some of them.
Brecht: they don't really have the issue anymore, but at one point the issue was there. Someone opened an issue for it and it got fixed. The only places where it still happens is more lower level libraries where they're explicitly documented. ⁓ Hey, don't set this for you for these or these reasons, but under almost all circumstances, you do want to set this. Some crazy examples of where you don't want to send this if you're sending this a latency networks that's pretty far and you don't really care about instant ⁓ responses but you do want to send them in batches ⁓ is almost never the use case but libraries do give you the option in that case
Glen (Plabayo): Okay perfect and then we started this episode around the service trade and it went through a very big evolution. I mean we didn't invent the concept to begin with it came I believe Java library if I remember correctly and then it was within Towers in Rust and then we kind of took it from there first tower-sync ⁓ library we made and then finally at Rama we have the current version but even then we had in the beginning not just an input but also a context which was meant for like all kind of including states ⁓ Eventually we to the conclusion it was the wrong abstraction, we removed that. Then we still had the fact, okay, we do need dynamic state, we have extensions. yeah, to begin. What are extensions, Brecht ? and And where do they live, why do we even want them?
Brecht: so let's maybe start at where extensions started. This is very similar if you've worked with other pieces of Rust codebase. Extensions are mostly you need dynamic content. Which is often the case a of different things if you cannot predict that compile time what everyone will need So the way it basically works in most cases is you have a hash map as the keys, you use TypeId of a generic object. When you do TypeId, Rust gives you a unique ID which you can then use to like add runtime, query this in a hash map. Which is also how we did it in the beginning. But the problem we noticed over time with a hash map is that people insert items, they remove items and it becomes really hard to see what's happening under the hood. How it got there, ⁓ there were leaks. different interactions of all that stuff. So trying to fix this issue, which is an issue that happens all over the codebase and that's also really hard to fix. We designed a new system where basically extensions would become a vector would be append-only. So you could kind of see it as a history of extensions being inserted and you can never remove an extension. You could use logic for example put it to this item is used, you should not use it anymore. But it's always there leaving a paper trail of what happened. So it's also a really nice debug tool to see what happened, where did it happen, in what order. At that point we had a really simple vector which needed mutable reference. That was kind working. we kind of moved it all over the place so when we went from a request to a connection we just copied everything over same in other places ⁓ was working for simple use cases but for the more complex stacks it leaked lot of information by leaking I ⁓ we transferred extensions the request to a connection then when you used a connection pool that connection still contained extensions from the original request under cases happened to introduce bugs. ⁓ of that and it really only meant as an intermediate step. So the next we have a structure that is and read-only for the other parts which means we should be able to optimize it. instead of using a normal vector, which got us the append-only vector. It's basically a structure that can grow and where you can insert items only ⁓ to self, no mutable reference needed. Because could use the properties of like, we can only insert items at the back and never modify items that are already in there. By changing reference instead of mutable reference, we could edit extensions from multiple places at once, which sounds like an anti-pattern in Rust, but is actually needed if you happen to have logic scattered in multiple places, which happens a lot for extensions, for connection extensions. user want see what the extensions of connections are. For example, is it broken? Is it healthy? ⁓ But the connection still need to be able to edit these. So by relying on that append-only structure with only reference, ⁓ we could do and ⁓ it's already in and it's working great so far. For the people that are already using it, if there's feedback, please open issues or message us on Discord. And happy to look at those. ⁓ But that got us to append-only VEC. latest transition was really splitting up extensions between and connections. So really seen as distinct extensions now. when a request goes through connector stack, we don't copy. the request extensions to the connector, we only copy extensions that should become part of the connection. If you say I want to connect only to this IP, this IP now becomes part of that connection because it is fundamentally how the connection is used. Other examples are which TLS version is being used. Once a connection is established, that should be part of the connection. If you reuse that connection again, the same information applies. If you put a retry policy on the request, we don't copy it over to the connection because a follow-up request might not want to use that. So they're really distinct. also fork at logical places, meaning when you have a connection, extensions are passed through. If you go from TCP to TLS, we reuse the same extension because logically it's still the same connection. However, when you do multiplexing, which HTTP2 when it's using stream, we fork extensions they have a new storage. They still keep the original extensions as parent so all the stuff that's in there still applies to a stream but a stream has also its own storage to stream specific storage things. ⁓ Examples there it is possible that a stream is broken but the connection is not broken so in that case we insert connection is broken on the stream but not on the connection which allows us to do a lot of complex things. and this is also made to work around upgrades because upgrades are complex items. If you do an upgrade on HTTP 1 you kind of replace the HTTP with the upgrade you just did. Nice, this is easy. But if you upgrade over HTTP 2 you're basically nesting a completely new connection inside a stream which could be doing TLS again, which could be doing a lot of other things again. And things inside that connection don't really affect the outer connection. so they should be isolated but they should still have access to the underlying connection because they do still run over that connection. system allows that. It works "magically" is kind of the word which is something we're removing but here it is needed to like just be transparent about the network stack but it also gives you all the building blocks to go as crazy as you want with this.
Glen (Plabayo): Yeah, and part of complexity or the difficult program space is of course because Rama is a framework it gives you all these building blocks but you can compose them how you want the bounds of ⁓ contracts as specified network specifications of course you will get other issues But as long as you respect the different specifications or how the world operates, you can make up whatever stack you want and Rama makes it pretty easy. In contrast to that, you would need to otherwise even either write all the network and protocol code yourself, which I don't think anybody does. Instead what most people would do is they either go for like a black-box solution where they can just use a config file or something, if that at all, or they would glue together all kind of random libraries, but of course plenty of mistakes and imperfections get made while trying to glue these things. Instead, Rama has all these building blocks, they work nicely together, you can compose them, because we cannot predict how the user will compose them, we of course don't want put them into a surprise scenario where they compose it somehow. Things happen with these extensions because how they passed around and now they get into these situations which, like you mentioned, we had. Because you could remove extensions and... maybe then it no longer contain information it should have contained or maybe shouldn't have and ⁓ it got pretty messy I'm pretty happy that that's where are now as I believe right now you can stack without any fear and I certainly didn't see any surprise anymore since since we arrived at this new design ⁓ I'm not sure how you see evolving from here Brecht
Brecht: I think the extension stuff is mostly finished, core logic, but do have a lot of other things that are using extensions, mainly around TLS special builders. Around these boring builders, were designed before this new extension logic was there, ⁓ so they of solved this problem on their own in a thing that worked at that point of time, but it's quite a complex struct, that does quite a lot of weird things first time you see it. the way I see it evolving is now rely on the new extension logic, really rely on the features it offers and simplify all the users of the extension logic by relying on what extensions offer. But yeah, that's mainly the extensions set
Glen (Plabayo): Okay, very cool. And then to run off this episode, you've been now not only like a of Rama, you're already that for quite a while. You've been using Rama now already at several companies as well as, for stuff that you do in free time, sometimes related to work sometimes not. Can you think of any story like, okay, I'm using Rama now here and it really helped me to unlock something that otherwise would be pretty difficult if not impossible to do.
Brecht: ⁓ is a big word because yeah nothing is impossible. It's just it might be impossible in the required time frame But that's where Rama shines so much like I've been surprised so many times where I implemented like Complex proxy stack or a complex ⁓ network stack in so little time and just worked like very little time and When I do all of these things I always add tests as well. Adding these tests is so simple with Rama as well. All the core stuff it just works which is nice. ⁓ I do this without Rama. ⁓ Let's I stay in the Rust ecosystem. I would have used Tower, Hyper and Tonic probably. But when I combine all pieces ⁓ there's so much glue I need to write myself. Like huge amount of glue. And Rama solves this by being a monorepo basically that just has one standard way of doing everything. It's applied very consistently everywhere and all pieces just work together. Like if you use gRPC, okay you're kind of in HTTP world, you can use all the HTTP stuff. The same kind of applies in tower world tower ecosystem is kind of stuck on an older Rust version which makes this way more painful. It's also way less consistent and the core logic is there but all of the stuff that combines it is not and it's often there that the most subtle bugs exist and where lose a huge amount of time so yeah is just nice it combines everything
Glen (Plabayo): Okay, very cool. And where do you see future going? how do you see ⁓ Rama in the future and what do you expect from it both as ⁓ a maintainer well as a user of ⁓
Brecht: the short term I mostly see Rama evolving on a feature set. Like there's some really nice that we still need to add to Rama. Perfect example of that is HTTP 3 and Quick protocol. ⁓ Because unlocks so many new use cases again. Like ⁓ once have that you can go really crazy. Like you can already go really crazy but that's like fundamental piece. In the world that's still missing. So short term I really see Rama adding more features or more developer tools to make things easier, make things better work with, to debug. ⁓ Dial 9 was added which again will help a huge amount to debug these crazy OS issues or Tokyo issues ⁓ something is stalled that you never expected So short term feature wise, then long term I really see us focusing on speed. performance, throughput just ⁓ out hardware easily. ⁓ nice thing is, if you just update Rama, you will get all of this for free basically, which ⁓ been the case ⁓ a lot of places where I use Rama. Every update, I can remove a lot of special workarounds and stuff I have in code just because right now Rama has solved this automatically for me. ⁓ and it just works probably better than the stuff I have. That doesn't mean I don't need logic anymore, but I can rely way more on the core logic, which also means if I needed an other project, I can just do that without having to reinvent everything.
Glen (Plabayo): Yeah, exactly. Because anyway, might be able to do it, but it distracts you from the business logic. A lot of subtle things can go wrong. And then everybody's basically solving the same books if they find them at all. Because there are so many ways to run something. ⁓ I getting surprised by it because ⁓ right now, Rama is getting used to more more different scenarios, different platforms, different use cases. The amount of books that come in is very little, but when they come in, they're very interesting always. And it solves Wong and then, like you said, Wong that solved, you do an update and you just get it to fix for free, which you wouldn't really get. Otherwise, because you also mentioned, I had this TCP new delay issue. Okay. I had to look it up. You see in your Google results, because you're using Google, like I'm getting 50 pages of people getting the same issue. And that's the kind of thing that you would get for thousands and thousands of different issues. If you need to start. building yourself with these low level components, gluing it yourself together, trying to figure out how to compose this, how to work with this. And that was always a goal to solve that giving someone the power to program their own stack because you do want to program it don't want have to resort to some silly YAML file or something like that where you're almost like pseudo programming in config language, would happen sometimes. But at the same time, you don't wanna have to program everything yourself either. So I'm very, very happy that it's working out.
Brecht: I do have example of something I didn't think about but just ⁓ that you fixed kind of recently is the system resolver in Linux is just blocking by design if you want to do it properly you need to use spawn_blocking in Tokyo ⁓ but if you have a proxy for example that can easily run out of blocking threads But Rama, if you use a system resolver, it has a small cache implemented for you to work around this problem. is one of these things you don't think about when developing these types of applications that you need to think about the DNS resolver being slow on your system or not slow expensive to call basically.
Glen (Plabayo): very much agree. And of course, when things do go wrong or things can be improved, we always accept contributions. Everybody developing network services might have opinions or ideas or might notice things and yeah, to highlight again, we will just have to fix it once and we all benefit from it. very much a community project and I'm also very happy that you are part of it, Brecht, and thank you for your view on the project today here in the podcast, as as the explanation about extensions and how you got here and how you came to understand Services.
Brecht: Thanks, was fun.
Glen (Plabayo): bye.
Elizabeth (Plabayo): Netstack.fm is brought to you by Rama, an open source framework for moving and transforming network packets. Rama is built and maintained by Plabayo a company focused on secure, open, and resilient infrastructure with rust, protocols, and purpose.
Elizabeth (Plabayo BV): This is netstack.fm, your weekly podcast about networking, Rust, and everything in between. You are listening to episode 38, our protocol short series, recorded on 20th May, 2026.
Brecht: Hey, good to be back.
Elizabeth (Plabayo): The theme music of this podcast was composed by DJ Mailbox.
Glen (Plabayo): Yeah, we were talking recently and it got to me that you are building all kinds of very interesting stuff with Rama. that, and also the fact that you have to maintain and evolve Rama, I came to realize ⁓ that are a couple of interesting things that we could discuss together and enlighten our listeners. So maybe before we dive into technical details.
Elizabeth (Plabayo): For more conversations like this, subscribe so you don't miss what's coming next. And if you know someone who could benefit from this episode, share it with them. They might appreciate
Elizabeth (Plabayo BV): In this episode, we talk with Brecht Stamper about how Rama helps developers build and test complex network systems. More Let's begin.
Glen (Plabayo): Can you talk a bit about the things you are building with Rama as well as within Rama?
Elizabeth (Plabayo): have experience in protocols, networking, or infrastructure, and want to share your work, your ideas, or experience, we would love to hear from you. Reach out at hello@netstack.fm.
Brecht: Yes I can. The I'm building with Rama are mostly high throughput, network moving ⁓ It's really because it's not a single use case, but the biggest use case is ⁓ a proxy gateway like you would expect it to be. But there's a lot of side projects I'm also doing with Rama, ⁓ either related my work or for work. And open source.
Elizabeth (Plabayo): Thank you for being here. See you next time for the next handshake.
Brecht: On the Rama aspect, I'm mostly trying to improve the developer experience and make everything less magical and explicit, which sounds boring, it helps in maintaining stuff and making easier to use for complex applications because magic was often times causing very subtle bugs which were hard to debug and slowed everything down. Glen is the same opinion. a little bit more expressive and avoid magic but everything clear and really scalable to however complex the application wants to be.
Glen (Plabayo): Yeah, totally agreed. that was indeed always the goal, but sometimes, of course, as you're trying to find abstractions ⁓ ways people can compose network services, with the framework, you at times have accidental magic and I'm very glad that you're helping to clean it up. Now, if we go back in time and think how you were introduced to Rama. remember that the concept of everything is the service was a bit overwhelming, I can say, or maybe a bit too generic. Can you maybe talk a bit about how you were looking at it and then how you slowly started to understand it is and how you understand it right now?
Brecht: Yes I can. So when I was introduced to Rama, I was everything is a service. Even most of file names at that point were called "service" in internal codebase which didn't help a lot. So to me, service was like trying to do everything with only one thing. And in the past, everything always clear. Like this is core storage, this is network layer, this is that. Suddenly everything is a service, which sounded very abstract and wrong. Service had a different meaning for me back then, but I've come to understand that service is not really an abstract idea, it's more of like a function signature that you lock in. But the in Rama is nothing more than this is the input I'm expecting and this is the output I'm gonna get. ⁓ Which allows you to stack it with other services and other layers which is magical at certain ⁓ as long as the input is what is expected and the output also matches you can just stack everything and anything together even things you should not stack which is incredibly useful for testing things but when you think of a service in Rama I really think of a contract between input and output signatures nothing more, nothing less, which is a very simple idea, but it allows you to stack until like crazy complex applications and it stays simple, magically simple. So yeah, that's kind of how it evolved for me.
Glen (Plabayo): do you remember a bit, when was the moment that you came to realize, okay, now do I understand it or did it just come very slowly and you have no idea?
Brecht: At one point I understood the idea but kept trying to fight it because I always thought "For this use case let's make a new trait for it that describes my use case better." Like doing a request you could call it do_my_special_request which is a little nicer to expose instead of just serve. But that's opened so many issues over time, like I want to unit test this, okay I have to implement this Well for service, everything and anything implements it, so as long as the contract agrees, it just plugs in the entire Rama codebase and everything around it. Every single time you do something custom, that doesn't happen automatically. need to do a lot of work yourself and in the beginning it's hard to con- vince people - myself - why that is so powerful. So for everyone exploring I encourage you to just don't commit to the Service idea, try to invent your own you're gonna very quickly realize why it's so nice and how it saves so much time.
Glen (Plabayo): Yeah. of course, reason why I initially liked it many years ago is because, okay, I can easily stack things because I compare it to functions where a function calls another function and similar with services, you can compose them. Meaning like first I do something here, then I pass the request on and eventually something will give me back a response or maybe something earlier gave a response because there was an error or whatever. But then another benefit I came to realize quite quickly was the fact that let's say I a web server and I have now my entire composition so like my middleware and my router my different endpoints so it's like fully the system is ready there normally what I would have done in the past is okay that point I need to spawn this ⁓ or let's say bind it to a socket and I need to test it with like a real HTTP client, which kind of means I'm also testing everything, even though really just want to test my business logic. And with the service concept, I could just send it, give it a request and it will give me back the response kind of like how I'm just using a client. I don't think that's immediately clear because it kind of ties in what you were saying that everything is a service and so it's hard to look past the abstraction, but beautiful the fact that a server and a client looks the same. But I wonder there ⁓ other examples like you can think of that like didn't immediately work clear to you, but The fact that everything is a service and the fact that how it composed so nicely and the fact that it's just a simple idea, like that it unlocked things that you couldn't otherwise have done.
Brecht: Maybe something I could have done in the past but that was always annoying in pretty much every other stack when you want to test stuff you have to mock it but mocking stuff always felt like Really hacky where you patch something you return something ugly. ⁓ Remember a few months ago gRPC support was added to Rama which was a huge PR and gRPC is in itself quite complex. It does a lot but you have a gRPC client and then you implement the service traits ⁓ the server side for the actual logic and it was crazy simple to just unit these things, the client and the server without needing HTTP, without needing TCP, without all of the stuff. If you do this in other languages, you need to spawn a server, you need to do all of that stuff. Here, you could just literally have the client ,and instead of giving it an HTTP client do the network stuff, you could just literally pass in the gRPC server, you just made, implementing the business logic, skipping of the network stuff and you could instantly do all of that logic. Just put the server inside the client as the network stack and works, which was again, like one of those moments I realized, wow, this is so clean, so simple. And it allows me to test these things I want to test here. Of course you ⁓ to test the entire system. It allows to focus on each module, really clear, really simple. Another use case is Orama we have a mock connector, which is basically a fake connector that you a stream. Which allows you to any connection that will need TCP otherwise, if you implement the test with TCP before, they're always tricky like either spawn them on port 0. need to wait until you get a port then combine everything which involves lot of Overhead is quite slow if you need it for a lot of tests it adds OS Complexity if you don't care about that like if you're implementing HTTP stuff. You don't care TCP you care about the TCP layer just get yourself a client a server and connect them with a fake mock connector and focus on what that layer needs to do, which allows you really test every layer on its own very simple without bringing in everything and ⁓ all things just for HTTP for example. Those are some of the examples they sound but to in the past doing these things how hacky it was or how much code you needed just works with Rama as long as input output match Everything just works, which is really nice.
Glen (Plabayo): Yeah, fully agreed. talking about gRPC, you've using it quite a lot in the recent months. I remember one point you had issues I believe it was timeouts, which was the end symptom. ⁓ And your solution was the fact that you needed a couple of low level configs on the socket, such as TCP, no delay. Can you talk a bit that? Because like, okay. First of all, what were you noticing? And then secondly how did you ⁓ that? Because could be quite interesting to learn about that. And then maybe finish off by how could a framework like Rama help guide the user better or even just help within its to prevent such issues. Because I don't think most people that will use gRPC will actually know that much about TCP to even think about those things.
Brecht: Okay, I noticed the issue because, on all of the gRPC stuff, I have really strict timeouts Which I advise everyone to if you expect it to be fast make sure it is fast ⁓ You can either crash slow or, at the very least, log it if it's slow so you know it's happening And for everything I do I set open telemetry which gives me metrics, traces, loss to debug these things And quite consistently, I was seeing above milliseconds for gRPC, got me investigating Like this is a really documented issue- it's all over the place - but ⁓ TCP, design, doesn't send very small packets because ⁓ it doesn't that ⁓ which means buffers them and there's some rules about when it does send them, but gRPC happens to be almost always small messages, which means you send a message, almost always trigger a timeout unless you're pushing more over the same connection. So you need to set TCP_NODELAY. Most gRPC libraries do this for you, always behind the scenes. In Rama when you use gRPC you provide your own client, so it's not set everywhere because ⁓ it's useful not set value for normal web traffic but for gRPC you do want it. that also brings an important thing with Rama. All the building blocks are there and we also provide abstractions but it's still important that you know about all of the stuff that's there if you want to do low level stuff. So for gRPC I imagine the thing we need to do and it's still on my to do is actually document this but we also have like the easy web client we could also provide a gRPC web client which does these tweaks by design or by default so other people don't need to find this the hard way like I did. But yeah it's really documented it's all over the internet so when I was searching: Why is my gRPC slow? Google gave me 50 pages with all the same answer: ⁓ Is set? No, it wasn't that opens the door but Rama is really powerful you really need to tweak all of the low level stuff as well. TCP stuff, Linux stuff, a lot there. But yeah, when you make a new stuff, definitely set up open telemetry or at least logging so you see what's happening under the hood.
Glen (Plabayo): Yeah, and of course both are integrated within the Rama And I agree, like you can do all the low level stuff, you can control whatever layer of the network you want. That said, as you also ended yourself, we do provide abstractions at various layers. And for most users, I don't think they will need to tweak this. And so I think next to documenting for sure. We need to also provide an abstraction there similar to how for most let's say users need to do simple HTTP, API calls. There is the easy web client, like you mentioned. I think it is right to also provide something like that, like a pre-built transport client either there or somehow provide that support within the existing client ⁓ to make it easier because yeah we only want them to have to worry about the network layer if if they really need to of course otherwise doesn't any sense That said, this is only like a client issue or that's also like on the server. And then next to that you mentioned that most gRPC implementations ship the support built-in like they have like just a single API and you get your client or whatever So then how come they still have the issue? as well or least some of them.
Brecht: they don't really have the issue anymore, but at one point the issue was there. Someone opened an issue for it and it got fixed. The only places where it still happens is more lower level libraries where they're explicitly documented. ⁓ Hey, don't set this for you for these or these reasons, but under almost all circumstances, you do want to set this. Some crazy examples of where you don't want to send this if you're sending this a latency networks that's pretty far and you don't really care about instant ⁓ responses but you do want to send them in batches ⁓ is almost never the use case but libraries do give you the option in that case
Glen (Plabayo): Okay perfect and then we started this episode around the service trade and it went through a very big evolution. I mean we didn't invent the concept to begin with it came I believe Java library if I remember correctly and then it was within Towers in Rust and then we kind of took it from there first tower-sync ⁓ library we made and then finally at Rama we have the current version but even then we had in the beginning not just an input but also a context which was meant for like all kind of including states ⁓ Eventually we to the conclusion it was the wrong abstraction, we removed that. Then we still had the fact, okay, we do need dynamic state, we have extensions. yeah, to begin. What are extensions, Brecht ? and And where do they live, why do we even want them?
Brecht: so let's maybe start at where extensions started. This is very similar if you've worked with other pieces of Rust codebase. Extensions are mostly you need dynamic content. Which is often the case a of different things if you cannot predict that compile time what everyone will need So the way it basically works in most cases is you have a hash map as the keys, you use TypeId of a generic object. When you do TypeId, Rust gives you a unique ID which you can then use to like add runtime, query this in a hash map. Which is also how we did it in the beginning. But the problem we noticed over time with a hash map is that people insert items, they remove items and it becomes really hard to see what's happening under the hood. How it got there, ⁓ there were leaks. different interactions of all that stuff. So trying to fix this issue, which is an issue that happens all over the codebase and that's also really hard to fix. We designed a new system where basically extensions would become a vector would be append-only. So you could kind of see it as a history of extensions being inserted and you can never remove an extension. You could use logic for example put it to this item is used, you should not use it anymore. But it's always there leaving a paper trail of what happened. So it's also a really nice debug tool to see what happened, where did it happen, in what order. At that point we had a really simple vector which needed mutable reference. That was kind working. we kind of moved it all over the place so when we went from a request to a connection we just copied everything over same in other places ⁓ was working for simple use cases but for the more complex stacks it leaked lot of information by leaking I ⁓ we transferred extensions the request to a connection then when you used a connection pool that connection still contained extensions from the original request under cases happened to introduce bugs. ⁓ of that and it really only meant as an intermediate step. So the next we have a structure that is and read-only for the other parts which means we should be able to optimize it. instead of using a normal vector, which got us the append-only vector. It's basically a structure that can grow and where you can insert items only ⁓ to self, no mutable reference needed. Because could use the properties of like, we can only insert items at the back and never modify items that are already in there. By changing reference instead of mutable reference, we could edit extensions from multiple places at once, which sounds like an anti-pattern in Rust, but is actually needed if you happen to have logic scattered in multiple places, which happens a lot for extensions, for connection extensions. user want see what the extensions of connections are. For example, is it broken? Is it healthy? ⁓ But the connection still need to be able to edit these. So by relying on that append-only structure with only reference, ⁓ we could do and ⁓ it's already in and it's working great so far. For the people that are already using it, if there's feedback, please open issues or message us on Discord. And happy to look at those. ⁓ But that got us to append-only VEC. latest transition was really splitting up extensions between and connections. So really seen as distinct extensions now. when a request goes through connector stack, we don't copy. the request extensions to the connector, we only copy extensions that should become part of the connection. If you say I want to connect only to this IP, this IP now becomes part of that connection because it is fundamentally how the connection is used. Other examples are which TLS version is being used. Once a connection is established, that should be part of the connection. If you reuse that connection again, the same information applies. If you put a retry policy on the request, we don't copy it over to the connection because a follow-up request might not want to use that. So they're really distinct. also fork at logical places, meaning when you have a connection, extensions are passed through. If you go from TCP to TLS, we reuse the same extension because logically it's still the same connection. However, when you do multiplexing, which HTTP2 when it's using stream, we fork extensions they have a new storage. They still keep the original extensions as parent so all the stuff that's in there still applies to a stream but a stream has also its own storage to stream specific storage things. ⁓ Examples there it is possible that a stream is broken but the connection is not broken so in that case we insert connection is broken on the stream but not on the connection which allows us to do a lot of complex things. and this is also made to work around upgrades because upgrades are complex items. If you do an upgrade on HTTP 1 you kind of replace the HTTP with the upgrade you just did. Nice, this is easy. But if you upgrade over HTTP 2 you're basically nesting a completely new connection inside a stream which could be doing TLS again, which could be doing a lot of other things again. And things inside that connection don't really affect the outer connection. so they should be isolated but they should still have access to the underlying connection because they do still run over that connection. system allows that. It works "magically" is kind of the word which is something we're removing but here it is needed to like just be transparent about the network stack but it also gives you all the building blocks to go as crazy as you want with this.
Glen (Plabayo): Yeah, and part of complexity or the difficult program space is of course because Rama is a framework it gives you all these building blocks but you can compose them how you want the bounds of ⁓ contracts as specified network specifications of course you will get other issues But as long as you respect the different specifications or how the world operates, you can make up whatever stack you want and Rama makes it pretty easy. In contrast to that, you would need to otherwise even either write all the network and protocol code yourself, which I don't think anybody does. Instead what most people would do is they either go for like a black-box solution where they can just use a config file or something, if that at all, or they would glue together all kind of random libraries, but of course plenty of mistakes and imperfections get made while trying to glue these things. Instead, Rama has all these building blocks, they work nicely together, you can compose them, because we cannot predict how the user will compose them, we of course don't want put them into a surprise scenario where they compose it somehow. Things happen with these extensions because how they passed around and now they get into these situations which, like you mentioned, we had. Because you could remove extensions and... maybe then it no longer contain information it should have contained or maybe shouldn't have and ⁓ it got pretty messy I'm pretty happy that that's where are now as I believe right now you can stack without any fear and I certainly didn't see any surprise anymore since since we arrived at this new design ⁓ I'm not sure how you see evolving from here Brecht
Brecht: I think the extension stuff is mostly finished, core logic, but do have a lot of other things that are using extensions, mainly around TLS special builders. Around these boring builders, were designed before this new extension logic was there, ⁓ so they of solved this problem on their own in a thing that worked at that point of time, but it's quite a complex struct, that does quite a lot of weird things first time you see it. the way I see it evolving is now rely on the new extension logic, really rely on the features it offers and simplify all the users of the extension logic by relying on what extensions offer. But yeah, that's mainly the extensions set
Glen (Plabayo): Okay, very cool. And then to run off this episode, you've been now not only like a of Rama, you're already that for quite a while. You've been using Rama now already at several companies as well as, for stuff that you do in free time, sometimes related to work sometimes not. Can you think of any story like, okay, I'm using Rama now here and it really helped me to unlock something that otherwise would be pretty difficult if not impossible to do.
Brecht: ⁓ is a big word because yeah nothing is impossible. It's just it might be impossible in the required time frame But that's where Rama shines so much like I've been surprised so many times where I implemented like Complex proxy stack or a complex ⁓ network stack in so little time and just worked like very little time and When I do all of these things I always add tests as well. Adding these tests is so simple with Rama as well. All the core stuff it just works which is nice. ⁓ I do this without Rama. ⁓ Let's I stay in the Rust ecosystem. I would have used Tower, Hyper and Tonic probably. But when I combine all pieces ⁓ there's so much glue I need to write myself. Like huge amount of glue. And Rama solves this by being a monorepo basically that just has one standard way of doing everything. It's applied very consistently everywhere and all pieces just work together. Like if you use gRPC, okay you're kind of in HTTP world, you can use all the HTTP stuff. The same kind of applies in tower world tower ecosystem is kind of stuck on an older Rust version which makes this way more painful. It's also way less consistent and the core logic is there but all of the stuff that combines it is not and it's often there that the most subtle bugs exist and where lose a huge amount of time so yeah is just nice it combines everything
Glen (Plabayo): Okay, very cool. And where do you see future going? how do you see ⁓ Rama in the future and what do you expect from it both as ⁓ a maintainer well as a user of ⁓
Brecht: the short term I mostly see Rama evolving on a feature set. Like there's some really nice that we still need to add to Rama. Perfect example of that is HTTP 3 and Quick protocol. ⁓ Because unlocks so many new use cases again. Like ⁓ once have that you can go really crazy. Like you can already go really crazy but that's like fundamental piece. In the world that's still missing. So short term I really see Rama adding more features or more developer tools to make things easier, make things better work with, to debug. ⁓ Dial 9 was added which again will help a huge amount to debug these crazy OS issues or Tokyo issues ⁓ something is stalled that you never expected So short term feature wise, then long term I really see us focusing on speed. performance, throughput just ⁓ out hardware easily. ⁓ nice thing is, if you just update Rama, you will get all of this for free basically, which ⁓ been the case ⁓ a lot of places where I use Rama. Every update, I can remove a lot of special workarounds and stuff I have in code just because right now Rama has solved this automatically for me. ⁓ and it just works probably better than the stuff I have. That doesn't mean I don't need logic anymore, but I can rely way more on the core logic, which also means if I needed an other project, I can just do that without having to reinvent everything.
Glen (Plabayo): Yeah, exactly. Because anyway, might be able to do it, but it distracts you from the business logic. A lot of subtle things can go wrong. And then everybody's basically solving the same books if they find them at all. Because there are so many ways to run something. ⁓ I getting surprised by it because ⁓ right now, Rama is getting used to more more different scenarios, different platforms, different use cases. The amount of books that come in is very little, but when they come in, they're very interesting always. And it solves Wong and then, like you said, Wong that solved, you do an update and you just get it to fix for free, which you wouldn't really get. Otherwise, because you also mentioned, I had this TCP new delay issue. Okay. I had to look it up. You see in your Google results, because you're using Google, like I'm getting 50 pages of people getting the same issue. And that's the kind of thing that you would get for thousands and thousands of different issues. If you need to start. building yourself with these low level components, gluing it yourself together, trying to figure out how to compose this, how to work with this. And that was always a goal to solve that giving someone the power to program their own stack because you do want to program it don't want have to resort to some silly YAML file or something like that where you're almost like pseudo programming in config language, would happen sometimes. But at the same time, you don't wanna have to program everything yourself either. So I'm very, very happy that it's working out.
Brecht: I do have example of something I didn't think about but just ⁓ that you fixed kind of recently is the system resolver in Linux is just blocking by design if you want to do it properly you need to use spawn_blocking in Tokyo ⁓ but if you have a proxy for example that can easily run out of blocking threads But Rama, if you use a system resolver, it has a small cache implemented for you to work around this problem. is one of these things you don't think about when developing these types of applications that you need to think about the DNS resolver being slow on your system or not slow expensive to call basically.
Glen (Plabayo): very much agree. And of course, when things do go wrong or things can be improved, we always accept contributions. Everybody developing network services might have opinions or ideas or might notice things and yeah, to highlight again, we will just have to fix it once and we all benefit from it. very much a community project and I'm also very happy that you are part of it, Brecht, and thank you for your view on the project today here in the podcast, as as the explanation about extensions and how you got here and how you came to understand Services.
Brecht: Thanks, was fun.
Glen (Plabayo): bye.
