Philipp: Hey and welcome to another episode of...
Nicolas: The walking devs.
Philipp: My name is Philipp.
Nicolas: I'm Nicholas
Philipp: You can hear it, is a little bit of a delay this time because we have to this recording inside. This is already the second take of this episode. Unfortunately, we had some audio troubles, which, yeah, I guess is something we have to deal with now. ⁓
Nicolas: was a bit of mistake. So yesterday I would have described to how we ⁓ walking in this beautiful Prata and were not able to walk through the Prata itself because ⁓ there a run going on and we had to walk in the woods, ⁓ which quite nice. But yeah, so after the whole recording, we figured out recording didn't work. ⁓ And so now we it again. And we also had to do a break last week because was a of a rough week and Philipp got sick as well. Let's start into this week's topics. I think I wanted to start it with the front end drama, but there is no front end drama.
Philipp: Yeah, I mean, there is now, interestingly enough, think on Sunday, something happened when like the kind of beef between Versal and Cloudflare went into the next round as Cloudflare was forking an open source package that someone from Versal was working on. As far as I understand, it's like just something they've been playing around with. But of course, Versal already made like a blog post about like, how is a bad take a thing to fork this So yeah, it never gets boring in this Cloudflare versus Vercell timeline. speaking of Cloudflare, there's something really cool to talk about from last week, and that's the release of ⁓ Void Zero, the behind the ⁓ famous Veed bundler, ⁓ has had a release ⁓ last week. and they released a new major version of Veed. They made Veed open source. So Veed plus is kind of a new all-in-one package that includes Veed, but also their new formatter, the testing library, the ⁓ ⁓ everything in one neat package. So Veed plus ⁓ now open source and you can ⁓ it. And more importantly, they also announced this new thing that I've teased earlier called Void Cloud. Void Cloud is a full stack JavaScript framework. includes both, know, the front end but also integrates with like things like database and caches and whatever you need to build like your JavaScript framework. And currently in You can request access to it, but I think it's not public. As of the moment we're recording this. And architecturally, it's tightly coupled to the Cloudflare runtime. So when you make a project with ⁓ void cloud, it assumed that you're to deploy it on Cloudflare, So especially for someone like you, Nicholas, this is probably something questionable.
Nicolas: Yeah, definitely. For me, it's a fascinating that. I'm from one side, I'm super jealous about the whole JavaScript ecosystem all the time because you work so much on tooling. the other hand, I constantly, every time I work on a project with you, Philipp, there is a new format that you tell me that I need to use. So I think it was prettier a few years ago than something else that they came up with. And now it's OX format or so. Like it constantly changes and... This is a bit hard to keep up with. then ⁓ you said, there is this weird thing for me that you always have this Cloudflare dependency for most of JavaScript projects. And I ⁓ understand why. So I'm imagining that the development work must be more cumbersome because you don't have tools you rely on like Postgres and Redis, but you always need to use these Cloudflare wrappers also. And then... ⁓ It's so weird for me that you don't choose the development deployment environment because of the framework you're using. That's really weird for me.
Philipp: Yeah. I think that the main idea of platforms like Cloudflare and Vercell is that they allow you to build your backend in a way that can scale across loads. First of all, transparently and also without having to invest in infrastructure at all. If you were building on Cloudflare workers, for example, never have to worry about vertically slicing your workers. It's just something that will work. You never have to worry about your database scaling because something that was built in. It's like building on top of AWS primitives, right, with managed databases, but ⁓ even to like a higher level of abstraction because in AWS, if you use cloud virtual you still have to think about rollout and like version drift and everything. this kind of level cloud providers are abstracting all of that away. I think main idea is to just let you focus more on the product development. Now, ⁓ if we about the linting stuff, there's next new setups. that's true. whenever something is slow, people in this kind of JavaScript ecosystem don't mind changing things. But personally, I don't even open the editor anymore. So don't really mind ⁓ as much anymore. It's just something that I set up once wise. ⁓ agent will run it automatically for me.
Nicolas: Yeah, that I get. ⁓ Regarding database scaling thing, but you don't have managed Postgres on cloud fair. You just have SQLite, right?
Philipp: That's true, yes, Cloudflare does not have a good database offering.
Nicolas: So you all store GcoLite databases then? Like I still can't see how this scales nicely, like after a few terabytes at least.
Philipp: ⁓ I think if you need a large database that handles more multiple terabytes right now, it's better to not use the Cloudflare database service.
Nicolas: I guess you didn't use planets Kato or so.
Philipp: Yeah, I think so too.
Nicolas: There's also the question for me, have all these services, you have this development environments from Cloudflare and then sometimes we might need Planetscale or so, like these abstractions, they become leaky somehow. have the feeling like the underhooked abstractions become more leaky with this approach.
Philipp: think arguably it's more leaky in the Rails world, where you have to know that it's Postgres and you
Nicolas: You don't need to know. Like in Railsworld you have this...
Philipp: You cannot deploy your app without needing to know that it's post-...
Nicolas: But you can switch to MySQL. the worst thing that could happen is that it rewrites your initial schema script because it needs to adapt to SQLite or MySQL
Philipp: Makes sense. I think it's similar with database or AMS or whatever it's called in JavaScript world.
Nicolas: Are we done with the frontend drama?
Philipp: I think we're done with it, yes.
Nicolas: There's another category I wanted to introduce from the beginning of this podcast, which is the bug of the week. And so I had a bug at work that is quite interesting. And is about a denormalization that didn't work properly anymore. ⁓ this took quite a while to figure out. everyone who doesn't know what denormalization is, it's pretty much you. example, have a table and you always need to join it with another table, which can be expensive at some point. And instead of always joining the other table, you put the data from the other table into that table. So you no longer need to join. This is cumbersome, but it has immense performance improvements at some degrees. I think at university or Any other database lecture that you have, you never get this suggested, right? That's a good idea because you duplicate data in the database anyway, I worked on something like this last year a lot ⁓ and this improvement worked, but it also got out of a few times. And then I figured out again, it got out of sync for some very rough edge cases where ⁓ you import data. into the system and while you import data, you move things around. And all of these edge cases are so hard to catch. Yeah. So this is my bug of the week. And the second bug was active record caching things and then not setting the correct data. And it was again, at least once a month, I write to Philipp after exploring stuff like this. Like, what the fuck? Philip, I hate Rails Active Record so much.
Philipp: Yeah, I can attest to that. I've seen this message multiple times, but I don't think what we have in the JavaScript world is any better, honestly.
Nicolas: Definitely not. every time, and then it's the other thing, I work on some JavaScript things. And then I write to Philip that I hate Grisel or I hate Prisma or whatever. Nothing against the companies that are behind that. I know that they put a lot of work in there, but I just don't get these ideas sometimes, especially with Prisma and RISL that try to infer the migrations from the schema changes. But sometimes it's also not making any sense at all because it tries to copy over the whole table to another table and then So renaming a column, for example, is not something you should do. ⁓ And when you rename the column, I think it does, it tries to copy everything to a new table and then adds a new column, which is the right way. But it's... ⁓ Easy to shoot yourself in the foot so I don't understand how you can work with it.
Philipp: Yeah, yeah. No, guess we are just like big brains and we all write the SQL by hand, the migration, and we don't use the migration tool. I would actually prefer that. I want to go back to your bug of the week because you mentioned these denormalized columns. I'm curious about two things. First of all, I'm in database new, but I know that there is something like views and specifically
Nicolas: No, I was...
Philipp: materialized views that can compute based on data. Is that something that you could also use for denormalization? And second, I'm also curious, how did you find the bugs? How did you debug them? Were you using any API to that for that?
Nicolas: Yeah, true. ⁓ So around the materialized views, you can't have a materialized view on a single column. like I can't just add them. This would be actually cool if this would be a feature, if you have a materialized column. But I don't think that's possible. So I would need to do a materialized view of the whole table. And at that point, would need to copy. It's not a copy of the whole table, but I need so many other columns as well of the table. It doesn't have a performance impact. So the performance is completely gone because we would always need to copy. For materialized views in Postgres, you also need to refresh. So you need to manually refresh the table. ⁓ really? Maybe that... I don't think it's automatically. Now we would need to Google. We know we're doing this podcast. But I think you need to figure that at some point. And this makes it... ⁓
Philipp: I demand a key.
Nicolas: problematic, So in this particular case, would be, I guess the coolest part would have been to have a database trigger. So you are ensured that this thing spawns. with database trigger is that, what was the problem? It's also hard to test, right? Because you can't feature flag it, for example. ⁓ That's one problems. yeah, so materialized views would not have worked, but it's actually one thing that we considered very, very early on. but had to dismiss ⁓ second part of the AI. Yes. so my problem is I can't just spin up codecs and then point it with an MCP to the replica. that's just not possible due to data concerns. But what I tried to do is like, I explained the AI debug Hey, this is the thing that I'm seeing, any ideas. And it unfortunately was quite stupid because. It went into the same direction I did. And we have this update statement where it updates this denormalized column. And here I chose an update statement that is atomic. So I want to ensure that this is course correct within the transaction. then problem was that I said, ⁓ this must be the problem. And then it went into this loop of, I can't be the problem because batch size is 100, so it can't be that only parts of it are And so it went into this endless loop of constantly realizing, ⁓ this might be the problem because this is where the view update thing and it's not possible because we have this acid criteria of POSQUARE. ⁓ So didn't help there, ⁓ but helped me with something else. this problem was that you import data from another system and the timestamps of ⁓ data is different than what it actually is. we this data in our system like three weeks ago, ⁓ but timestamp says it was five years ago because it wasn't imported data. And for me, was weird to see that I didn't that this data got imported. I thought this is old data. So I was thinking about something different. And then at some point I figured out it was importing then it gave me this right direction of, Hey, you can't check the timestamp, but you can check the IDs and the IDs are quite new. me a hint that I should just check the IDs that are around this data and see what timestamps they have. So I can pinpoint it to a specific time. in our logs. And then I explicitly saw this problem where imported data got moved at the same time. exactly at this timestamp, the move happened. And then after this timestamp, everything was corrupt. So this is quite nice debugging this and ⁓ to debug this.
Philipp: Nice. Yeah, that definitely sounds like a problem that only arises once you are operating at GitLab level of scale. Yes. terms of, I think, complexity, but also just how old the code base is and different kind of writes happening at different places.
Nicolas: That's definitely, it's also something you only see in production. Of course you can think this through as an edge case, but it should have been covered actually. So it's not that we didn't think it through, it's just that it was a super rare edge case and it only appeared after five months being in production. So that's the problem. And also a discussion, but it's sometimes for me the problem when I see all of these black coded stuff and then I'm thinking of the production side of things and I'm like, Yeah, it's nice that this works now, but real code only proves itself in production and things are different. And there's one more thing that I figured out only when I worked at AI teams. At AI teams, you don't need to worry too much about data, ⁓ which quite nice actually, because data is always produced, right? ⁓ It's a chat or whatever, it gives you some data back and it generates stuff. and then you save it somewhere, doesn't get modified. This is quite nice in this generative AI world where you build stuff like this instead of having to deal with existing data that gets mutated.
Philipp: Makes sense. Cool, I think we should move on to the positive section of the week. I think it was a little bit of a hard week to find something positive, but I personally ⁓ thought it's pretty cool that ACP, the agent-client protocol that's developed to talk to coding agent ⁓ it's a coding protocol, ⁓ is getting some more traction now. I think personally the coding agent is becoming more more prevalent and abstractions over it for workflows or something are very much welcome because personally I've been using different coding agents. Even this I've been using at least two, maybe even three different implementations. So yeah, I'm pretty happy that this is getting more traction now.
Nicolas: This is super nice. it's basically the LSP for coding agents.
Philipp: Yeah, that's, that's true.
Nicolas: Okay. And this was your highlight of the week. And for me, the positive thing was actually that we hacked on something Do you want to tell us about this?
Philipp: Yeah, sure. Nikon and me had some time last week to actually get our grinding gears on again. ⁓ we were thinking through kind of how software can automate more stuff in the future. And what we've done is like a very simple prototype of like a server component where you can spawn coding agents in their own little sandboxes can run on premise, like either on your own server or even ⁓ a ⁓ hardware that you have in your office or something. So we were playing around with the ideas of how much can we automate in this workflow and building the infrastructure for that. And I think one tool that ⁓ of us have gotten pretty excited about over that ⁓ few days that we had done was Inkis. It's a orchestrator, I guess. So you can use it to spin up your containers. It supports both LXC like Linux containers, the kind of lightweight thing Docker uses, but also virtual machines. you can connect multiple remote servers to it. And it will also support snapshotting and kind of all of the things that you would expect. So really can use that to spawn like a lightweight def instance that can be stateful as opposed to some of the other solutions.
Nicolas: Yeah, this was really cool. It was a cool finding and it was so Philip and I had this vague ideas, similar vague ideas about how this sandboxing should work. And as always, I give this idea to Philip and then Philip does the research and he came back with Incos, which was quite nice. was interesting to work together. ⁓ We didn't this for a long time, ⁓ that serious. it was also interesting our different approaches. So. For example, I absolutely didn't care about the contract initially of the APIs. So how do we pass the data from the sandbox to our orchestration server, to the web server and so on. I was just ensuring this thing works at all, like that I can pass through data. And then I wanted to go to the next step of how does the data is shaped like. And it was quite fun to see that we had problems in our collaboration there. It was easy to resolve, but we had a different mindset Philip was only thinking initially about the contract side of thing, how the APIs communicate to each other. And I was thinking about it works, that they can communicate.
Philipp: We kind of just the data flow kind of first approach versus the API contracts first approach. But eventually we were able to figure it out. the agent did a lot of slop that we are now slowly un-slopping, but yeah.
Nicolas: And ⁓ I'm always the one complaining seems, ⁓ because I was constantly complaining was in JavaScript ⁓ that this should be Alexia ⁓ it's a perfect opportunity to sandboxes that you control and orchestrate, right? Like it's a gen server ⁓ basically in the Alexia land. yeah, sorry that I'm always complaining, I'm, was quite impressed that this works so neatly with JavaScript and It is also one of the reasons where Alexia doesn't shine is the integration with libraries. So there was an ACP protocol library already for TypeScript. And so I don't need to reinvent this, which would have been the case in Alexia, I guess.
Philipp: Yep. the ecosystem is much broader for sure.
Nicolas: Yeah, I agree.
Philipp: With you that Alexia would have been better for the kind of distributed system that we're building here, of course.
Nicolas: Yeah. one more fun story So, Philipp and ⁓ I, there's ⁓ this Linux thing, which a quite safe sandbox to run things on. ⁓ And it's so fun when you think back, 15 years ago, we both worked on a product was allowing Everyone to compete against each other with algorithms. This was one of the student projects we did ⁓ our high school. So build a game basically, and then you can build an algorithm for this game and you try to beat each other. And this means that the code users need to run on your own server. And back then we didn't know about Linux containers. ⁓ And I'm not sure even if they though. ⁓ I hope they didn't exist back then.
Philipp: The things we had to go to make this somewhat secure was horrible. I think back then it was like we did this in C sharp and in Java, which also again horrible. But for the avid listener, I think it's even open source on GitHub somewhere. I'm like 90 % sure. Yeah. Yeah, the whole sandboxing we built back then.
Nicolas: No, this code should not have been in the public. No one should see my old 15 year old Java code.
Philipp: ⁓ the internet never forgets, Nico.
Nicolas: ⁓ no. Good. ⁓ But I think that's a good wrap up for our episodes, our different episodes where we didn't walk.
Philipp: Yeah, thank you so much everyone for listening.
Nicolas: Yeah. Thank you a lot. ⁓ Thanks for listening in again and have a nice rest of your day.
Nicolas: The walking devs.
Philipp: My name is Philipp.
Nicolas: I'm Nicholas
Philipp: You can hear it, is a little bit of a delay this time because we have to this recording inside. This is already the second take of this episode. Unfortunately, we had some audio troubles, which, yeah, I guess is something we have to deal with now. ⁓
Nicolas: was a bit of mistake. So yesterday I would have described to how we ⁓ walking in this beautiful Prata and were not able to walk through the Prata itself because ⁓ there a run going on and we had to walk in the woods, ⁓ which quite nice. But yeah, so after the whole recording, we figured out recording didn't work. ⁓ And so now we it again. And we also had to do a break last week because was a of a rough week and Philipp got sick as well. Let's start into this week's topics. I think I wanted to start it with the front end drama, but there is no front end drama.
Philipp: Yeah, I mean, there is now, interestingly enough, think on Sunday, something happened when like the kind of beef between Versal and Cloudflare went into the next round as Cloudflare was forking an open source package that someone from Versal was working on. As far as I understand, it's like just something they've been playing around with. But of course, Versal already made like a blog post about like, how is a bad take a thing to fork this So yeah, it never gets boring in this Cloudflare versus Vercell timeline. speaking of Cloudflare, there's something really cool to talk about from last week, and that's the release of ⁓ Void Zero, the behind the ⁓ famous Veed bundler, ⁓ has had a release ⁓ last week. and they released a new major version of Veed. They made Veed open source. So Veed plus is kind of a new all-in-one package that includes Veed, but also their new formatter, the testing library, the ⁓ ⁓ everything in one neat package. So Veed plus ⁓ now open source and you can ⁓ it. And more importantly, they also announced this new thing that I've teased earlier called Void Cloud. Void Cloud is a full stack JavaScript framework. includes both, know, the front end but also integrates with like things like database and caches and whatever you need to build like your JavaScript framework. And currently in You can request access to it, but I think it's not public. As of the moment we're recording this. And architecturally, it's tightly coupled to the Cloudflare runtime. So when you make a project with ⁓ void cloud, it assumed that you're to deploy it on Cloudflare, So especially for someone like you, Nicholas, this is probably something questionable.
Nicolas: Yeah, definitely. For me, it's a fascinating that. I'm from one side, I'm super jealous about the whole JavaScript ecosystem all the time because you work so much on tooling. the other hand, I constantly, every time I work on a project with you, Philipp, there is a new format that you tell me that I need to use. So I think it was prettier a few years ago than something else that they came up with. And now it's OX format or so. Like it constantly changes and... This is a bit hard to keep up with. then ⁓ you said, there is this weird thing for me that you always have this Cloudflare dependency for most of JavaScript projects. And I ⁓ understand why. So I'm imagining that the development work must be more cumbersome because you don't have tools you rely on like Postgres and Redis, but you always need to use these Cloudflare wrappers also. And then... ⁓ It's so weird for me that you don't choose the development deployment environment because of the framework you're using. That's really weird for me.
Philipp: Yeah. I think that the main idea of platforms like Cloudflare and Vercell is that they allow you to build your backend in a way that can scale across loads. First of all, transparently and also without having to invest in infrastructure at all. If you were building on Cloudflare workers, for example, never have to worry about vertically slicing your workers. It's just something that will work. You never have to worry about your database scaling because something that was built in. It's like building on top of AWS primitives, right, with managed databases, but ⁓ even to like a higher level of abstraction because in AWS, if you use cloud virtual you still have to think about rollout and like version drift and everything. this kind of level cloud providers are abstracting all of that away. I think main idea is to just let you focus more on the product development. Now, ⁓ if we about the linting stuff, there's next new setups. that's true. whenever something is slow, people in this kind of JavaScript ecosystem don't mind changing things. But personally, I don't even open the editor anymore. So don't really mind ⁓ as much anymore. It's just something that I set up once wise. ⁓ agent will run it automatically for me.
Nicolas: Yeah, that I get. ⁓ Regarding database scaling thing, but you don't have managed Postgres on cloud fair. You just have SQLite, right?
Philipp: That's true, yes, Cloudflare does not have a good database offering.
Nicolas: So you all store GcoLite databases then? Like I still can't see how this scales nicely, like after a few terabytes at least.
Philipp: ⁓ I think if you need a large database that handles more multiple terabytes right now, it's better to not use the Cloudflare database service.
Nicolas: I guess you didn't use planets Kato or so.
Philipp: Yeah, I think so too.
Nicolas: There's also the question for me, have all these services, you have this development environments from Cloudflare and then sometimes we might need Planetscale or so, like these abstractions, they become leaky somehow. have the feeling like the underhooked abstractions become more leaky with this approach.
Philipp: think arguably it's more leaky in the Rails world, where you have to know that it's Postgres and you
Nicolas: You don't need to know. Like in Railsworld you have this...
Philipp: You cannot deploy your app without needing to know that it's post-...
Nicolas: But you can switch to MySQL. the worst thing that could happen is that it rewrites your initial schema script because it needs to adapt to SQLite or MySQL
Philipp: Makes sense. I think it's similar with database or AMS or whatever it's called in JavaScript world.
Nicolas: Are we done with the frontend drama?
Philipp: I think we're done with it, yes.
Nicolas: There's another category I wanted to introduce from the beginning of this podcast, which is the bug of the week. And so I had a bug at work that is quite interesting. And is about a denormalization that didn't work properly anymore. ⁓ this took quite a while to figure out. everyone who doesn't know what denormalization is, it's pretty much you. example, have a table and you always need to join it with another table, which can be expensive at some point. And instead of always joining the other table, you put the data from the other table into that table. So you no longer need to join. This is cumbersome, but it has immense performance improvements at some degrees. I think at university or Any other database lecture that you have, you never get this suggested, right? That's a good idea because you duplicate data in the database anyway, I worked on something like this last year a lot ⁓ and this improvement worked, but it also got out of a few times. And then I figured out again, it got out of sync for some very rough edge cases where ⁓ you import data. into the system and while you import data, you move things around. And all of these edge cases are so hard to catch. Yeah. So this is my bug of the week. And the second bug was active record caching things and then not setting the correct data. And it was again, at least once a month, I write to Philipp after exploring stuff like this. Like, what the fuck? Philip, I hate Rails Active Record so much.
Philipp: Yeah, I can attest to that. I've seen this message multiple times, but I don't think what we have in the JavaScript world is any better, honestly.
Nicolas: Definitely not. every time, and then it's the other thing, I work on some JavaScript things. And then I write to Philip that I hate Grisel or I hate Prisma or whatever. Nothing against the companies that are behind that. I know that they put a lot of work in there, but I just don't get these ideas sometimes, especially with Prisma and RISL that try to infer the migrations from the schema changes. But sometimes it's also not making any sense at all because it tries to copy over the whole table to another table and then So renaming a column, for example, is not something you should do. ⁓ And when you rename the column, I think it does, it tries to copy everything to a new table and then adds a new column, which is the right way. But it's... ⁓ Easy to shoot yourself in the foot so I don't understand how you can work with it.
Philipp: Yeah, yeah. No, guess we are just like big brains and we all write the SQL by hand, the migration, and we don't use the migration tool. I would actually prefer that. I want to go back to your bug of the week because you mentioned these denormalized columns. I'm curious about two things. First of all, I'm in database new, but I know that there is something like views and specifically
Nicolas: No, I was...
Philipp: materialized views that can compute based on data. Is that something that you could also use for denormalization? And second, I'm also curious, how did you find the bugs? How did you debug them? Were you using any API to that for that?
Nicolas: Yeah, true. ⁓ So around the materialized views, you can't have a materialized view on a single column. like I can't just add them. This would be actually cool if this would be a feature, if you have a materialized column. But I don't think that's possible. So I would need to do a materialized view of the whole table. And at that point, would need to copy. It's not a copy of the whole table, but I need so many other columns as well of the table. It doesn't have a performance impact. So the performance is completely gone because we would always need to copy. For materialized views in Postgres, you also need to refresh. So you need to manually refresh the table. ⁓ really? Maybe that... I don't think it's automatically. Now we would need to Google. We know we're doing this podcast. But I think you need to figure that at some point. And this makes it... ⁓
Philipp: I demand a key.
Nicolas: problematic, So in this particular case, would be, I guess the coolest part would have been to have a database trigger. So you are ensured that this thing spawns. with database trigger is that, what was the problem? It's also hard to test, right? Because you can't feature flag it, for example. ⁓ That's one problems. yeah, so materialized views would not have worked, but it's actually one thing that we considered very, very early on. but had to dismiss ⁓ second part of the AI. Yes. so my problem is I can't just spin up codecs and then point it with an MCP to the replica. that's just not possible due to data concerns. But what I tried to do is like, I explained the AI debug Hey, this is the thing that I'm seeing, any ideas. And it unfortunately was quite stupid because. It went into the same direction I did. And we have this update statement where it updates this denormalized column. And here I chose an update statement that is atomic. So I want to ensure that this is course correct within the transaction. then problem was that I said, ⁓ this must be the problem. And then it went into this loop of, I can't be the problem because batch size is 100, so it can't be that only parts of it are And so it went into this endless loop of constantly realizing, ⁓ this might be the problem because this is where the view update thing and it's not possible because we have this acid criteria of POSQUARE. ⁓ So didn't help there, ⁓ but helped me with something else. this problem was that you import data from another system and the timestamps of ⁓ data is different than what it actually is. we this data in our system like three weeks ago, ⁓ but timestamp says it was five years ago because it wasn't imported data. And for me, was weird to see that I didn't that this data got imported. I thought this is old data. So I was thinking about something different. And then at some point I figured out it was importing then it gave me this right direction of, Hey, you can't check the timestamp, but you can check the IDs and the IDs are quite new. me a hint that I should just check the IDs that are around this data and see what timestamps they have. So I can pinpoint it to a specific time. in our logs. And then I explicitly saw this problem where imported data got moved at the same time. exactly at this timestamp, the move happened. And then after this timestamp, everything was corrupt. So this is quite nice debugging this and ⁓ to debug this.
Philipp: Nice. Yeah, that definitely sounds like a problem that only arises once you are operating at GitLab level of scale. Yes. terms of, I think, complexity, but also just how old the code base is and different kind of writes happening at different places.
Nicolas: That's definitely, it's also something you only see in production. Of course you can think this through as an edge case, but it should have been covered actually. So it's not that we didn't think it through, it's just that it was a super rare edge case and it only appeared after five months being in production. So that's the problem. And also a discussion, but it's sometimes for me the problem when I see all of these black coded stuff and then I'm thinking of the production side of things and I'm like, Yeah, it's nice that this works now, but real code only proves itself in production and things are different. And there's one more thing that I figured out only when I worked at AI teams. At AI teams, you don't need to worry too much about data, ⁓ which quite nice actually, because data is always produced, right? ⁓ It's a chat or whatever, it gives you some data back and it generates stuff. and then you save it somewhere, doesn't get modified. This is quite nice in this generative AI world where you build stuff like this instead of having to deal with existing data that gets mutated.
Philipp: Makes sense. Cool, I think we should move on to the positive section of the week. I think it was a little bit of a hard week to find something positive, but I personally ⁓ thought it's pretty cool that ACP, the agent-client protocol that's developed to talk to coding agent ⁓ it's a coding protocol, ⁓ is getting some more traction now. I think personally the coding agent is becoming more more prevalent and abstractions over it for workflows or something are very much welcome because personally I've been using different coding agents. Even this I've been using at least two, maybe even three different implementations. So yeah, I'm pretty happy that this is getting more traction now.
Nicolas: This is super nice. it's basically the LSP for coding agents.
Philipp: Yeah, that's, that's true.
Nicolas: Okay. And this was your highlight of the week. And for me, the positive thing was actually that we hacked on something Do you want to tell us about this?
Philipp: Yeah, sure. Nikon and me had some time last week to actually get our grinding gears on again. ⁓ we were thinking through kind of how software can automate more stuff in the future. And what we've done is like a very simple prototype of like a server component where you can spawn coding agents in their own little sandboxes can run on premise, like either on your own server or even ⁓ a ⁓ hardware that you have in your office or something. So we were playing around with the ideas of how much can we automate in this workflow and building the infrastructure for that. And I think one tool that ⁓ of us have gotten pretty excited about over that ⁓ few days that we had done was Inkis. It's a orchestrator, I guess. So you can use it to spin up your containers. It supports both LXC like Linux containers, the kind of lightweight thing Docker uses, but also virtual machines. you can connect multiple remote servers to it. And it will also support snapshotting and kind of all of the things that you would expect. So really can use that to spawn like a lightweight def instance that can be stateful as opposed to some of the other solutions.
Nicolas: Yeah, this was really cool. It was a cool finding and it was so Philip and I had this vague ideas, similar vague ideas about how this sandboxing should work. And as always, I give this idea to Philip and then Philip does the research and he came back with Incos, which was quite nice. was interesting to work together. ⁓ We didn't this for a long time, ⁓ that serious. it was also interesting our different approaches. So. For example, I absolutely didn't care about the contract initially of the APIs. So how do we pass the data from the sandbox to our orchestration server, to the web server and so on. I was just ensuring this thing works at all, like that I can pass through data. And then I wanted to go to the next step of how does the data is shaped like. And it was quite fun to see that we had problems in our collaboration there. It was easy to resolve, but we had a different mindset Philip was only thinking initially about the contract side of thing, how the APIs communicate to each other. And I was thinking about it works, that they can communicate.
Philipp: We kind of just the data flow kind of first approach versus the API contracts first approach. But eventually we were able to figure it out. the agent did a lot of slop that we are now slowly un-slopping, but yeah.
Nicolas: And ⁓ I'm always the one complaining seems, ⁓ because I was constantly complaining was in JavaScript ⁓ that this should be Alexia ⁓ it's a perfect opportunity to sandboxes that you control and orchestrate, right? Like it's a gen server ⁓ basically in the Alexia land. yeah, sorry that I'm always complaining, I'm, was quite impressed that this works so neatly with JavaScript and It is also one of the reasons where Alexia doesn't shine is the integration with libraries. So there was an ACP protocol library already for TypeScript. And so I don't need to reinvent this, which would have been the case in Alexia, I guess.
Philipp: Yep. the ecosystem is much broader for sure.
Nicolas: Yeah, I agree.
Philipp: With you that Alexia would have been better for the kind of distributed system that we're building here, of course.
Nicolas: Yeah. one more fun story So, Philipp and ⁓ I, there's ⁓ this Linux thing, which a quite safe sandbox to run things on. ⁓ And it's so fun when you think back, 15 years ago, we both worked on a product was allowing Everyone to compete against each other with algorithms. This was one of the student projects we did ⁓ our high school. So build a game basically, and then you can build an algorithm for this game and you try to beat each other. And this means that the code users need to run on your own server. And back then we didn't know about Linux containers. ⁓ And I'm not sure even if they though. ⁓ I hope they didn't exist back then.
Philipp: The things we had to go to make this somewhat secure was horrible. I think back then it was like we did this in C sharp and in Java, which also again horrible. But for the avid listener, I think it's even open source on GitHub somewhere. I'm like 90 % sure. Yeah. Yeah, the whole sandboxing we built back then.
Nicolas: No, this code should not have been in the public. No one should see my old 15 year old Java code.
Philipp: ⁓ the internet never forgets, Nico.
Nicolas: ⁓ no. Good. ⁓ But I think that's a good wrap up for our episodes, our different episodes where we didn't walk.
Philipp: Yeah, thank you so much everyone for listening.
Nicolas: Yeah. Thank you a lot. ⁓ Thanks for listening in again and have a nice rest of your day.
