Using AI to Complete a Half-Finished Rewrite (Ep. 169)의 전체 녹취록

the idea is okay, we want to they want to show the power of AI. They say, look, we this is Isaac: You Jeffrey Sherman: a ten year old problem. Can you and another developer in two weeks finish this thing off using the leverage of AI? You know, get get the V three API up to parity with the V one API and finish this rewrite. ten years later. Isaac: Okay. Jeffrey Sherman: Right. Ten years later. we you know the Odyssey is a is a thing right now. Isaac: So how's that going? Is this the war or is this the journey home for the 10 years? OK, that's the analogy here. V1 is the war. Jeffrey Sherman: it's the journey home that's been 10 years, right? Because the war happened. The the V1 right? The Iliad was going from the V1 to the V2 to the V3. And the V3 is in production. We are telling customers to use it. And the V1 is in production. We are telling customers not to use it, but they have to because there is no no equivalent. So now I'm trying to bring the ship home. We're through the Iliad, Isaac: Got it. Jeffrey Sherman: we're into the Odyssey. We're keeping it. you know, timely by referencing twenty three hundred year old stories. That that's how we roll. Isaac: I mean the movie just came out so it's pretty timely. Jeffrey Sherman: Yes. And so we are at the beginning of this project and I want to lay out the project and we'll do a part two and see do I succeed or fail? Because I will publicly own my success or failure. Isaac: Okay, and then list out, yeah, you'll have to list out which hurdles you ran into and map them to Odysseus' enemies that he's run into, the islands that he's Jeffrey Sherman: I haven't seen the movie and I read the book more than thirty years ago, so Isaac: Okay, something is the siren, something is the cyclops, you know. Anyway. Jeffrey Sherman: Yes. But if you want to know about all this references in our book, which is coming soon, we talk about that 'cause we use the we use a lot of Greek metaphors and we definitely reference the sirens from the Odyssey. But anyway. Isaac: Stay tuned. Jeffrey Sherman: Shameless plugs. Isaac: Mm-hmm. Jeffrey Sherman: So the the project is you know get V3 up to parity. And so before I even started, I made sure to set guardrails on this thing. You know, one is I we're at feature parity, we are not at a one-to-one compliance. The V1 was an organic ad hocke kind of a thing. The V3 is a RESTful API. And so there should not be a w there should not necessarily be a one-to-one translation of all the endpoints. Because not all the endpoints map cleanly. But functionality Isaac: Hmm. Jeffrey Sherman: wise, right, the the goal is functionality wise, there should be nothing that you can do in the V one API that you can't do in the V three API. that is the goal. and for clarification I also specified we're talking about the documented V1 API, so not covering anything that exists in the code base that we never told customers about. And Isaac: Right. So using some logs on the endpoints to see which ones are

worth porting, there might be some, even some that are live that Snowen uses. You don't have to port. Jeffrey Sherman: Well, there's probably stuff that's live that's in use internally, but customers don't know about it. And so that's not the and it's the comparison against the live documentation for the V3 endpoints. So you i as part of this thing, if something exists but we haven't told customers about it and we haven't told them that they can officially use it, it doesn't count. Isaac: Okay. Jeffrey Sherman: And so I had a an agent, in this case I'm using Devon AI as my main tool, because I'm familiar with it and it works pretty well. said, okay, you know, spin up, compare the documented v1 and the documented v three things, give me a list of everything in v1 that's not covered. And it gave me a list of about seventy five things. Isaac: Okay. Jeffrey Sherman: And you know, it some of it or some things are, hey, this feature is missing entirely. Like that you're missing all the crud. You can Isaac: Mm-hmm. Jeffrey Sherman: crud on v1, that doesn't exist at all on v3. So that's four things, five with the list, and boom. but then there's a lot of things where it's like, well, you can do this in v1 and you can half do it in v3, but you can't do, and then it would give me a list of all the things you can't do. Isaac: Got it. Jeffrey Sherman: So I got a good list. And so from there I said, okay, well, started making my burn down. Basically, I'm like taking this as a burn down kind of project where I know what success looks like. I have it bounded domain. And you know, step one is sort of, okay, here's here's my gaps. Now that's how am I going to close these gaps? And so the next step was, okay, now actually analyze the code. And tell me Isaac: Mm-hmm. Jeffrey Sherman: what do I already have that could close these gaps that's not in the documentation? And that, you know, it it spun around and it's like, okay, about a third. Isaac: Okay. Jeffrey Sherman: Maybe. Right, because there's a lot of stuff. And so I then had it the I then worked up a skill where I'm like, okay, right, here's the gaps, and I def showed it how to do the gaps. And I'm like, okay, now here here's how I want to close the gaps. Because remember, closing the gap requires that I update the documentation. So Isaac: Mm-hmm. Jeffrey Sherman: you know, any gap I close, it's code release. Update the documentation. Three steps, three individual changes. Well, not necessarily individual changes. And then because I'm taking it, looking at it from a I've done millions of rewrites, and I understand I'm trying to land the ship and not necessarily trying Isaac: Yeah. Jeffrey Sherman: to close the gaps. I've invented another requirement of okay, now also I have to go back. To the V1 API documentation and document what the what the v3 endpoint or endpoints that we want you to use instead are. So not just it exists, but now I'm actually telling you that it exists and what you should use instead. Trying to nudge, right? It's been a decade that we've been saying this v1 Isaac: Yeah. Jeffrey Sherman: endpoint, you shouldn't use it. Now I'm actually telling you what you should use instead. Like that the nud. I'm not it. It's not a much of a nudge, but look, look, here's a documentation so that when 'cause it's not I'm not talking to humans really. I'm I'm talking to agents, which are gonna scrape the Isaac: Yeah. Jeffrey Sherman: documentation. And if I tell them, hey, this is deprecated, use this other endpoint instead, the agents will do it. They'll be happy to do it. Humans will move slow, humans will ignore all that stuff. But with agents, updating this documentation is a key piece of actually getting the V one di to to not be used anymore. Because Isaac: And this is public documentation too, not just internal documentation. This is updating the actual public facing, how to use, or what to expect from these endpoints as well. Jeffrey Sherman: Correct. The the done means that the public documentation is up to date. Right. So so publicly to to redefine it, done means that everything that you can do in V1 exists as a documented endpoint in V three and a requirement that I've added, the V th the V1 documentation has a link to the V three documentation that you should use instead. Isaac: Good. Jeffrey Sherman: And again, we have no particular plans to sunset or to remove any of the V1 endpoints, but after more than a decade, every single V1 endpoint is still in use. and we did some analysis. It's fifteen percent of our total traffic is the V1. Isaac: Wow. Jeffrey Sherman: So, you know, it's a case of we haven't landed the ship. First of all, we don't have anywhere to land the ship. And second of all, we haven't done a good job at actually trying to land it. And this, you know, going back to our policies or our discussion of this is the the standard disaster of a half launch rewrite. You have the new thing, you have the old thing, people are going to use both. Isaac: Mm-hmm. Jeffrey Sherman: You're in a worse position than you started with. Isaac: Right. Jeffrey Sherman: Where if you had Theseus shipped. then you wouldn't have been in this situation. You would have evolved. Isaac: So what's your method for then in this case, practically from coding? Are you getting ready for a strangler fig style cut over? Besides the documentation updates, what's your actual step-by-step CTShift approach programmatically? Jeffrey Sherman: Well in this case, the the assignment lends itself I don't need to strangler fig, or I'm not going to strangler fig because I have an A and I have a B and A and B don't have to be the same thing. So effectively I I'm free to launch everything dark and migration is not part of the project. Isaac: Okay. Jeffrey Sherman: So you can have the V1, I can set up the V three, as long as the V three works correctly, that is sufficient in terms of this project. Because I don't have to do the implementation, I don't have to do the

migration, it it changes the scope. So Isaac: Mm-hmm. Jeffrey Sherman: instead of focusing on how I'm going to cut over, because I don't need to, the the question in my approach is how am I going to have the agent do all of my testing to prove that the two Isaac: Mm. Jeffrey Sherman: are the same? Right? Because you you can have unit tests, and I do, and you can have sort of integration tests, but how do you actually Prove to yourself that hey this new one is the same functionality as the old one. Isaac: Right. Jeffrey Sherman: And what I've stumbled on or not stumbled on, what what I've what I'm building towards is okay. Let's start with the gets. You got the v1 get, you got the v3 get. Isaac: Mm-hmm. Jeffrey Sherman: I i if we're saying that the functionality is the same, it should be the same. If it you know for whatever parts of it it covers, it should cover. And I can have the agent write that. so for each get that I'm doing, I have the agent write a comparator in Python. And I'm using a large, densely cluttered test account so that I have like real data, not just canned Isaac: Yeah, yeah. Jeffrey Sherman: data, but like, okay. You know, you here's this account that has all this fake data, realistic but fake data, that's go right. And Isaac: You just want to make sure that every column returns essentially if you have like a series of data here. Jeffrey Sherman: it's different, so you would notice, you wouldn't like, well, you know, it's every user is test user. It's it's dense, and I'm having the agent write that script for me. and then I'm r executing it myself. mostly because I I want to actually verify with my own eyes that it it did do the same thing. Also, I don't want to give it the keys, the API keys to make all these calls. Isaac: Fair enough. Jeffrey Sherman: and so that is where I am right now, because it's the beginning of the project. I'm like, okay, do the gets, verify the gets. My next my thought is, as I'm gonna iterate on this, is okay, do the gets, great. Now do a post. you right now making more elaborate thing, do Isaac: Mm-hmm. Jeffrey Sherman: a have it do a v1 post, have it do a v2, v3 post, and then use the v1 and the v3 gets to compare. the data between the two. And then do the same thing with the puts. and then do the same thing with deletes. Isaac: Is there any, whenever you're comparing what the responses are, are you also doing comparisons for how the data is mutated, if it was mutated? Well, I guess you haven't gotten to the puts in the post yet. But just to make sure, even if the response says 201 or whatever, that something actually changed rather than it just spitting back data that you've given to it in the post. Jeffrey Sherman: Yes, so l let me go because I think I did a bad job explaining. So I thought is if I know the get is good, Isaac: Mm-hmm. Jeffrey Sherman: then I can do a v1 post and a v three post, and I can then do a v3 get on both of them for the for the post from the v1 and the post from the v three, Isaac: Mm-hmm. Jeffrey Sherman: and then use the gets to compare Isaac: Yeah. Jeffrey Sherman: and verify that the two things are the same. Isaac: Got it. OK. But still no database checking for any kind of mutation that might have happened that wouldn't be part of the response. Jeffrey Sherman: I did not I I have not I debated it I and I was like I don't have direct database access. Well I mean I do. I manually do, but I don't have it programmatically. and so I'm Isaac: Yeah, and then mean, honestly, the agent might catch it anywhere. I can test for what? You could catch it in some kind of integration test at rights or whatever. Jeffrey Sherman: R and for the early stuff, for the v restful stuff, it's Isaac: Mm-hmm. Jeffrey Sherman: fairly simple crud. So there shouldn't be wild side effects and if it made an endpoint with the wild with you know, that did anything abnormal it would stick out like Isaac: Mm-hmm. Jeffrey Sherman: a sore thumb. So hopefully I would catch it. Isaac: So then what is your release process? If you have an A and a B and you can launch silently, are you still going to try to release not in a big bang cutover style, but release kind of roll out to a certain subset of customers? Jeffrey Sherman: no, instead I was going to do a soft release. Simply merge at do Isaac: Mm-hmm. Jeffrey Sherman: the additive merging. release as it goes, you know, in the regular cycle. So my work is not disruptive. The regular software releases go, we you know we do once a week. So the change whatever changes get done in that week will go out, and then I will update the documentation. And then I will update the V1 documentation. And so that Isaac: Okay. Jeffrey Sherman: gives us a soft rollout of okay, this thing's here, everyone could use it, but nobody is. But then usage will slowly ramp up. Right, because it's live. Isaac: I see. You're just going to keep them both live. This is just additive. This is just parity and additive stuff, whereas most of your customers are still using V1. So they still have the ability to use V1, and you're nudging them. Hey, Jeffrey Sherman: Right. Isaac: now that we have real parity, you should actually do that. So the project's not really done until you get all the customers actually using it, and you've sunset V1. Jeffrey Sherman: Well, I would say the rewrite is not done until you actually sunset Isaac: Mm. Jeffrey Sherman: the V1. But I'm trying to use AI in this case to finish, to get the this is a a rewrite that is you know been floundering for 10 years now. Isaac: Mm. Jeffrey Sherman: Using AI to push it forward, because this is something we talk a lot about i in the book of it's not enough to simply write the code. But it is necessary to write the code. And that's one of the the things that people get wrong about rewrites is simply writing the rewriting the code. If you're going to write all new code, that's only the first part of your problem. Now you actually have to get that code into production

and to get people to use it. And and in this example, we are ten years in and we haven't even finished writing the code. And so Isaac: Yeah. Jeffrey Sherman: this project is to use AI to finish off writing the code, which is a necessary but not sufficient. thing to finish the rewrite itself. Isaac: So do you already have the plan to get people to use v3 done? Like is the planning for that? for me, it seems kind of odd, guess, commit to finishing it for the sake of finishing it, unless you just have free time. If there isn't like a rollout plan or someone's already got the plan for getting people to migrate off of that, because otherwise you just kind of finished it, but nobody really has an incentive to move. Or do you want to sunset the old one? Is it OK if they use the old one still? Or are you just trying to let them use the old one, but then hide the V1 API docs later? That way everyone can only find V3 or something like that. Jeffrey Sherman: Great question. So the impetus is everything is agentic and right now our MCP server we we've added an MCP server, but we only pointed it to the V3 endpoints. Isaac: Ooh, okay, that is intensive, okay. Jeffrey Sherman: And so finishing off the V the migrate or getting to parity unlocks the full agenc options for our MCP users, which in and of itself is y a necess a a huge win. Isaac: Yeah. Jeffrey Sherman: And it also right like it it as we talk about iterative development, in two weeks I can close the gaps, but in two weeks I cannot get everyone to move. But as I mentioned at at the start, it's okay, I added a thing of okay, I'm going to update the documentation to link to the V3. And that will certainly for any agentic development, which is probably most development at this point, right? I I can't imagine there are many developers who are going to look Isaac: Yeah. Jeffrey Sherman: at our API docs and write a manual integration at this point in time. Isaac: Right. Jeffrey Sherman: Though the having the documentation updated, all net new stuff will very quickly move to the V3. which won't change all the V1, but net new will all go to V3 very quickly because we'll have asked the agents to do it, and the agents will be like, sure. Unlike develop unlike humans who are persnickety and recalcitrant. Isaac: Yeah. Jeffrey Sherman: And so that gives us a migration path. And then yes, it will still take a ti time, a lot of time probably, for the V one stuff to trickle down to the point where we're we can actually talk about, okay, can we get rid of this? Isaac: Yeah, and I don't know how many customers you have using it or if they even have an incentive to swap because I imagine some of them would but maybe the other ones they're not tech savvy and they set it up a long time ago and it works fine and they don't care. know, but maybe those are people if they're big enough accounts you can reach out to them and like work with them to you know get them ported over if it's like necessary to change for like a security reason issue or something like that. Otherwise, yeah just let them keep coasting. Jeffrey Sherman: So we'll say Isaac: Well, yeah, I was going say, see what what I'm eager to hear about the progress in the part two for anything else you run into as you try to complete all of the other types of requests, like the puts and posts deletes and what they when they call it done, like on your end, you know, so. Jeffrey Sherman: Yes. So stay tuned for part two in two to three weeks. That the timing doesn't align cleanly with when we record, so Isaac: Well, either way, we'll just rename the episode and then we'll mention to everybody this is the second piece. I'm sure it doesn't really matter when we actually... Jeffrey Sherman: Thank you all for listening. I'm Jeffrey Sherman. Isaac: And that is my Zygoss queue, and this is Never Rewrite.