TRANSCRIPT 🔗
speaker-0: We had shipped a version of codex which had the current date in the starting part of the system prompt and then every night it would page us because suddenly the cash levels dropped.
speaker-1: Let's just say GPT seven comes out tomorrow. Can you just drop that right into the new codex and expect things to go well?
speaker-0: The idea of currently the code review is essentially making sure that you are sending a PR for code review only after Codex agrees with you. I mean until like GPT 5, Codex used to use a different model for code review compared to the one that used to generate code.
speaker-1: Welcome to the show, so excited to have you on. Can you tell us a little bit about yourself?
speaker-0: Thanks for having me, Ajay. glad to do this. I'm Kiriti. I am the currently the CTO at Level Up Labs. previously I have been working as like a very early member of the Codex team at OpenEI. Codex is now I guess like very household name at this point where it's not just a coding agent, but it has been basically the agent to like get things done. it's kind of like re rebranded into Chat GPT now. Before this, I've spent like a decade working at companies like Google, Databricks, Samsung, primarily building AI infrastructure as well as like applied AI. And yeah, super excited to do this.
speaker-1: Thanks for coming on. My goal is to get pretty deep into the technical side of things this episode. So if you guys are interested in AI engineering, harness engineering, prompt engineering, loop engineering, all that fun stuff, this is for you. So I want to start by quickly having you define kind of the difference between prompt engineering and harness engineering and how you look at those two things differently while you're building something like Codex.
speaker-0: Awesome. Yeah, that's a very important question. And I also feel like it's very commonly misunderstood, especially because of like so many articles coming about like different kinds of engineering and people who haven't even probably built the systems like, you know, kind of like trying to write about this. And I s I feel it's like good to talk about that. if you if you take essentially take a look at all the systems that are being built around AI, the core foundational piece is the LLM, which is essentially take tokens in and tokens out. it Initially, like you when Charge GPT was launched around to 2023, it was essentially text tokens, but then over the time it has improved in terms of modality and stuff. But at the end of the day, it is going to take some information in, a process it, and like, you know, give you the answer regarding that. Now, earlier when the capabilities of the models were not that amazing, like the ones that we have right now, it was sufficient to like kind of like prompt it and treat this as like a one-shot system where you would send something in and then like get an answer. but as the models started becoming more capable, you could do more things by essentially like you know giving the model capabilities like tool calls, giving it capabilities like different kinds of like you know, prompt compaction and things like that. So prompt engineering was a time where you would like engineer how to send the prompt in a certain manner to get the results that you want. But the models have become more generalized and being able to carry this conversation longer. you could actually define like higher level goals for the system. And then the hardness is the layer that you build, the software layer that you build around the model so that you are not redirecting, like, you know, hand holding the model to like each and every step of the system. You're kind of trying to offload the orchestration of like, you know, different environments and different tools. And how do you build that in an efficient way so that the model can keep like working on larger tasks? And that's what essentially hardness engineering is, is being able to make sure that you are extracting the fullest capabilities of the models that we have to be able to solve the problem. And then like it depends on like what kind of domain you're building for. So the harness engineering is mainly about not just understanding how do you prompt the model, but all the all the scaffolding around it so that you can keep innovating in terms of like longer and longer tasks. You see all these benchmarks written about Models being able to work for like one day, two days on like a particular task, and it's not possible by just giving a single prompt. It is only possible by building all of this structure around it.
speaker-1: So can you tell me the I guess if you had to break down a harness into a couple components, what would those components be?
speaker-0: Yeah. So for example, if you're talking about building a system for customer support, let's say. initially you could you know, when like a rag was just getting popular, the idea was that you as a company had a lot of policies, documents, and then customers would need to know like, okay, can I return my product in like 10 days or something? And it was always possible to look up the policy, use the right set of chunk to provide the context to the LLM and answer it. And in one or two conversations, it's done and like the conversation ends and you start a new conversation. But now then the models are becoming more capable. You can carry out extreme list of functionalities by talking to the model. So the model can do like checking up your orders for you, it can you know issue refunds for you, or it can also like, you know, kind of look up previous experiences from your purchases and suggest more products for you. And like there's a bunch of possibilities, which means the context That you chat with the model starts getting longer. And whenever you have this long conversation, for example, LLMs have a limited context window, which means you need to be able to understand how do you compact this information so that you can able to
Load full transcript
carry this conversation for longer. So now we have arrived at like one of the important parts of harness, which is compaction in the sense that how do you continue the conversation for a longer time? And then the second part could be you know, like when you're building this coding agents like codex, you want people to be like, you know, more safe and secure in terms of what actions the LLMs can take, which means you need to be able to approve what kind of tools are executable. And you also need to be making sure that the model is not taking any dangerous actions, which means you need like a nice sandbox so that any kind of actions the model is taking is not impacting without your permission. And then there's like a whole different layers of memory, different layers of like How do you do tools and how do you support skills and plugins and all of these things? So as the capabilities increase, the layer around this should become interoperable so that you're able to like, you know, do more with the harness. So those are the kind of components I would put into the harness engineering bucket.
speaker-1: Who do you think needs to know about harness engineering? Because it's obvious people building things like Cloud Code, Codecs, sure. But if you're an average person building, let's just say an LLM wrapper, to put it simply, are you building your own harness or are you using someone else's harness out of the box or how does that look?
speaker-0: Yeah, that's a very good question. I think there's this notion that hardness engineering is this arcane probably field where only some people might need to know. I would think of it as, you know, how you would do a computer science degree in college and you would take like systems and operating systems, like a course on operating systems. not everyone who graduates computer science is going to build an operating system like Windows or Mac OS or something like that. But then there are extremely beautiful concepts over there. That it is really important to know about different algorithms, different kinds of optimizations that you could be applying in like a completely unregulated scenarios. And I would treat the harness engineering concept as very similar to that, is understanding how this gets done in the right manner really lets you customize your application and kind of like have develop an eye for looking at the problems and thinking of the solutions that needs to be implemented for this. And again, like harness engineering is not the discipline where you start building things from scratch. A lot of the companies are actively tweaking different existing open source harnesses for their own domains. For example, companies like Glean, they are like an enterprise AI assistant and they often deal with like in insane number of systems when they are like, you know, kind of like building company knowledge. So they have like up customizations and they're right about this in terms of how do they do tool search, how do they do tool calling. Orchestration and code mode execution is another thing where codecs and cloud code kind of like defined how do you effectively string together multiple tools to make sure you're s saving the context window. And this is a very common technique that can be implemented for many other systems that are that have nothing to do with coding, for example. So learning from these principles and applying them to your domain is like extremely important. And that is what I think like you know summarizes hardness engineering. It it really kind of has this like crucial building components and I think everybody should be everybody who's like serious about building these applications should be actually looking at those and like learning more about that.
speaker-1: Okay, following up on this, I'm curious. I can't remember exactly what benchmark it was. It may have been Arc H E I, but I wanna say when OpenAI released the public benchmarks, they released a second benchmark for Astra using a custom harness. And it was like a massive step up from using the standard harness. I'm curious how much you think the harness actually affects the performance of the model. and if you think there's value in people like making their own harnesses or having many different harnesses for many different tasks, because clearly In this case it was a huge benefit for that benchmark.
speaker-0: Yeah, I think harness essentially, if you think of it as the layer that guides the model to do a task, it controls kind of like everything that's going into the model. And the key reason for that, like, you know, jump in RKJ perform benchmark performance is that the standard harness was kind of like missing, giving it like the thought process or like the chain of thought which was used to arrive until up to that point. So the model was kind of frequently, you know, kind of like getting instances of amnesia and it was kind of forgetting what its chain of thought was. And then like when the harness was actually improved to kind of like package all of that information along with the along with the task, it kind of improved the performance on. This is like a again like a very particular use case for designing your own harness but a lot of common world like you know like scenarios where you want to Orchestrate the model in a specific manner, which is useful for your domain. I think hardest engineering is really crucial to get that output out of that. So for example, let's take a very maybe like a medical use case where you're trying to kind of like diagnose diagnose basically like you know what kind of like follow-up steps that a patient needs to take based on like the conversation with the doctor. as long as you have availability in terms of like these systems of record and the list of steps that need to be taken, any general agent would probably like, you know, do a decent job in recommending this. But if you're building a system in this area, you want to be more opinionated in
the sense that you want to validate the steps recommended against some set of guidelines, which means you're trying to define your own evaluator or like your own rubric, and you're also trying to define whenever like a long conversation happens. what are the set of information that I can keep in context or I I must keep in context. So you might need to define your custom compaction compared to like what coding agents would generally do because they are general purpose agents. So there are these like you know set of h tweaks that you need to do to extract the best performance out of the models. And I think companies are like only now like, you know, generating or like publishing content around this. But actively like many companies are tweaking different layers of the harness to be able to like solve this. So it is very crucial to understand your domain and understand what's happening in the harness to be able to like get the most out of model.
speaker-1: Okay, before I move on from this, one thing I want to touch on is compaction itself because a lot of people think it's like this magical thing. And I'd love to hear your like simple explanation in how compaction works in a coding agent like Codex, for example.
speaker-0: yeah. So compaction, you know, like taking a step back is as you keep talking to the LLM, for example, like let's take Codex and you ask it to analyze your like last year's financial statements, for example. then the model does a bunch of tool calls. Like it opens say PDF files, reads different segments of it, and then understands like you know how what to do next, and then it keeps like calling like say like 20 tools. All of these tools for every model call. keep getting accumulated. Think of it as like a record that keeps like adding new lines to this. And then whenever you kind of like send a new call to the model, all the previous information gets stored there. And as you keep conversing to the model, you're kind of like filling up the context window. Think of like a bucket getting filled up with water. And you want to basically remember what is like the important list of information from the current context window. So that you're able to continue and make more space for like the newer conversation or like newer set of tool calls. So that's the idea of compaction. And then the way compaction happens in coding agents, like any generic coding agent, is give that current context window to another LLM and ask it to summarize what are the lists of information that I can keep track of, what was my goal, what is my objective, how far have I reached, and anything that I need to remember. Like a generic s prompt that is used for compaction, and that is what kind of like you summarizes like a 256k context window into something like a 20k or like 15k context window. And then you get all the new space. But in codex, we have a different compaction strategy which is happening on the server side, which is more of a like like there's no control in terms of you don't need to define exactly what needs to be compacted. There's a separate model that is trained to compact and come up with like a latent representation. that effectively, you know, like turned out to be like way more impactful in terms of retaining the information but also consuming less space. So this compaction b works by providing the current context printer to the OpenAI responses API. It gives you back like a token, a token which you sent for any further conversations downstream. And then like the compaction is like working like in a very seamless way.
speaker-1: So if I'm building on top of the open AI API, like you handle compaction for me. Yep. That's pretty that's pretty nice. okay. I had one follow up on this. When you compact, is the chat history stored somewhere? So if my agent had a specific question and wanted to look back at its own history, is that available in some database I can go read from?
speaker-0: if you're talking about codecs, for example, which is like running on your local system, every tool call, every user message, every system message is kind of written into a JSON file, which essentially can be like literally a file that agent can go and read and it is available there. And even if you compact like the system of record is still on your like local JSON files. And similarly, for cloud agents, it is being tracked in the same way. And if you've been following the Astra release, one of the nicer like new methods that was released was about like how do you do compaction in a different way. Instead of doing the server-side compaction, it was going to have like these notes that are being written. And these notes can be used to actually like you know look up information when required rather than carrying on with this like, you know, reference ID. So it's like a new style of compaction. It is kind of in an experimental stage, but it is proven to be more effective in terms of like con continuing the conversations for longer.
speaker-1: Interesting. Yeah. So moving on slightly, I want to know how you go about evaluating your harness because, you know, obviously the models get better and better and better and things change. Like I remember this is an example with Claude Code, but I think after I moved from I want to say it was Opus four point seven to four point eight, like performance for me at least was like noticeably worse. And and it probably wasn't all the harness, a lot of it was like my skills, my setup and all that, but It was an interesting change. Like the model change now suddenly felt like my setup was ineffective. So I'm curious how you go about evaluating the harness and how much your harness changes model to model and what kind of testing you do around that.
speaker-0: Yeah. I I mean, to be honest, like evaluating hardnesses is like an extremely challenging problem, especially when you're building these generalized agents, which people could be using to write blogs or like you know,
actually build back end applications or like that do front end. So you kind of have to make sure that you have like the right representation of like each domain of use cases that you're going to be using the system for and keep continuously evaluate like you know, evolving these. data sets to make sure you're reflecting what you're seeing in the production. So like there are like obviously test cases that are used for model benchmarks, but even for like a codex harness benchmark, we have a lot of evaluation data sets that we constantly keep track of whenever we are making a change. Like even for like a system prompt change or even for like changes in ways we orchestrate the tools, all of these are frequently like you know made taken care of by like the evaluation But having the static evaluation data sets only tells you like half of the picture. There's a lot of instrumentation that is done to understand how the user performance is looking like. You know, like a lot of A B testing experiments keep happening whenever you're deploying a new change. And it actually really adds like valuable information, which is why we keep mentioning that like, you know, production monitoring is as important, if not more important, than having the right evals as place. yeah, there are like Cases where you have this kind of like a regression after launching a model, a good number of times it turns out to be some, you know, like a unnoticeable bug that's kept into the harness, which is either compacting it the wrong way or like, you know, destroying your prompt cache performance. So it is actually very useful to, you know, keep in touch with the community as you're launching new versions to just like get the public feedback and like, you know, kind of iterate on that as well, in addition to all the instrumentation we have.
speaker-1: Is prompt caching a function of the harness?
speaker-0: Yeah. So prompt caching is basically making sure that you are not like removing or like and changing prefixes based on like what you're trying to do next. It is really impactful to make sure like you are not touching the layer that has done before. But only if you're like, you know, if you're actually like doing compaction, then your prompt caching isn't actually going to be bad for that particular turn. But after that I think like harness plays a really important role.
speaker-1: Yeah, let's take one step back and can you briefly explain what prompt caching is, why it's important and why destroying the prefix like basically breaks everything.
speaker-0: Yeah. So I think like just like we were probably talking three minutes ago, as you're talking to the model and model is running tool calls, you have to remember that LLM is a stateless application. I mean, stateless intelligence endpoint. Essentially, once it serves your request, it forgets like what you even said before, which means whenever you are trying to follow up a conversation, you kind of add the prefix of like what happened before. And then also the current context of like what is your new question and then like get the answer based on this context. Now, if you implement this very naively, an LLM is going to process all the input tokens every time and then is going to answer the question, which means you keep keep getting charged for tokens that you're just repeating to provide the context, not just the new one. Now the prompt caching is the optimization that. You kind of store the state of the LLMs, state of the transformer for like a small period of time, like say 30 minutes, so that if the the same prefix gets matched to your earlier question, then you can just like take that stored weights and proceed from there. It's basically due to the nature of how transformers do its computation, the inference works. you're able to check checkpoint like a particular state. And the checkpoint is only useful if your prefix exactly matches whatever was done before, which is why like you're able to like you know not pay the repeated costs. So the prompt caching is essentially saving your computation state and then kind of resuming from there. So whenever in you're like you know using any models, it's important to make sure your prefixes are matching so that you're able to take advantage of the prompt caching because prompt cache tokens are at least like 10 times cheaper than like regular tokens. So you are like, you know, making advan taking advantage of that when you are like building your applications.
speaker-1: Yeah, I can't remember if I told the story on my last podcast, but I was building an application and basically I had this thing where I was storing changeable information in the system prompt. And it was at the very beginning, and my API bills like crazy. Like every single one of my customers was just using it and just nuking my wallet. And eventually I like did this complete auto and I was like, my gosh, I'm the biggest idiot. But it's like very simple concept in hindsight, like just You're better off appending. Never change the history. That's an oversimplified way to put it, but you know that's
speaker-0: No, I think it's really important and it's easy to get it wrong as well. Like we had shipped like a version of codex which had the current date in the starting part of the system prompt. Yeah. And then every night it would pages because suddenly the cash levels drop down and then like what is happening? And then it's hard to identify that. But I think yeah, it it is easy to get it wrong, but it's also very important to keep monitoring so that your cash level cash hit rates are like you know significant.
speaker-1: I wanna go I wanna talk a little bit more about evals because it's very interesting to me. and something I see a lot, let's just like go back to the community around if you guys are on X, which I'm sure most of you are, anytime there's a new release, everybody wants to throw in their two cents on the code quality of these models and what
they're doing and whatnot. So I'm curious how you're evaluating for things that are hard to measure and people might not have the same opinion on, like. code quality, architecture quality, whether or not an agent is making unnecessary changes, and kind of ultimately like the mergeability of code written by an AI. Like how do you design for these things that are like very, you know, opinionated?
speaker-0: Yeah, it is really hard to, you know, like say that there is like one benchmark which would solve for everything. So it is all about like iteratively and increase like improving the evaluation data sets we have, as well as like kind of like completely gen having like new data sets that can be kind of like guided well to understand this. I think if you for example, remember maybe GPT five point four, but it was extremely bad at like front end design. it was It like Codex has been always extremely good at like back end as well as code review. But front end was not something it was doing an okay job, but compared to the competitors, it was a bit subpar. So a lot of the a huge push during the next model releases for the 5.516 were about like improving the performance of the front end, like basically doing it at the model level. And the main idea there was to kind of like, you know, have an extensive set of like data sets that you can actually provide during the post-training phase to make sure you are having the right set of like environment configured to actually build these tasks and right set of way to capture these rewards. And I'm kind of like oversimplifying it, but essentially have like a judge, have like series of judges written in a way that it is being able to like you know evaluate the front-end performance better. And a good good reason why we had to why we could like achieve that jump was just like, you know, finding innovative ways to describe the taste and kind of like codifying that into the model and kind of also like teaching it like what are the common like, you know, pitfalls in design and like things that you don't want it to build. And that kind of like process led us to like another like a better design. And as like the models are being used for like, you know, very frequent scenarios, I feel like this fly l flywheel is going to like become better and better. You're able to glean what is actually useful and what is something that you can like, you know, keep pushing to the production. it's not going to be like one model release that's going to like solve all of the problems. It's going to be like a hill climbing approach in the public as well. And I think like that is going to be like a problem that both the engineers using this problem, using these harnesses are going to work towards, but also the model labs are going to do that as well.
speaker-1: It makes sense. I'm curious about going back like ten minutes ago, you mentioned the production evals like always on running. How do you go about detecting if something is going well or poorly, like inside a user session, for example, in real time?
speaker-0: Yeah. most of the monitoring is passive in metadata level in the sense that you need to like obviously respect all the zero data retention and all kinds of like customer information retention policies. So you there is no question of like using that data to like design evals, but it's also useful to understand what kind of like information is going, what kind of like you know issues or category of issues you're seeing. So we have like a lot of classifiers that run. Which essentially take the current context and look out for things that might be wrong. Like for example, if the user is frustrated, I'm like very simplifying it down. But like if the tool calls are kind of like you know, repeating itself without actually the model learning what's happening. And there are like numerous lists of classifiers that are running. These are like very low cost LLMs that are going to quickly give like a opinion on like you know what's happening in the context window and then We track these kind of like metrics on our observability dashboards. And that is something that we use to kind of like get a vibe of how an AP test is running for like a code review or something like that. Especially, for example, a concrete use case that came to my mind is like code review when we were shipping different models. okay, maybe take a step back. maybe not many are aware of this, but Until like CPT5 or 5. Something around that, like Codex used to use a different model for code review compared to the one that used to generate code. And it kind of like worked really well in identifying like, you know, all kinds the diving really deep and identifying what kind of bugs could happen. And it worked really well. And then over the time, like research actually did a great job in combining both these capabilities into a single model. But we were kind of skeptical in understanding the real-world performance. So the things that we had to keep track of was how many tokens are being spent per review task so that you can on aggregate level measure if you are performing worse or like better in terms of cost. How many turns is it taking, like the model turns? And all of these kind of like metadata metrics really become useful to measure this. And yeah, like then we have like dashboards that are tied to A B experiments so that we kind of like and understand whether this is like a significant causality, there's a correlation, and we use stat sick for doing this, all of our experiments for that.
speaker-1: I wanna go slightly back. you mentioned like obviously, and this is clearly the case for everyone, that you can't look at like user data or exactly what they're saying. But given that you can't, like how do you like how do you use this information in a useful way? Because as a developer, it's like the most obvious, easy thing to do is just say, like, okay, we see that these people are frustrated. Let's look at their
trace, their logs, whatever, and say, Okay, they tried to do this task, it failed. But you You can't really do that, right? So like what does your like anonymization process or how does that data get like disaggregated in a way where it it becomes useful to you to actually improve the models?
speaker-0: Yeah, I think an important functionality that we added whenever like somebody wants to provide explicit feedback is like the slash feedback command where it kind of like, okay, you're giving permission to actually look at like what is happening in that trace and then we can use that to like dig deeper to be able to assist better. That kind of functionality I think like many applications ideally, if they can, should be implementing so that you're able to get like finer level access into what's transpired in that like conversation. And even at a higher level, I think we are essentially any kind of like trace or like in a conversation kind of passes through a lot of like anonymization filters, PII filters, that you kind of like at the end of the day have like very generalized trace. And even those traces are not something that are like, you know, actively fed or anything into the model. It's more of like trying to run different kinds of like data analysis to generate these kind of like classifier metrics. And then do data analysis on top of it. So to identify like, you know, what is the mean number of turns that an agent took to solve problems for every conversation that is talking about front end. So that now you can understand whether your harness regressed on the front end or not. And typically if you are somebody is using for I don't know, let's say like like like like customer support kind of use cases, then you can kind of at least identify what domain this conversation in. And then you can also track like how many conversations are appearing in each domain so that you can see maybe your harness started doing bad in customer support kind of thing. So you're kind of like, you know, losing track, lose losing traction in that area. And then like maybe you need to like dig deeper into that. So it is still very much operated at like a metadata level. You can't really look at that. But the idea is to identify how many attributes in the metadata that you can come up with for you to like get better insights into this.
speaker-1: That makes sense. Okay. I'm one of the things I'm curious about is like, let's just say GPT seven comes out tomorrow. Can you just drop that right into the new codex and expect things to go well? Or are you kind of going ground up, like we're gonna re examine every single layer of this harness and see how it works with the new model?
speaker-0: Yeah, it's mostly the latter. like you can drop it and obviously it works. And like even when we ship like GPT six, it codex as a hardness should still continue to work with like the previous models and you cannot make like drastic changes. So we are obviously like tied in terms of the backward compatibility. But I think like there is somebody who asked me this question recently about something in the similar lines where hey, whenever I ch change the model. Does codex like know change the harness as well behind the scenes? And that's not that's not like how it generally works. Maybe there are like some system tools that we give to make the model effective, but most of the harness is still like general purpose and kind of like works for different models. But whenever you're like say building your own agent and a new model drops, I would definitely take a step back and understand whether your existing harness is kind of like restricting the capabilities of the model. Or it's like trying to put too much harness around it so that the model is not able to exercise its like you know new capabilities very well. And also kind of like understand whether the tool calls can be reorchestrated in a way so that your agent can now effectively re re effectively orchestrate these things rather than you setting very specific checkpoints, which was like previously done. So a lot of like, you know, ground up thinking in terms of and like Tweaking different layers is one one thing that we definitely do. But an advantage is these models are also trained with code XSD harness. So a lot of good information is already flowing back from the research side in terms of what works and what doesn't. So it really helps us to like, you know, make this more effective.
speaker-1: Yeah, that all makes sense. Okay, so you recently left OpenAI. I wanna hear a little bit about what you're doing now and yeah, we'll dive in from there.
speaker-0: Yeah, for sure. yeah. I have left OpenAI to start my own company and basically joining Ashwarya, my co-founder in Level Up Labs. And the key the key reason I took this decision is that there is so much intelligence in terms of model capabilities, but the adoption of AI in terms of solving the problems is is is n definitely like nowhere near the North Star. And It it has like two main issues. One is like just the education about like how do you build these systems in a better way. I think if you go outside the bubble of Silicon Valley or even the bubble of like just the Bay Area, and it's really hard to understand like whether companies can actually get value out of this, like ROI in terms of like investing and how do you better build these systems. So There's a lot of like lack of best practices and like, you know, teaching from the experience of building the systems. That is one wing where like in level up labs we kind of like have AI adoption or like AI transformation kind of audits and like services that we do where we work like really close with enterprises to understand their domains and kind of like guide and advise on what kind of capabilities that can actually speed up their processes and what what is the best way to get the most out of this. latest model releases and things like that. And the second part is also
about applied AI, like the FTE kind of work, where it is really becoming like, you know, kind of common knowledge now that you cannot really shift and lift systems built for one use case into like an entirely different use case. A lot of fundamental ground up thinking needs to be done in terms of defining your evaluations, monitoring observability tools, and like kind of compliance. All of this become really important that it's not transferable to all the companies. So we work in that space as well to kind of like guide companies on how to do this effectively. And a lot of the engagements that we do with companies have been recently about, you know, fixing the data layer, fixing how data gets exposed, how how does the model know the context about different kinds of data that's lying in the system and kind of building that unified layer of access and That is really like a good amount of work before you start seeing advantages of the agents.
speaker-1: Awesome. How is it different now, you know, working more in the application layer instead of actually being in the nitty gritty inside one of the Frontier labs?
speaker-0: Yeah, I think it's really surprising with many conversations that I have are about, you know, how do I make sure that my ROI is actually like, you know, reasonable as a business that I can invest. Like in OpenAI I had like unlimited tokens. I could just like run any number of agents to solve like different kinds of problems. I could run agents just to, you know, like merge my PR because I was too lazy to do that. But obviously, like you know, when you're stepping outside, like our tokens are no longer cheap. And you want to be able to not just estimate the ROI in the correct way, but also make sure you're using the right sized intelligence for different kinds of problems. So this has become a prime topic of conversation. And I think it's I'm also learning a lot when you're talking about like a combination of open source models and like you know, like the frontier models. Where to combine them, how to combine them, but also keep the visibility in the system high. So there are like a lot of these interesting questions that companies are dealing with. And the second part is also about just trying to learn about the domains well so that you're not just you know always jumping into fine-tuning. There's a lot of performance that can be extracted from existing foundation models. So that is also a good area of like, you know, learnings.
speaker-1: Is there a standard kind of winning architecture or combination of models that you're seeing like you know, let's just say Astra orchestrating and Luna executing, or is it all just very like case by case basis?
speaker-0: It is pretty much case by case basis, but there are some common patterns. I think like that is definitely what we are looking at, like Astra as the orchestrator and you know, like a combination of Deep Seeker, like Luna being the workhorses that are trying to do things then. But
there are cases where it actually impacts the performance in a negative way because there's too much handoff and the c the use case is actually not complex enough. In which case we feel like you know, just having Astra or soul level model the task end to end is actually helping. So it is very much problem dependent.
speaker-1: And what kind of tests are you running when you're trying to figure out what tier of intelligence to use or what combination of agents? Like how do you go about determining what the right solution is?
speaker-0: Yeah, I think the first step to identify how do you get in get like usefulness out of these kind of systems is defining what good looks like. And that is where we kind of like you know sit down with like n like subject matter experts and folks who are like actually building already have systems to do this or like they're trying to really place a system or optimize a system in a better way, kind of like really look at some representative use cases. construct like a data set that gives you a good example of like what the input, output and the intermediate toolcases should look like. And that is kind of like forms the foundation on deciding what level of intelligence you need. And actually for a good number of use cases, latency becomes extremely important, in which case you cannot really go and use the latest frontier like the high max reasoning mode. So that also kind of like guides us to find very innovative like you know solutions to keep the latency low but also keep the same level of intelligence.
speaker-1: Interesting. What percentage of code that you ship today is actually handwritten by you?
speaker-0: don't think I've written code in quite a while by hand. it has completely been like, you know, agents writing code for me and kind of like building the system around this to help the agents do this in an effective manner so that I have to like do less of the tweaking. It is it it is like you know incredible in terms of like what we are able to do with the agents. Like when I joined OpenAI, I think it was like GPT 4.1 that came out, which was nowhere compared to like the performance that we have right now and like kind of like writing code and like you know using this as like an autocomplete to write some parts of the code. And I think like for the last like two to three months, in my opinion time as well, it was yeah, it was not really the case where I was writing code. It was mainly about kind of like building systems around it to make sure it is producing like a good quality code and also being able to like, you know, do some automated reviews so that it is implementing the functionality that you desire.
speaker-1: Okay, walk me through your exact coding process if I asked you to implement a fairly large scale feature.
speaker-0: yeah, I think a good amount of you know like speed up is kind of like not shipping like like drop shipping the agent into like some entirely unfamiliar code base. There's a lot of work that gets done to make sure the environment in that code or like in that system is agent friendly, which includes like you know, kind of making sure like your agent's MD files are in place that is ex describing the intended architecture. And what kind of like tests to run to make sure like you are not disrupting like, you know, some functionality in an unintended way. And also some style guidelines in terms of how that should be done. and all of these, like, you know, ki are the first things that I would target to make the system more legible to the agent. And then in terms of like when I start implementing a new new feature or like need to build a new thing, first thing is to understand like how the existing system works. If you're not familiar with this, or if you're already familiar with this, then it's kind of makes it easy, which is where like the plan mode kind of really helps to just like quickly brainstorm with the agent on, I'm trying to do this, these are my constraints. And try to be more like, you know, vague about it in like the implementation details so that you're not giving the agent a exact solution to deal with, but more of brainstorming it like how you would do with like an e-collegue. And try to you know, like ask it to point out like, you know, examples where this could fail or like you know, coordinate cases. And then as you're like brainstorming, I think this would easily take like like a half an hour to one hour kind of like brainstorming where I'm not trying to think through different parts of the system. And once you've agreed upon this, it is good to write this out as a plan. like a plan that and again, I'm not going to like give this and plan always to the agent entirely, but a good architecture map so that I'm sending this as a code review to the team so that they're able to like, you know, they review the plan. I think this looks good or any details that the agent might have missed out. More in time you spend on making this correct, the less time you spend later down the line to actually like, you know, take step back in terms of architecture and having to redesign things. So this becomes like really critical. And once you have this, then the next case is identifying what kind of observability you have in the system to use that information and figure out like what is the right level of chunks that you can divide this work into. Like if you just say, build me the next big database, which is like most optimal, obviously it's going to come up with its own metrics and things and like you know build something very hacky. So the idea is to make sure you have like good instrumentation and then split up your plan accordingly. And then I would typically use like goal mode to goal mode in codex to basically give it very concrete goals on identifying whether it's like target has been achieved and kind of like let that run. And also a good part of my personal productivity was making these things in like a parallelizable way
and then using cloud agents so that you're able to like execute on this like in a more parallel way and kind of making sure that you are then like tying these things together down in in like whatever is like a non-merge conflict manner, and then and then like basically start from there. this is kind of like at a very high level how I operate. And I think a lot of the work goes into defining what goes into each of these phases.
speaker-1: So it sounds like you still play a pretty heavy role in like the system design side of things.
speaker-0: Yeah, I think like it it's it it's very good to make sure that you are actually thinking about like these things in a in a first principles manner because it it the answer would differ if you're like building some, you know, like a wipe coding kind of project. It's not going to have a lot of scale. But when you're building systems at scale, it is really valuable to make sure like some constraints are not violated and like your performance is like kind of hitting the same threshold. So There is like a good co engineering, if I may say, with the agents that you would like arrive at this plan.
speaker-1: That makes sense. Do you use any specific skills or do you have any specific things in your agents.md that you find are generally useful?
speaker-0: I am very much AGI pilled in the sense that I think like as the models are becoming better, they need less guidance in terms of how do you like less broad guidance in terms of, hey, break this problem down into five bullet points, write this template of design doc. And I think those are not really super helpful as you have like the current capabilities of the models. The main things that are important are defining good hooks so that the model is like not you know, like running it around itself like in loops to get a unit test passing, kind of like making sure you're delegating what is deterministic to a deterministic tool so that the intelligence is used for something that actually requires intelligence. And sub agent orchestration, I think codex does decently well, but it helps with guiding what kind of subagents to use for different kinds of purposes. that is useful as well. And I think the third important thing is to kind of like have a way to represent or like present the discussion. Like instead of me reading like walls of text, I kind of like ask it to build a prototype to show me how it would look like or like a HTML page to show me where different components are fitting and how it works. So that I'm able to like understand the output better. So these are the things like I j typically use. but there are like a lot of best practices plugins and stuff which people already develop in OpenAI from our like inch for our team and that is like also super useful for us.
speaker-1: How do you go about reviewing code, both from your end when you send someone a PR? Like are you handwriting that? Are you, you know, sending them a document with context? And what are they sending you? And kind of what does the code review process look for you now?
speaker-0: Yeah, code review is is like extremely automated and I think we at OpenAI definitely had this like you know privilege of codex being like a very good model at code review, which kind of like took the burden off in terms of you know like looking at niticity thing like nitpicking things. the idea of currently the code review is essentially making sure that you are sending a PR for code review only after codex agrees with you. Like the it's it's it's the responsibility of the author to go through loops, like author or the author's codex to go through loops to make sure the code review is satisfied. And then everything, like you know, all the tests are passing and all the P0 P1 issues are addressed, for example. And only then like a human reviewer could come into the play into the picture. And even for the human review, It's no longer about like reading every line of code. It's kind of like having an agent identify what are good things to focus on. And also we have this system internally called Flow, which kind of like summarizes each chunk of like a PR into what this chunk is trying to do. For example, if you are making a new API, it kind of shows you where it the data is coming from, what kind of transformation it's doing in that endpoint. And what kind of unit tests are written in this like higher level like you know constructs so that you can go dig deeper into one of those rather than reading code and forming that picture in your mind. So it's it's essentially like a new way of like reviewing code and you're only caring about like the intention there and the intent is being reflected in the code and not. And that essentially is like the code review process currently.
speaker-1: Are there any products on the market that do that currently externally?
speaker-0: I I think like CodeRabbit is decent. Yeah,
speaker-1: Yeah.
speaker-0: which does that. they have like a bunch of functionality in terms of like configuring what files to review and stuff like that, which I think we also recently shipped in Codex. but I think there's also one more component. It's probably yeah. Yeah. They also do a decent job. I think Code Review is I don't think it will look the same way that we are doing currently, like six months from now, I feel it's going to be like very much changing. I think the idea is to have like the right interface for code review and like what is what does it mean by actually doing a code review. And even if you like humans need to do the code review at all, should it be like a design level review? I think like this is going to be like a very interesting space.
speaker-1: Have friends working in big tech right now who are, I mean, there's no better way to put it than leadership thinks that they're reading every single line of code. And it's just not possible. I mean, with the amount of code that people are shipping right now, and to the point where leadership
is like, How are you guys like shipping so much code? Like, how are you reviewing all that? And they're like, we read quickly. And I'm I'm like, This is gonna change 100%. The way we review code, I agree with you, is the old way is done. I mean, I was even just thinking about. Literally combing through lines of code and saying, like, I don't like your variable name here. Like, come on. You know, like it's just it's it's over.
speaker-0: Yeah, I remember a lot a lot of times, like I was a very big stickler for beautiful code. I really like code that is written in an understandable way, extensible way, variables are named correctly. You're not like, you know, dumping your 300 line function over there. I think it's people think that when you're asking an agent to review the code, you are letting go of all of these things, which is not true. The idea is that you're able to codify these things. in a review guideline kind of approach so that the agents are automatically enforcing this. So if anything, I feel agents reviewing the code makes the spread of best practices way more faster than having this one guy in a team, you know, bothering everyone to do things the right way.
speaker-1: The one guy that nobody wants to send their PRs to. So how can somebody get good at the new process of writing and reviewing code in your mind?
speaker-0: yeah, I think a lot of it actually comes from doing the groundwork of making your system legible to the agent. when I say legible, it is kind of being able to for example, if you're building a full stack application, the agent should be able to spin up a container which can k bring up your application, make a bunch of API requests to that, measure what is the latency that is being like you know observed at different endpoints. And report that back to the agent so that it's able to like then optimize, do this process again. Otherwise, you are kind of like, you know, stuck being this meet proxy where the agent says, Okay, I optimize this, and then you go, okay, let me deploy a solution, measure that, tell the agent back, and it's going to be extremely inefficient. So a good amount of groundwork needs to be done to make the like agent have the right set of tools to be able to achieve the objective. I think this comes in a number of formats, like, you know, yeah, being able to spin up these test systems, being able to spin up sub-agents where they can have their own working environments if required, like either spin up new work trees or doing this in like different cloud agents. And then another way is to make sure you're spending good enough time. if you're a front-end developer, kind of like, you know, spend good enough time to codify your like design choices. or like your tricks in terms of like different skills so that users are able to use these skills and depend on them to guide on the judgment. spending more time on making sure your agents are able to move faster and unblocking them wherever you can and really having an eye for looking at those bottlenecks and unblocking them, that I think like is a really important thing that I would say like, you know, you can g get better at coding because you should always build with a mindset. that the agents and the harnesses are going to become very good at like taking on more work. So any kind of like effort that you put into streamline that is going to give you like at least 10 X rewards in the future. So you have to like keep building towards that. And honestly, like there are many you know like best principles and like kind of guidelines on how to do that. But I feel the best way to do is to learn. Sorry best way to learn is to do it. And that is when you kind of like see the rough edges of the system and form your own opinion on what works and what doesn't for your use case.
speaker-1: Completely agree. The first thing I do on any new project or when I'm spinning up a coding agent is like, okay, whatever I would give to like a junior engineer working on this, like my agent is getting that. Like if you need access to read my database or to look at my stripe sandbox or to spin up a test environment, like if it can't do that, then I'm doing it. I don't want to do it, you know. Exactly. So so
speaker-0: Your is a tool for the agent.
speaker-1: Exactly, exactly. I do not want to be a tool for the agent. Okay. Awesome. This has been great. Where can people find you and learn more from you?
speaker-0: I'm I actively write on LinkedIn and I recently started writing these on like Substack as well. these are like my two popular places where I keep like writing about it. And then also Ashwarya and I like teach a couple of courses on Maven where we are kind of like you know inculcating these best practices and how do you apply this to different systems. So our course is called like problem first approach of designing systems and It's really hard to and to like, you know, under st overstate how many times people are running behind, you know, like building multi-agent systems or like having this elaborate setup and really kind of like losing sight of the problem they're trying to solve. So it's really important for us to like whenever like a new technology like this comes up, take a step back, think from the principles, think about the problem you're trying to solve and write this. So most of my writing and like, you know, any kind of like talks that I give is focused pretty much towards this. And I feel like yeah, I would be like really glad if through this like more and more people learn about that style of thinking.
speaker-1: Awesome. I will link all of this down in the description. And you guys have a bunch of free lessons on Maven too. And I've seen a bunch of them. Yes. They're awesome. That's a if you want to get a taste of their teaching style. Good place to start. All right. Thank you very much.
speaker-0: Awesome. Thank you. Thank you so much, Archie. Yep.