Episode 8: AI generates code. Who carries the understanding?

Vitaly Sharovatov: Hello folks, welcome to another podcast episode for our beautiful community Beyond Quality. And in this episode, we have, as always, Anupam and Maria. And now we have a guest, Liliya, who is a head of QA at JetBrains. And in this particular episode, we will be discussing testing in the AI era, because that is exactly the research that we, I, and Liliya started on Beyond Quality. And we've got so much to discuss. So join, subscribe, like. Share everything. Hey, everyone. Hey, nice to see you here. Hello, hello. So I would like to start this podcast with the problem definition. ⁓ Lila, can you tell us the story how we agreed on doing this research? That was quite fun. I think you've just met at the bar and we were discussing usage of AI and I have... I can tell you while the developers are accelerating now, they using the AI more more. Eventually, will be natural that QA will become a bottleneck because development is producing more and one way or another we have to process it. So an error increase will cause bottlenecks. So somehow I should do something with it and this is what I'm thinking of right now. And that's how you have provided me to do the research actually. That's super cool, thank you. I think that our industry definitely has a problem that the traditional approach to doing testing after the development, I would say it's quite doomed at the moment. Because as you fairly said, developers are producing five times more code or 10 times more code now. And it is pretty much impossible to cope with this load increase. if you're doing testing after. But I would also argue that the problem isn't new. I would say that even before, ⁓ you could hear discussions like managers were saying, why is testing delaying us so much? When will you guys and girls be finished with testing? We need to release, we need to release. I think that's another symptom of the same problem that testing was at least perceived as a delay to development. And another particularly interesting idea that approaches like chaos engineering emerged and they emerged simply in my opinion, because we understood that we cannot deterministically and reliably test our systems after they are being developed. So we have to find different ways of assuring that these microservices that we have thousands of still work together. So we decided to start breaking them in a controlled manner. So it's as if we accepted that our systems were becoming black boxes of some sort. Don't you think that this ⁓ introduced that the emergence of AI simply showed that the problem that already existed is just a bigger one now? Yeah, I mean, actually I'm really triggered by Every time I hear why QA is a bottleneck or some approach by Xase, I already can spot that there is some problem in the ⁓ vision, in the common vision. Because at the end of the day, we are all precipitating in the single sub-period development cycle and we are all working on very same problem, very same product. We have a common goal of delivering it the way that our users would like it and would continue using it. And it's not like developers have done their jobs and that's it. It's like you still have to verify that everything works nicely. And yes, I agree that QA shouldn't be just in a way of texturing the results and that's it. It's not sustainable with our time. I see AI as a multiplier. This multiplier works most of the days. If your processes are doomed, if you have any issues in your processes, those would be multiplied by AI. And vice versa, everything works smoothly. this would be easier for you to integrate AI just as a new instrument there and continue working like that. Amen. So what we're seeing with the advent of AI is just an amplification of an existing process problem rather than a new problem, right? And that points to how We need to modify our processes to fix them as opposed to ⁓ looking at this. I mean, think what that also signifies is if this was a process problem and not an AI problem, we already have solutions to it. We have old solutions to it and we don't need to think of radically new solutions or those are not out of question because the transformation is quite radical. But at least if this is the assumption or the hypothesis that we're going on. If this is a, I mean, like if inspecting afterwards is a process problem and which we have very well established through, you know, through movements like shift left in the industry. So it's just a question of if we were to shift left, would this problem go away? think that would, maybe we can flip the question around and think about this from the other perspective. So if this is a problem that is ⁓ process problem, can a process change alone fix it or is there more to it? Right. I would jump in here and say that first, I would strongly agree that we cannot, at least that's how I see it, fix the problem that we have with more testing. And this is one part of my answer. So I see Cloud Code Review product brought by Anthropic for the problem they created. So essentially they... helped us starting delivering 10 times more code, now they are seeing that there is a queue in every company for code review, and they're like, okay, here's our tool for that. And I can barely see how it can fix the situation, because in my opinion, when a tester tests code or a code reviewer reviews a code change, a pull request, they have to get and they have to have a good understanding for why the code was there, what it should do and what it shouldn't do. So, Cloud Code Review as any linter, yes, I'm like not comparing Cloud Code Review with a linter, but still as any tool which has zero knowledge of the business component of the risks of the domain area, it can't possibly do a proper code review. It can find some syntactic issues. It can find some patterns issues. It can find some... issues which are common to all products, but it can't find issues which are unique to your product. So I think that the industry is still trying to fix the waste in terms of cues on testing and code review with generating more tests or with writing more tests. And I think you're right on the point that we need to answer the question how we can fix this problem earlier. Because from the little law perspective, the throughput of the whole system is defined and limited and constrained by the slowest link in the system, right? So the more we produce, the more queues we get. We can easily see that on any traffic light. The road speed, overall throughput depends on the traffic light or on the traffic jam, not on the speed of cars. So the industry is still moving towards generating more tests. And I think we need to stop. or maybe to do two things in parallel. Yes, generate more tests and find the value in them, but now understand what we can take from the proactive practices that Domain and others were advocating for the last 70 years. Because for instance, when I'm writing code now in a TDD manner, when I am writing tests first and my cloud writes the code then, absolutely everything is fine. Because I... encode my learnings and knowledge of the system in the tests and plot code can just simply go and implement the code. But this approach doesn't give me what marketers love to market as 10x engineer results. This gives me 1.5 speed of my coding, of my writing. So this maybe won't cut it. So then the next question is, how do we work with agents so that the learning process that we had implicitly working when we were developing code is now given to the agents. That is, in my opinion, the hardest part here. But I actually want to take a step back, because I read through the research and the discussions that both Vitli and Lilia had. think the broader, I mean, like we've so far focused the discussion on a process aspect in terms of how we have shifted a bottleneck or rather the processes bottleneck has shifted and therefore earlier ⁓ the process of generating code was much slower and now the process of verification afterwards ⁓ is going, mean the volume of code that's being generated upfront is so large that this is no longer feasible, the verification afterwards. But let's touch upon other aspects, right? I mean, I think you touched upon several aspects. In fact, ⁓ the research scope right now is quite explorative and it's looking at how the software industry in general, but testing and quality engineering in particular is being affected by AI. So what are the other ways in which we are seeing some of these impacts? Well, to me, the undervalued or under scrutinized aspect of software development that is hugely affected by AI is comprehension and learning. Cause if you remember, there are testers on LinkedIn who are very, and in our sphere, who are very against treating testing as just an appraisal activity. And I understand them because what they're saying is that testing produces learning. This is very hard to understand until you get to a certain level of experience where you actually understand that When you're testing something, you're learning something. So this is like a meta level. Okay, so I've tested this module, I've tested this system, and I've learned more and more about it. I then went and collaborated with you Anupam, with you Maria, with you Lilia. We improved something in the system. So our mental model of what we are building is improving. This is implicit pedagogy or implicit learning. I'm not even sure how to call it properly. But when we were writing code and testing and doing everything, with our hands, we were learning all the time about what we are building. This is, think, one of the reasons why Agile emerged as the concept, as the approach. We agreed that upfront plans are unfeasible. Huge upfront plans are unfeasible, that we will have to learn through iterations, through constant delivery. So when we outsource our production of code and then testing to AI, I believe we are losing the learning. And in my opinion, this is very, very similar to what I have like, had I like 20 years ago when I outsourced my first products, projects to build to an outsourcing company. I didn't know what they were building. So this I didn't know or this that I'm having now when I, when I asked Claude to build something from scratch, I don't know what it is building. And this impacts testing and development a lot. Because if I don't know what they're building, it's a complete black box for me. So I cannot trust their tests. If you employ a bad outsourcing company, can you trust their results? You know they are bad. You know that the only thing they need from you is money and that they can disappear. Can you trust their code? Maybe you can, maybe you can't. There is no way to tell, right? It's a complete black box. So this is the problem. We are losing comprehension of what we're doing and how to... I don't know, Lilia, what do you think about this? I have to actually advocate for the L a bit about it because, well, ⁓ you could make a difference between an AI agent developing something for you and an outsource company. In case of AI agents, nowadays we have more evaluated approaches, elaborated approaches to development and you could use AI browser as a companion than just a junior or another developer who does job for you. And if you are prototyping with them, if you are designing that, if you are doing research and then you still pre-arm what is there, what are the results and still it's you who are making the decisions, then you are still good. You are just advancing yourself. You are bringing up your mind from browser, infinite Google links, and so on and so forth. You are speeding up yourselves here. But then at the end of the day, the design is done by you, the assistance of AI, and the limitations that could be done by you or under your supervision. And this is a bit different. You still get to keep this context and you still know what this system does and why. And this would enhance a little with the entire picture. Another thing that I think could improve this entire situation is feeding the AI agent a bit more of the context so that it uses in pool everything that you are actually producing. So, for example, when you have NVR waking back to the process of course, when you are having a review of the pool request review or you have your code base with commit messages, those are information bits as well. You could set for the agent the pool request reviews history. or you could fill it with commit messages, or you could provide it as an entire access to your bug tracking system. Or if you are doing root course analysis, which is also part of the learning loop, then you could also add the context there, and it could work for you. If... you are doing all those things that you were doing before AI, so you kind of gained the base. But if not, then you are in trouble right now and you're taking back to the initial discussion. Yeah, I can see the problem here. I mean, I totally agree that we need to evaluate if it is possible to make learning explicit. Because if it is possible to make learning explicit, then we can feed this information to an LLM and its 1 million tokens context window is more than enough for many and many and many decisions to be made. However, I agree with you 100 % that If learning wasn't managed properly in a company, even implicitly, if people didn't have time for do to do pairing, TDD and other, other ⁓ wasteful in quotes, approaches that some managers believe are wasteful because you're paying too much for doing one thing, then these processes, I just don't see how they can be fixed. ⁓ Cause when we are doing development again, ourselves, we are learning implicitly. Now, if we are doing this work with AI agents or outsourced to AI agents, we need to make learning process explicit. And that's, in my opinion, a very hard problem to solve. Yeah, you're highlighting a very good point here. And what I see on the projects in the teams that I'm working with right now is the growing tension between ⁓ like being a tester. between a developer team, development team and ⁓ a product ownership team because developers, they're leveraging more. They have the tools to ship more code within a time unit. And then the project ownership team, they have the expectations that ⁓ more stuff is going to be delivered sooner. and then we in the team testing teams, We are kind of in between because we have the same tools kind of, we also have AI capabilities, but then when we go and ask development teams, how does it work? Could you please explain it? How does it work? Because I need to know it to properly test it, right? And then while we don't have these answers, testing is going to be delayed. testing is going to drop in quality and that's the growing problem and tension between the teams that is not being resolved just by injecting one more tool. ⁓ And I think we need to ⁓ kind of... ⁓ make the change in the whole process and rethink how ⁓ we inject testing sooner and later and throughout the whole cycle and maybe ⁓ change the way we think about testing. Maybe doing pair programming. Maybe? I don't know. How do you see on this kind of problem in your organization and if you see the same trends? Just yesterday I read this really interesting paper and it touches upon actually the topic that we are all commonly discussing now, which is, mean, Mithili, you spoke about comprehension and the loss of comprehension. So right now there are like two competing terms in the industry to capture this called comprehension debt and cognitive debt. think cognitive debt is winning out because the paper used cognitive debt. And it also, it actually introduces a triangle of debt when it comes to software development. The first one being technical debt, which we are all familiar with. It's an old term. It's an old problem. And it's a problem that we've known, mean, that known solutions ⁓ exist, right? to which known solutions exist. Then, add it to technical debt, that is cognitive debt and intent debt. Let's try to break those things apart. Technical debt is a result of using shortcuts or bypassing proper processes in order to write your code. And it can be used strategically to get something in front of customers earlier and get feedback on it. But at the same time, it needs to be paid back. Otherwise, it slows you down with time. And cognitive debt, mean, sorry, technical debt was something that development teams owned and they fixed and they maintained. Engineering leaders were ⁓ mandated with making sure that they could always be fast and they kept their technical debt levels low, right? Now, when comes to cognitive debt, that's what Maria said is very interesting because ⁓ as testers and quality engineers, your understanding of the system is really, really important. for you to be able to test it properly because it's actually not feasible to test every single tiny behavior of a system. You have to test those behaviors that matter. And your ability to do that testing depends entirely on your understanding of the system. The better you understand your system is the more you know where the risks lie and the more you know where your tests should catch these risks. So we as a profession are directly affected by the rise of cognitive debt. And then comes intent debt. Intent debt is what is the system for? What is its intention with which it was built? This is something that product owns. And right now we are seeing a pileup of cognitive debt, but in the future this could lead to intent debt when we no longer understand our system as a whole because there's lots of AI generated code in it. And we have not invested enough effort into cognition and comprehension. So it's very interesting to see this triangle of debt play out because technical debt is actually, you can actually tackle technical debt with AI generated code, but you might end up generating the other two kinds of debts, which are even more dangerous, arguably. I totally agree with this. I know that when I'm prototyping something with Claude, this comes back to what Lili was saying about injecting more information. And I'm very far from fine tuning models just yet, but I understand that if I did something with Claude and gave it to our team, And next week, can't remember what I was doing there. That's obvious. But then I tried keeping records, almost like ADRs, almost like discussion records, what I was discussing with Claude, how we approached this problem, what were the solutions we were choosing from, why we chose this and that. This helps tremendously, not only for AI, but also for me. So for Yai, it became kind of soft constraints for further changes. So now when I asked Claude to go and change something in that little product I built, it tells me, ⁓ by the way, we had this decision made there, we should stick to it. Good. But what you're saying about the intense debt, I think this is even harder to measure and think about. And this is, I believe, one of the biggest problems in our industry that we tend to... not even look in the areas which are hard to measure, hard to see. Even with tax debt, for instance, the most common processes are to have some fixed budget for tax debt weekly, right? Like 10, 20 % of time just spend there. So the teams are not trusted to go and fix the most critical parts of the tax debt when needed. So that is why the fixed budget is given. So with softer things like comprehension debt and or cognitive debt. and intent that I believe the situation will be even worse. And what also is added to this complexity is that, for instance, when I am building something in Claude, I try to generate skills from the processes I polish. So not only I do development with Claude, but I also have and create skills for myself to help me in this development. And these skills are untestable because they are just natural language instructions. So the more we rely on AI, the less I see ways for which you really advocate. Like, yes, we can repay some of the debt, comprehension debt, decision making, understanding debt with AI, but I cannot see how we can replace our implicit learning with this. I just can't at this moment. I think that maybe one of the ways would be to have an observer, someone, an LLM, an agent. seeing what I am doing with Claude and being this team lead of some sort for me, but holding the comprehension. I'm not sure. Well, actually, think this is not a question of choosing something. We could view the system where we use all those approaches and combine all those approaches. So I'm not stating that we should just minimize the flow, but also, of course, this is what sparkled me when Maria was saying that there is a growing tension between QAs and the product owner team. So the thing is that you're still doing the information analysis, and you still got to learn somehow what exactly you are testing before you test it. And the question is, how do you do that? And here we could also involve AI. could ask it to create a context to analyze a full request or code changes and combine it with some other knowledge that we already store in our system one way or another to produce some kind of summary that you cannot get from the developers, that you still have to get it. And after that, there are two ways. You could still pre-reply it after all with the developers, or at least you could use it as a helping thing for yourself when you are testing. And the other thing that I was thinking of when Maria was saying that, well, There is a development process, a designing process, when a developer designs or develops something with an agent. And we could think of a of pair programming. But what if we have applied this concept of three-amigas to this situation and said that, okay, there is not only developer and AI agent, there is also a PUA. who is one way or another bringing their vision to the project, to the product. And one way or another, we are doing these requirements testing at the very beginning of the process. We could, of course, eventually we could make a skew out of it. I'm not sure, maybe it's more generic enough to do so. But we could start from these three Amiga's approach, but with AI. What I like about this approach is that even as a developer, once the AI generated code is there and you're tempted to go on further because it works, are by having this approach of pairing with someone, you have to explain to the other person as to why it works. And that actually tests a lot of the assumptions that you make implicitly and breaks down illusions of understanding. And I can speak through this through my own experience when I had to present my own code in a workshop. I realized that I didn't understand a lot of it. So I had to go down deeper and figure out why this is there and that is there. All of stuff that I thought I had understood, but actually hadn't. Right. So I think that's, pairing is definitely promising there. But there's one aspect of this discussion that still makes me uncomfortable. And ⁓ that is that we are trying to, mean, what we are doing is we are encapsulating experience into words and prompts and context. So context is. I mean, at least when it comes to LLM's context is English language and the ability of words to express feelings is, or experience is very, very limited. And we know this already. mean, so many years of ⁓ literature that come to prove that, you know, your literature is only ⁓ a shadow of your experience as a person. Now we are using this experiential learning implicitly when we are building software all the time. And we are trying to replace and substitute that experiential learning through text based documentation. Now that is what makes me a little very uneasy, but where I'm optimistic is when we have seen emergent properties occur in systems, we've always found a solution as to how to solve that. And the solution is unknown. I maybe we'll find a new way in how, like Lilia said, how to balance out these different aspects and still find a solution. So. Yeah, that's the tension I have in my own head. I have a hunch, very, so what Lida just said started the whole ⁓ set of parallel processes in my head. So basically what Lida said, think is amazing. If a developer pairs with a human being, with a tester, with a product owner, with anyone while using the agent, then the learning is still there. Yes, we are giving more to the compiler, to the computer to do, but the learning is captured implicitly. So maybe, just maybe, and this is something I want to think later more about, maybe the way how to use AI is not solo, but rather if I am your tester and you are my developer, maybe we should be sitting together. just doing one thing together in a pair, asking Claude on one machine what to do. This, like very recently in my marketing team, I start helping other marketers build their agents to like be a personal assistant, do various kinds of activities. And I specifically designed the process absolutely intuitively this way so that we sit together on a call, they open up Claude and we think together with Claude how to implement agents for them. This could work for the software development as well, for sure. So we were pairing on thinking on the whys, on the intent and on the hows, on the like how it's actually gonna be done. So while we were doing that, if it was software, we could design a set of tests straight away. And I could tell what edge cases and negative cases I would instantly see. Because what my colleagues are saying to me, what value they get from me guiding them and mentoring them and helping them and pairing with them is that, I quote, I have never thought of a constraint like that. So this piece of feedback shows that my engineering experience, which was quite extensive, helped them do it better. So, and again, when I'm speaking to Berna, she's helping me understand her peculiarities of the process better. So it's not that I give her the code and she says, nah, but it's rather we work together on it. Maybe that would be the way to solve all the problems here. Yeah, actually we ran a few experiments ⁓ when we did pair programming sessions involving a developer and a tester, and we gave really good results. And the constraint here right now is that Usually the ratio of development to testing ratio, personal, is quite different. It's like five to one, 10 to one, whatever, but the tester is only one usually in the team. yeah, you see. So the results ⁓ of these interactions were amazing. Like code wise, communication wise, knowledge wise, everything was perfect. But the tester can't be everywhere because the development team is leveraging a lot of at the same time. So maybe that's the future, like to have ⁓ another dev to test ratio. I don't know, but the results of this tiny experiment says that maybe we're going in this direction. And I recently heard from one of my speakers at a meetup, he's preparing a talk where he argues that we will have in the future QA first development so that QAs and testers actually will drive developers in what they're doing. That's a very interesting approach as well, because testers can build constraints or guardrails that we literally spoke with you a lot about. that then help developers produce better code. It kind of also echoes what you said about test-driven development being ⁓ what you see as a ⁓ means to redirect your AI agents into building something that is understandable and also tractable from a quality perspective. I'm imagining myself going to a pairing session and working with this tool. What I really enjoy with this tool is the speed it gives me. you know, the flow that the speed produces, I guess it will require like an equal measure of discipline to go slow, but actually, I mean, go slow in the moment, but go fast overall, because the speed comes at a risk. It's almost like you remove the brakes from a car and you drive it on the highway. It's fun to drive it fast. But ⁓ the fact that the brakes are missing is something that you will only experience a little later. Yeah. Lilia, what guardrails do you already have in your teams? ⁓ that's good question. I would say that certainly everyone is reviewing the pull request, it's... Again, it was there before, you cannot just accept a pull request without reviewing it, and that is there still, we cannot just accept anything, even though it's generated by AI. This is for sure. Another thing that is actually helping us there is that we have a concept of code owners. So we have a huge code base and it's split into a kind of subsystems, you call it. So for example, you have certain parts of compiler or certain parts of ⁓ build tooling for you. For example, some part of the code base ⁓ is responsible for building the project. this Gradle. Another part of the codeways is it's possible for parsing your code in code correctly. So in the end of the day those are, let's say, subsystems and every subsystem has their own owner, which is responsible for the code qualities in the end of the day. So this is kind of long-term process, but still if you notice a degradation there, ⁓ which is obviously would be visible through our analytics of our relatives, then we would see that something is off and for example the co-owner is not doing their job properly or when they are missing some important aspects. probably with a G-rated code or something else and we have to look closer to them. But what bothers me here is that it's obviously a very long feedback loop and we would like to have it shorter, so for example having some quicker metrics like code coverage which is changing from depending on one or another border grass and saw. yeah, I think that is the source or the biggest guardrails. And of course, our developers are not rushed to deliver the features. So I think that this might be an issue that is again amplified by AI in other companies that you have key PIs. who are doing features. So of course, you want to deliver more and more, you don't care that the months later, you have to fix all the issues which AI has injected in your code base. And a year later, you have to deal with completely undone architecture. But in our case, we are not rushing to the delivery. So. Really, developers can take their time and design features properly. this is very ⁓ specific of the way we are hiring engineers also comes. So they are really passionate engineers. They are curious about things and they want to do fun stuff. They want to design things. They want to make it nice. So they would not negate everything to AI. At end of the day, the features, because it's interesting thing. So they are balancing it with AI here. But that is not true for Buckethead still. So kind of really balanced here. That's very interesting what you've said. This kind of... brings me back to this topic of untangible soft aspects, invisible aspects of software development or overlooked aspects of software development. Hiring people who care for professional ethics, building processes without incentives where people are not incentivized as only factory to produce more machines of some sort and put them on the market regardless of their quality. Suddenly, This starts influencing the outcomes a lot, right? This is so interesting. And the opposite is also true. Now, if you replace some people who brought that, you know, that implicit and I would almost say intangible aspect, right? To your development environment, to your development culture and your team culture. And now you replace them with AI because you view them as someone who's doing a task. only that task, which is tangible. Now you also have a loss of that, right? Like, I mean, think the inverse is true. And I think the inverse is even more worth paying attention to as companies now are looking to shortcut paying wages and, you know, in lieu of having some AI licenses or AI generated coding. Yeah, that's very true. Unfortunately that our industry, as I would say pretty much any other is influenced by markets a lot. Well, we eat from this market, we earn from this market. And of course we are influenced by the markets, by the AI hype that pretty much every company wants now to put a marketing label. are AI first and we have 10x our software developers. are now producing. and ⁓ they're consuming so many tokens now. This was very fun, like a KPI for consuming tokens. I was laughing at it severely. ⁓ But what I wanted to come back here to is I think we all today discussed a lot the psychological, maybe it's not the word, like soft aspects of what we're ⁓ Understanding the intent. The why, the importance of this understanding will be growing with time, because producing code at scale become pretty much free. I mean, I can spawn five cloud code with 10 sub-agents each and it will go and build something. So it's almost free, but understanding how it's building it and guard railing it and constraining it, and then understanding why it's building it, this is important. And this is just for me working with Claude. If I am on the team, we need processes. ethics, incentives, everything combined so that I am not rushing. So if you're on a pump, come and talk and try talking to me like Vitaly, what are you building? I'm like, I know on a pump, don't want to talk to you right now. My cloud is working. I'm giving him another task. If we have this tension, well, things will break. The comprehension loop, learning loop, cognitive loop will break. Sharing knowledge process will break. The intent. will dissolve and disappear with time. So all these aspects are now being put in the focus of our industry. And what I believe is very important for us in this particular research, very important for me in this particular research, is to get to the properly deep level of why this all is important. Try explaining it maybe with numbers if I can. Try showing it, try proofing. it to the industry and to the managers that these aspects are now of the utter importance. And if we don't pay attention to them, we're just going to lose it. I've seen it in my little projects that I'm losing an understanding and it's easier for me to rewrite it now. But rewriting it will yield a different project because I've lost the intent. I've lost the understanding of what was supposed to this project to do. So this is very interesting. We need, I think, to show people the importance of the aspects we're just discussing now. This is really important thing and I believe that this for some portion of people this would be really important to see the numbers and see the estimations and this is what I like about the recent papers that Microsoft has released. So they were applying code for pilots agent in their .NET repositories. and they have analyzed how they did go for them over the year. So this is quite a fresh paper that counts their experience till 22nd of March. And they have analyzed how many pull requests has the agent brought and how many pull requests were then merged. say, if those were left there or those were reordered, it's also worse. By going deeper, they have also counted, say, amount of tokens, which were broadspent. And this is where we are coming to the economic side here. Like, okay, you have increased your productivity, probably you have fixed your technical debt, but at what cost? Did it worth it? Did it worth yet another engineer's wage to fix the spec log? And did it bring the corresponding value to the business after all? And the side aspect is that if you are more curious, if you are still spending those tokens and while meeting even KPIs to spend the tokens, then... Are you just once or twice because after that you have to play more for the support of this code or for the rewriting of this code or the re-complicating or for the extended time of your engineers trying to realize, okay, what do they have now at their code base and so on and so forth. So this Microsoft's paper is really helpful in getting the quantitative data for further research, but also, course, we need to angle it with the economics of testing and also economics of software engineering. Yep, these are some of the next steps we're gonna work on. Dear audience, I do suggest you go and check the research. I will link it to the YouTube video. Dear friends, what have we got to discuss? And let's wrap up. Yeah, thank you very much. Yeah, it's it's a, it's an interesting time. mean, I'm really excited by all the developments that are happening, but I know that this will cause major disruption and not everybody will land out on the positive side. So I think we all need to be prepared just as we are excited about trying out this new technology. Yeah, exciting times ahead of us. So I'm looking forward for the change and I'm really happy to be part of this change. And I, ⁓ And I think that your research is going to make this progress ⁓ better and ⁓ kind of give us some inputs to think about and apply on our products. Well, thank you very much for this discussion and see you later. Thank you, everyone. See you at the next episode. Liliya, thank you very much for coming. I do appreciate and let's see each other in the research. Thank you. Thank you for having me here. Hello. ⁓ you