The Negative 10x Engineer Will Collapse A Company

Nahshon Cook-Nelson: Last radius. The scariest engineers are not the slow ones, it's like the ones who are doing something fast and very confidently. I had someone say, like, ⁓ it seems like you're against AI, but you're also like helping people implement AI. Like, where do you stand? We're against the negative poor execution because we know what can happen. It feels like you are creating so much value so quickly. At first it looks like it works. The problem happens when suddenly realize that your passwords got leaked, your credit cards got leaked, and you don't know even why. Welcome back. To the Return on Intelligence podcast, where we're all about giving you access to practitioners who are actually helping people implement AI day to day and allowing you to get a peek behind the scenes so you can avoid making mistakes and make the most of your investment into artificial intelligence. Today I have two of our very own ML engineers here to share some things that they actually flagged to me and said, Hey, I think it's urgent that we get this on the podcast as soon as possible. One of the first items here I have on the list. is ⁓ this vanity metrics that's happening. Can you guys talk to me about that? I think I can start. So I think right now, like a lot of you know, a lot of news, a lot of LinkedIn posts, everything is talking about productivity, AI posting productivity. But I think the very interesting question that not everyone is tackling is are we measuring the output or are we measuring actually the outcomes? So from what we have seen that AI usually is not increasing engineering value, it's increasing engineering output. And sometimes those are like not the same things. If we would compare it to some of the, you know, some of the companies, some of the people evaluating performance based on the number of PRs, number of lines of code, you know, the number of closed tickets, that kind of stuff. It was very loosely connected to real productivity even before. Right now, like with AI, I would say AI snapped that link fully. So a new PR right now, it can be just an evidence that someone pressed an enter button. It has nothing too close to reality. One engineer now ships five PRs and ⁓ thousands of lines a day with no more reliable systems, no fewer incidents and no happer customers. So that's why it's very important to invest into judgment instead of the output or like lines of lines of code. I wanna play stupid for a second, just for those who are non technical like me, who are a lot of listeners to this. What exactly is a PR and and why do so many engineering teams focus on that number? Yeah, so PR is ⁓ any change in the code base, ⁓ basically. It could be a bug fix, it it could be a new feature shipped, it could be ⁓ some ⁓ extra tests added. Basically, it works like that. So any sort of code change ⁓ change is called a PR. So ⁓ before Gen AI, before Cloud Code was at scale, people were still using that as a like productivity achievement. Number of PRs you mean? Yeah. ⁓ in in a way, yeah, but ⁓ the thing is like Talius ⁓ has said ⁓ to me before, ⁓ i if you know on on what you they measure you, ⁓ you can adjust how many PRs do you ⁓ input basically and ⁓ adjust the metrics alongside of that. If your ma your productivity is measured by the count of PRs, you just can ship ⁓ a lot of PRs, a lot of small ones, and ⁓ that would be all. Like and you will be considered productive. I think I could add even like an example from from from my own experience. So like people able to gamify any, you know, any system in which they are. We have had an experience ⁓ working together on a project together with the engineers from the client. So it's basically what like two people from our side, two people from their side. So it was assumed that you have four engineers, so like the output should have been like as from the four people. And ⁓ what we felt, we felt a little bit short on the output from their side. So It was a bit of a misalignment, you know, of of contributions. And the metric which leaders decided to choose was the number of PRs. And like number of PRs, ⁓ it it showed that two engineers from the client side created like three times less PRs than like one engineer from our side. So they're okay, this is the problem. But what has happened during the next month is that they created like four times more PRs than during the previous month, which on the paper was impressive. In practice it bloated the entire code base that things stopped even working which were working before and the system went into like an into the hound downhill ⁓ regression because it's so many these small things that matter because like number of PRs it it tells nothing really. Same goes with the number of code lines. ⁓ so let's say you you would like to measure ⁓ code lines in your like company and and s and such like how how much code do average developer produces. So you can basically bloat the code instead of writing it optimally, ⁓ you could just increase lines of code and go round and round around it and ⁓ increase the code size without increasing functionality. So and that's counterproductive later on s since the code base bloats easily, easily bloats. And then ⁓ more Architectural problems come out of this ⁓ thing, and yeah, that's how it goes. I also have another recently like new new metric that ⁓ I see more and more companies like trying to add is token maxing. So token maxing is the new trend right now to use as much clawed code as possible to ⁓ for any sort of work. So ⁓ the incentive is that the more tokens you use, the more productive you are. But the thing is, like every single token. costs and you can be more productive until the certain point, of course, with ⁓ with more tokens used. But ⁓ a afterwards ⁓ at some point of equilibrium like that progress drops significantly so you are using too much tokens and not producing that much of a value with them. I'm curious, I want to talk about the cost of doing this because I think the token maxing. The thing is, it's like ⁓ people always just say, Yeah, I spent a lot, but look how much we achieved. But they're not talking about the bloat of existing products. Marketing teams, for example, is like all that that matters is sales and leads. Like it doesn't matter how many blogs or whatever you put out. And it's the same with lines of code. So I'm curious for engineers, what's the cost? What if we bloat the product? Who cares? Like, especially for people who aren't billion dollar companies. If we were ⁓ let's use an actual example from our website, we're a wine vendor, we want to make our search queries better to find wine faster amongst our millions of SKUs. Who cares if it's bloated and what's the cost there? This is a very good point. I think if you are creating a quick demo that you want just to show to one of your friends, you're totally fine to bloat it and do whatever you like if you're not deploying it anywhere. If you're talking about real life scenario and that's where most most of the businesses do operate. The problem with Bloated code basis with with spy coding in general. This is a very easy to be tricked dopamine because it feels like you are creating so much value so quickly and like twenty-four hours I have hundreds agents generating like the code. Thousands of lines are coming in. At first it looks like it works. The problem happens when first of all, when you deploy it, and then like so and then you suddenly realize that your passports got leaked, your credit cards got leaked, your ⁓ sensitive information from your clients got leaked and you don't know even why because you have no idea what is going on and where. And even if you're gonna introduce like senior experienced engineers like they're gonna bang their head into the wall while searching for like these tens of thousands of lines of code. This is the first reason because like whatever is produced, it can't be guaranteed to be reliable because basically LLMs are they have been trained on open source code. They haven't been, you know, GPT, Claude, whatever, like all of these models they haven't been trained on ⁓ closed source data with the perfect documentation, everything. They are the approximation of level of code which is right now available on internet. And this is far from perfect. If you're gonna Google, you can find people's open AI keys like whatever, whatever. So this is the first problem. The second problem is complexity of maintainability. So you are more effective in the short term, but you're really shooting yourself so much in the long term. Because when you're gonna need to add a new feature, for example, first of all, token prices start going up. And if for adding every feature the LLM has to read through the context of tens of thousands of tokens, this is already gonna be costing you like a lot of money. And then things start breaking because Changing of one thing. If you don't have a proper architecture, everything is very tightly coupled up with so many tailor points about which you are not even aware, about which LLM forgets every time because LLM's by its nature they are stateless. So basically it should have the full context of what's going on in your system every time it produces any kind of the updates. So things are gonna start falling down and you're not gonna be even aware how or like how or where. So yes, you can build something very quickly in a couple of hours, and then you can spend like three days debugging or like Trying to figure out one of the edge cases. And probably and there is a big chance that while fixing that one thing, you're gonna create five different edge cases that you don't even aware of. Yeah, so so to add up a little bit, it definitely crashes the maintainability. If you ship ⁓ a lot of lines of code, the more lines of code, the more points of failure. So you need to take this to account optimally. Like I'm not saying that minimize the amount of lines of code written, but ⁓ to have it as understandable as possible. And The other thing is to add you have to own the code. Like ownership is really, really valuable in this case. If you do not own the code, ⁓ later on when somebody asks a question about it, why why why did you do that? What was the reason behind this decision that you made how to make this ⁓ feature? And you just point out because Claude said so. What ownership is that? And regarding also about the features, what what what's wrong with having like 200 of different features and all. The problem comes like ⁓ probably, yeah, your product is ⁓ more customizable for differ different sorts of clients. If you have ⁓ multi-tenancy there, like multiple different clients that are using the system and they are using them in different ways. Eventually, what what's going to happen is more and more different bugs will be reported on each of the different feature. Some of the features even might not be usable at all. ⁓ e even though that you made them a f free years ago or something, but that client is not using that feature anymore. So it's s sitting there doing nothing and just wasting code space, basically. Your AI agency that you hire, because next we're going to talk about hiring, should be a previous software development agency. That is the conclusion I get from this. Because otherwise, you're sending someone in who is Basically, just letting Claude do everything. And either if they're a product manager or an engineer or you know, someone who has professional experience, they can understand how to maximize Claude. I was talking with Michelle, one of our product managers yesterday, and she was like saying, It's you with Claude or it's us with Claude. And it's like the Thor hammer. Like you can barely pick it up and you're like, ⁓ I have it. I have it, and I can like technically electrocute someone, but it's like we pick it up and now we can fly through the sky and we can hold you. We can take your business with us. But instead you're like, no, I have it. I don't need you. I can I can do it myself. Would you guys agree with that? Yeah. We see the these types of clients. ⁓ they usually return back after six months too to us. Yeah. To to to fi fix all the problems that they ⁓ all the hurdles that they have found on the way. And yes, Vitalius you can Yeah, I mean I totally I totally agree. Like on the point. So that's that's the exact, you know The intelligence of LLM models is not something unique anymore. So if you're gonna look into great example, like the Stan Stanford AI index report from from 2025, 2026, right now the difference between the model rank number one and l model rank number ten in terms of intelligence is like five percent. So what's your specialty? So like if everyone can use Claude and you know there is It's it's literally round rounds up everything. That's why you need those people who who can judge where to use it and how to use it. This is very important. Because if you just throw everything, then you have like literally a copy of ChatGPT or Clot or Gemini, which anyone can use like for free. That's why it's so important to think about what kind of business edge you have as a company, whether this is your proprietary data, whether whether this is this are some of your project ⁓ some of your procedures that you want to automate. And like you need people, you need proper engineers to realize that. You don't need ⁓ AI a one list who're gonna sell you like an you know an an API for that. You can you can do that absolutely like yourself. I mean unless if you're not like super technical, you want some some guidance like for sure. But For proper procedures you need engineering. AI didn't solve these kind of problems. And it's a big question whether it's gonna be able to solve these kind of problems like in the future, by the ambiguity of the world of the businesses where a lot of you know data isn't structured, data is scattered all across, like everywhere. You need people to reason about the architecture because AI is giving you a toy and is not telling you that this is a toy. You need people to realize that this is a toy and they're they're gonna be need to be adjusted. Because if you blindly trust, it's not the question whether you're gonna pay. The question is gonna be when you're gonna pay and how painful is gonna be that paycheck. If you want to figure it out and cover every time, this is definitely up to the people. But in general, you need people who have been in the market for quite a while who have built things before this, because this is the fun. I think this is a very interesting time to be now it's like AI engineers. Before that we were called the machine learning engineers. Before that we were like, you know, data scientist. The interesting thing about being in the market that hype is being changed, the names are being changed, but when you look under the hood, principles are Absolutely like the same. People are thinking about the revolution, about you know, this is something that has never happened before. It's literally the same. It's the same as it was five or ten years ago. It's just some new shiny features, new types of the models, but the principles aren't there. Like coding is still like absolutely the same as it was before. And things which were important before, they're still important right now. Even if AI can mimic some parts of it, you you should you should grasp, you should understand where the pain points are because AI are are not gonna find your pain points. Wow. I want to talk about this ⁓ 10x engineer's ugly stepbrother, the one that no one wants to talk about. You guys call it the negative 10x engineer. What is that? I think it was a bit funny because you know everyone is talking about you know 10x engineers, hundred x engineers, you know, spinning up swarms of hundreds like of agents. ⁓ the thing is Anyone who has been working closely with AI have a very clear feeling that AI didn't eliminate mistakes. It's literally increasing their blast radius. And people, for example, who before without AI would never touch something, now two comments offline and it's all across the code base. So AI removes the speed limit. So now the bad decisions scale so architectural depth security holes flaky systems those are shipped very fast and very confidently because on top of it AI is generating a lot of code everything this plotes ⁓ PR code reviews because usually when you want to add new code into your into your code base you open the PR and then you know some person has to review it and that's the problem of a lot of engineers right now people tend to overtrust the AI that you know I'm I've generated it I trust it good enough. If what, the code reviewer is gonna, you know, is gonna find like the problems in there. So it it breaks the trust procedure between the engineers in the first place. It's people get so get overloaded because generating code is very fast. You can generate two thousand slides of code. But if I as a senior engineer have to review it, it's gonna take a big part of my a big part of my day. And it's you know, this is the price of your speed of shipment. It's way smaller if you need to involve more and more people to actually review it. And ⁓ the scariest engineers are not the slow ones, it's like the ones who are doing something fast and very confidently. That's what's happening right now. So right now to to guard one single, you know, you could have like one senior and like five juniors, and that was all right. Now to guard one junior, you need one senior who's like checking every line of code that something's bad hasn't happened. Weak engineers well like less experienced ⁓ engineers, they were bottlenecked by implementation. Now they're bottlenecked by judgment and some people who want to achieve results faster, they by bypass that judgment just to speed up the processes. But it's going wrong in very many ways because first of all, systems are not guarded because people don't invest into say they just think this is good enough code and it goes like to the production. And secondly, it breaks The engineering learning curve because I think it's a more general problem that right now less and less like junior engineers tend to learn like the why behind, which was always crucial for the whole time of engineering and just doing stuff. If it works, cool. If it doesn't work, I just write the prompt, just fix it. But you don't have any reasoning, any reasoning why it's it's like that. But this is how you grow good engineers. It's it's a scar tissue, you know. You need to break things on your own. You need to understand how thing you need sometimes build things from the scratch to understand how it works in what kind of scenarios to use one thing or another. You can't blindly rely on that. And this is what usually you do as a junior. Right now this is shifting and we're in very tricky position because we have juniors who don't learn any more bases because it's not fancy anymore. And we have senior people who with every year individual contributors are shifting more and more, you know. Managerial positions and everything. That's why we have bigger and bigger demand, but the number of proper engineers it's it's getting smaller and smaller because people just don't understand how the things are being built. They overrely on AI coding and really not capable to build things themselves, which is it's not on the same level as a proper engineer. Secondly, you are so codependent on the vendor of your code generation. Because if you wanna code properly, you're probably gonna need to use like one of the newer vendors. What you're gonna do when, for example, GitHub Copilot. So like one of the I mean there are like so many, you know, products on the on the market. One of the popular ones is GitHub Copilot, where it's like an iAgent which helps you like to code. So just randomly, I think it was beginning of June, they increased multiplication of tokens for the newer model, for example, GPT five point five by like X fifty. So imagine like right now, like what used to be something cheap now For you just to be able to code, you're gonna spend thousands of dollars like every month. What's it gonna be if they decide to make it x hundred and you're not even capable to code? That's the problem. You use it for every little nate peek. You need to throw entire problem, so much context into it because you don't even have a reasoning behind it. Cause in general, in the last couple of years, we're generating far more code than we are improving. So it's so many companies are trade maintainability for the feeling of speed and that technical depth is catching up already on so many companies and I totally believe it's gonna caught up even to even more that some of the projects are gonna be it's you're gonna ask every any engineer, all engineers hate to work on legacy systems. So this is even worse than legacy. This is even worse than legacy because behind the legacy there were some people. Now there are gonna be some businesses who gonna need to hire people who gonna need to clean up all the generated mess and sometimes build it from the scratch because you know to figure out what how it is working there and improve is gonna be more expensive expensive than actually building it from the scratch. So it's it's very interesting. It's very interesting. Marius, do you have anything on that? Yeah ⁓ I would like to add a little ⁓ Two cents from my side regarding junior ⁓ junior engineers right now. For them, yeah, it's it's really a tough market. They ⁓ from from their side. Companies are expecting a lot. That's why the hiring on the junior side froze a little bit. But if you have juniors at your side and they are shipping code and using Claude agents and such, with every single PR, what you need to do is to grill them down on what's going on in the code. They should definitely know what's going on in every single line, why they wrote every single line with Claude's help regarding the feature that they are updating. And that's that's how they will. basically start growing to to a proper engineers eventually. So so you have to double down on the explainability. Why have you written that? Maybe why not ⁓ you have written the code, but why did you create the PR with this ⁓ this line and exactly and and and and they have to know every single piece of ⁓ of the changes that they're made. I love that. It sounds like you're getting into the difference a little bit. like the solution and it's like scope driven development, but then like it it all depends on your team as well. Like if you can't afford to hire someone who knows what they're doing and you have to use juniors, then putting them in a position to actually be held accountable for every single line of code. That's cool. And I I like I think sometimes I had someone say like, ⁓ it seems like you're against AI. Like nine two three is against AI, but you're also like helping people implement AI. Like where do you stand? And it's just we're we're against the negative like the poor execution because we know what can happen. That's a very, very, very good point because it feels like probably this is this is just how the news work. It's like the more extreme opinion you have right now, the louder you can shout into the into the news. So it feels sometimes that right now there are your mainly, you know, two Two different parties, like the one is like totally against AI, you know, AI is bubble, you shouldn't use it for anything, you know, this is like pure evil. And the other side, which has like AI automated like twenty four seven, like their full life, and they're not doing like, you know, anything, coding from their phones. I think our sense is that both extremes are wrong. So avoiding AI completely is a problem. Like it's twenty twenty six. It's stupid, you know, to avoid it. But outsourcing your judgment to AI can be even a bigger problem. So And for that, you need to have an experience. You need to understand. You're like a like a doctor, like a surgeon. You need to know exactly where you can use it, where you shouldn't use it, where you can use it with less checks, where you should check like if you decided to automate it, where you need to check like every line of a code because it's a critical. And this is something that you only learn by building sys systems yourself. Because if you don't have that experience, you don't know whole, you know, like Seniors, more experienced people, it's not like they know more technologies. They just went through so many disasters that you know, you just have that inner feeling. You know, you see that, you know, this is something that can, you know, blow into our face. And same with the AI with experienced people. That's what you do. That's why all the teams in nine to free, we are using AI in the places where it makes sense, where it really gives you the contributions, minimizing the cost because We know where it's actually creating extra fiction because there is no such thing as a Swiss knife. There is not no such a single system which does everything perfectly. This has to be understood. And that's why, even working with so many clients, like you know, 2026, when everyone is screaming about LLMs, you know, yeah, we have so many use cases where we come and we're like, you don't need LLMs. This is not your use case. You need something from the classical ML. The project, for example, like one of my latest projects. 80% of the project is a classical ML because just for this specific for your specific use case, there is zero reasoning to use like LLMs because it's expensive and it's inaccurate for your use case. And ⁓ this is why you need people to who are giving you options, who are honestly giving you positive and negative things of the choices, who understand how the things should work, because if everything that you know is LLM. Then you're gonna propose all the solutions with LLMs, whether this makes sense or it doesn't make sense. And if you're gonna ask, and that's quite frequent when you ask for architecture with you know with clause something, it's gonna propose you multi-agent orchestrator vector dB. So you know, all the fancy like words when literally everything that you know at a fraction of a cost and it's gonna be way better, is gonna be just like a proper, I don't know, data ETL decision tree and some some monitoring. So if statement if statement. Yeah, that's a great example because it's ⁓ you don't need to overcomplicate things because if unless you you want to have AI in the pro you know it there has to be a reason to use one technology or another. You're not like okay, I want to use AI and then you search for a problem. It's vice versa. You have you have your pain points as a business and then you find the most appropriate solution for that. And if it's LLM, amazing. We do work with that. ⁓ but there are some problems which are just by the nature of this solution of this framework. It's language models. So if you're gonna have ⁓ time series forecasting, you know, if the person is proposing you LLMs, it doesn't make sense. It's just it's it just doesn't make sense. Or if the person is proposing you only one solution, probably they are not experienced enough to understand how these things ⁓ should work in the first place because there are always, you know, drawbacks to to to all of the solutions. I think that's ⁓ that's our sense. We are very pro AI, but we are very pro AI in the in the things where it matters and where it doesn't b you know ⁓ blows into your face after we left you because What is the point? That's I think one of the cool things at ⁓ nine to free. When we do projects, we always the idea is that you would be able to maintain it after we are left. So, like at the end of a project, we have a call with your engineers. We talk with you, you have can have any kind of a questions regarding the code, regarding the reasoning between the architecture pros, cons, what can be improved. And we talk to you talk through it ⁓ and from there you can take it. Because if consultancy builds you a black box. Which you have no idea how it works, it's just a third-party vendor dependency for for all your life because you have no understanding how it works. It's just gonna turn into one of the legacy system that you can't improve. But the idea is that this is something that can be maintained, something that can be improved and built on the top, because that's how great infrastructure is being built in the first hand. You know, you build and you expand. So like divide and conquer. Yeah. I I I have ⁓ only a few sentences to add up on this topic. So basically I don't think that ⁓ Us basically ninety three are pro or against AI either. We are pro ownership. We can definitely use AI tools like to to improve everything, but if there's no ownership, then what are we doing? Like we we are not helping anyone ⁓ in that point of view. So we we are taking ownership of every piece of code that we write, even though ⁓ it might it might not be with ⁓ with help of AI either way, like we are taking ownership of it. That's how we roll. It doesn't matter the dimension on which we land or pro AI or against. Yeah, if if there's no ownership, then yeah, we cannot do something. You you hit a powerful point, Marius and Vitalius, because owned intelligence, we actually have a PDF we'll put up here that Andrew and the team just finished working on where it walks through a five year AI plan. And at first when I saw this idea of the title, I was like, this is crazy because most people are talking in days and weeks, not years. But part of what you talked about, Vitalius, that has not changed. The fundamentals are still there. And Marius, you talked about ownership and your unique procedures that are unique to your company. You need to build on that. And that will not change whether it's Mythos or GPT eight. That will continue to be the same. And if you're going to replace a new KPI with AI, then you need to be looking at KPIs that actually matter to your company, not PRs, not lines of code, but actual achievements for your clients, your customers, reviews, feedback, ⁓ from your actual outputs. What else should people know walking away from this episode? I know we talked a little bit about that taste, that domain expertise that matters. I want to give them a little hope as they end this episode on what things they should be doing and what matters today. I have a small thought. Right now, since this is AI revolution, we ship more code than usual and ⁓ we can build features really, really fast. And what's happening is like ⁓ amount of features is bloating, even though like do we really need them or not? So what we need is a sense of a direction, which usually comes right now from PMs, from ⁓ project managers and such. So right now Project managers are really, really ⁓ useful more than ever before since they provide a sense of their direction on what features do should we build, what should work definitely for client and and whatnot. So I think the thing is like, yeah, we we can ship a lot of stuff and really really fast, but is it that necessary stuff that we need to ship? So I I believe I I would like to ⁓ leave an accent on this part really. That's that's a good one. I would maybe just add it's like it's it's a bit similar that try to work with expert matters that or understand your business or are ready to go deep into your business. You want to hire people who will search for your pain points and find the most appropriate, the most feasible the solution, like in terms of return of investment or like Customer satisfaction just for your specific use case. Don't search person who is leveraging a single technology and will just find for the where to you use it just for the sake of for the sake of ⁓ using it. Because the cost it's gonna cost a lot in production. If you're gonna go with someone like that who's gonna give you the reasoning, gonna give you all the risks, it's not gonna overhype you for the technology, gonna clearly tell you how things are gonna go, what kind of a pain points and how hard it's gonna be to maintain it in the future. That's important. If you are listening to Vitalius and Marius and feeling like this all sounds great, but I don't have the team to be able to do that. This is why we exist. And we could actually give you a free AI audit, a free discovery session. You can come to us whether you're just AI curious or you have a specific workflow that you're looking to solve. You can use the first link in the description. To book a free session with us. Marius Vitalius, great episode as usual. Thank you.