# The Interesting Changes Aren't Happening in the IDE
**Kevin:** [00:00:00] Hello, everybody, and welcome back to the It Depends Lessons in Technology Leadership podcast. I, as always, am your host. My name's Kevin Goldsmith. I've been in the software industry for thirty-five years, the last ten of which as a CTO. And in every episode, we talk about different aspects of technology leadership.
This week, this is gonna be an interesting one for me because this is a subject I think about all the time. It's also a subject I have avoided talking about too deeply on the podcast, and that subject is AI. It's clearly something that's completely changing our industry. It's clearly something that both me and every other technology person talks about all the time.
Why have I not talked about it on the podcast? Why have I not dedicated an episode to it? There's two reasons. Number one, a lot of people are [00:01:00] talking about it, so I didn't wanna just be another voice. I like talking about things that I have a unique approach to, meaning that even if you've heard about this subject from someone else, you'll get something new from it.
The other reason is that because I also don't like talking about things when I'm in the middle of them. I prefer to talk about things where I have the benefit of not only doing the thing, but figuring out the mistakes, or having done something multiple times and developed a little bit of experience around it, because I don't wanna give you bad advice. I don't wanna tell you something that's wrong.
So rather than talk about how you do this, to talk a little bit about here's been my process of addressing this new technology that's moving very quickly. Here's the lessons I've learned thus far that I feel relatively [00:02:00] confident about.
And here's some ideas of how you can approach this in your own company if you're also going through this transition to this new, incredibly powerful technology. And I promise you, you all are, whether you're in the middle of it or not, whether you're actively engaging with it or not, whether you've been avoiding it up till now or we're all in it.
That's what we're covering today. Now, as always, One of the ways I figure out what I want to talk about in each episode is I think back through the conversations I've been having, usually with mentees, sometimes with peers.
As I said, every conversation almost I have with a peer or a mentee involves AI for certainly the last year and a half or so. In this particular case, there's a group of alumni from my university. The school set up this sort of mentoring program. This group I've been working with for a couple of years now.[00:03:00]
It's mostly earlier career engineering leaders, people who are either moving into engineering or technology management or existing technology managers who want to move into more senior roles. So we were talking. And when the subjects came up, we started talking about AI slop.
Slop, by the way, I learned, Webster's word of the year last year, makes sense. Every one of us was addressing AI from a different angle. Obviously, none of us had completely figured it out. These are all very thoughtful, capable people. The honest version of the conversation was the version that doesn't show up in the blogs or the trade press.
Most of what gets said about AI in engineering leadership, is two different things. One is red-pilled, we love AI, very booster-ish, and I have to admit, I'm closer to that camp versus the other side, which is [00:04:00] this is a fad or the quality of this isn't ready for real production environments or, the doomer version, and I hear both.
The real thing is somewhere in between. I've been waiting to feel like I had something to say about AI. This conversation got me off the fence and really made me think it might be time to talk about I still don't feel like I have it solved, but my thinking has changed enough over the last two years that walking through the evolution might be more useful than waiting for a tidy answer.
What I can do is tell you what I've changed my mind about.
The surface question is: how do we adopt AI? Every company in the world is at some point on that process. That question, though, is a trap. It assumes there's a destination, and if you reach it, you're done.
But the technology is changing faster than any of the systems that surround it. AI is [00:05:00] changing faster than your hiring practices, faster than your review processes, faster than your testing infrastructure, faster than your team structures, faster than your career ladders. The force is pulling at every engineering leader right now.
There's pressure from above to show productivity gains and cost savings. There's pressure from below, from engineers who either want to adopt aggressively or are quietly worried about their role. Is this gonna put me out of a job? There's pressure from peers in product and design who are now using these tools and generating things they couldn't generate before.
And obviously, you're putting pressure on yourself because you don't fully know what good looks like. And if you're, like me, a CTO You're expected to be modeling this for the company. You are the head of technology. This is a new technology. It is your responsibility to make sure the company is adopting it and adopting it well.
And you're trying to do this [00:06:00] across the entire enterprise often, and you're dealing with people who don't know what good looks like. None of us know what the endpoint is gonna be like. If you wait until you have a complete answer before you act, the answer will have changed by the time you act on it.
I've talked to folks about, "Oh, we should get AI training for the enterprise." In the time it takes to generate a course, new models come out, and they completely change the game.
What's at stake when leaders get this wrong is either you over-rotate, you move too fast, and you ship slop your customers will eventually feel. Or you under-rotate, and your team falls behind a market that is moving 'cause your competitors are also figuring out how to use this technology.
Both of these have a cost, and that cost is asymmetric depending on your business. The patterns I'm seeing, I'm seeing across a wide variety of companies, and the [00:07:00] patterns are surprisingly consistent across very different environments. Large enterprises, small startups, heavily regulated and lightly regulated, engineering-led companies and business-led cultures.
A few patterns I've seen that have come up repeatedly. Adoption metrics that look great while the underlying work quality is uneven.
Review processes that started fine and quietly stopped scaling. Product and design capacity becoming the new bottleneck because dev throughput went up. We talked about this a little bit, in the systems thinking episode, where AI is completely changing the system. Senior engineers are spending more of their time evaluating output rather than producing it, often without actually being acknowledged as part of their role.
Their CEOs and executives who are now generating their own output and assuming the engineering team can run with it. The pattern across all of them, the technology adoption [00:08:00] is the easy part, relatively. The organizational adoption is where everyone is struggling. And it's because almost everyone is making it up as they go.
If you're looking for the AI transformation, don't look at the IDE, look at the org chart. Because AI is not just changing the tools, but by its very nature, it actually has to change the system that it's in, the processes of that system, the structures, the team structures of the system, all that has to change. It is that transformative.
As always, I like to give you some sort of structure or framework you can use to help. This one is the four layers of AI adoption. The further down the list you go, the harder the layer is and the more leadership it requires.
Most companies are working at layer one and ignoring two through four right now. So layer [00:09:00] one, tooling. Which tools, which models, which licenses, adoption metrics, cost measurement. This is the layer that vendors are constantly pitching you on both The big players and all the new startups that are trying to become new AI tool vendors, there's so many of them, they reach out to me constantly.
That's what most companies are focused on, just the tools.
Layer two is process. What does code review look like when one human approves and one AI approves, or two AIs approve, or one AI approves and no humans? What does testing mean when pull request volume goes up several times over? What does a sprint look like when the ticket sizes get bigger because agents handle more in one shot?
Every engineering team I know that's made some progress on AI [00:10:00] adoption starts having these problems relatively quickly.
Layer three, structure. Team shapes change. Smaller teams, different ratios of product to dev, different ratios of QA to dev. Designers and product owners are taking on work that used to live elsewhere because dev throughput has outpaced product capacity. Where does QA live when AI is starting to generate test plans and then generate the tests? There's career ladders that are built for human velocity, but not human plus agent velocity.
The last layer, layer four, is judgment. When is the output good enough to ship? When is the output slop your customers will eventually feel?
When does the output need a deterministic system underneath it because the cost of being wrong is real? This is the layer that no one is hiring for and everyone needs right [00:11:00] now. We're all figuring this one out.
So what to observe in your team right now? Where does friction show up? Is it in the tooling, the process, the structure, or the judgment? In prior episodes, we talked about tracing work through the system. This is the kind of thing that is really helpful right now.
The system is changing. The expectations around where the bottlenecks are, are being rewritten. Those expectations no longer are valid. So now you gotta start really understanding where your new bottlenecks are. For your organization, are your skeptics quiet or loud? Are your eager adopters producing more or producing more output that someone else has to clean up?
Is your review process scaling with your generation rate? These are all important questions for you to look at in your organization today and figure out the answers to these questions because that is setting the [00:12:00] conditions for how you address it.
So how do you figure out where to invest? Start at whichever layer is currently failing, not the layer that's the loudest. If your skeptics are loud, but really what's failing is your review process That's where you gotta put your energy.
If your tooling is fine, but your reviews are drowning, the problem is process, not tooling. Layer two, not layer one. If your reviews are fine, but your team shapes are wrong, your problem is in your structure, layer three.
Some common failure modes. Buying more tools to solve a process problem. This predates AI. It gets worse with AI 'cause everything's changing much faster, and plenty of people are there to sell you tools to solve each of these problems. Tools don't solve process problems.
Each of these compounds because you're [00:13:00] spending energy at the wrong layer. Talk through those four layers. Make sure you're putting your energy into the right one.
What can you do this week? Ask your team where the friction actually is, and listen for which layer they describe. They won't have heard this podcast. They're gonna talk in generalities or specifics, but you need to map to which layer and then figure out how to address it. Look at one pull request from the last week and ask whether your current review process is doing its job. Notice who in your org is doing more cleanup work and whether that role is named or invisible.
The slowest part of your team will tell you which layer to work on. It is rarely the one you wanted to work on.
Another important aspect of this, within engineering or if you're driving a larger AI transformation in your company, this is a transformation. This is a culture change. This is not a tool rollout. This is not a new HRIS system. This is a [00:14:00] massive change.
It's closer to a culture change than a software adoption. treat it the way you would treat any major change initiative, with actual change management discipline.
The good news is we have decades of practice running change initiatives, and most of those tools apply here.
Let's talk about the different people on your team and some of the reactions you're gonna get or some of the reactions you probably have right now.
You get those eager adopters. These are the ones with the most personal upside from looking like they're ahead or just really excited about new technology.
They often produce real value. Sometimes they also produce the slop because they're so excited about it and they're basically using it without having really thought it through. They're just adopting, adopting, adopting. That slop ends up getting cleaned up by someone else who now has a negative view of this technology 'cause they're cleaning up somebody else's mess.
These early adopter folks, they [00:15:00] should be rewarded. They're helping you evangelize this new technology within the team. It comes better when it comes from actual practitioners, not just leadership. However, that reward should be for quality, not just throughput.
Let's talk about the skeptics. Often, these are the folks who are the senior engineers whose craft is being challenged. These are people who have spent decades becoming exceptionally skilled at writing software, and now here's a tool that writes it almost as good as they do, or maybe even better.
These are sometimes the most thoughtful people in the room. They're very articulate about why this technology is not good. Some of them are right, and some of them are protecting their identity, and you need to tell the difference. Their concerns about deterministic outputs in financial, legal, or safety-critical use cases are usually correct. [00:16:00] So you need to hear them. You need to listen to them. You need to support them. What I've found, both in my company and others, is a lot of these folks were skeptics originally, and as they saw the technology evolve in near real-time, and they started to figure out how they could use it, and they learned how to better direct it so that they were getting higher quality output from these tools.
Those skeptics, those craftspeople have figured out, you know what? This just makes me more productive. And now that I know how to make sure the quality is where I would expect it, I'm happy. A lot of these folks I've heard about from my organization or other organizations were your early skeptics and now are your e-enthusiastic adopters. You just gotta give them [00:17:00] time.
Let's talk about the folks who are quietly worried. Some of these are folks who are not against AI, but they're worried. What's my job becoming? This is a group that often gets ignored in adoption planning. They're not gonna necessarily raise their hand because they're gonna be hearing from you, they're gonna be hearing from other leadership about how important this stuff is, and so they're afraid to be vocal about it because they know they're challenging the direction of the organization.
So they're not gonna raise their hand. You're gonna have to ask. Some of these are early adopters who fell behind. So they initially got really excited, and they started to use it. This technology is moving very, very quickly. This is one I didn't necessarily see coming In my own and other organizations, they started excited. They were evangelizing it. As the team progressed and the bar [00:18:00] moved, those senior engineers who were original skeptics and then became really excited about it and got very involved and started learning the technology and became quite skilled at it and are now leading conversations about advanced topics in AI.
Some of the earlier adopters who got excited about it as a tool could now be feeling like it's moved past them, and now maybe they're feeling like they're-- should be further along, maybe feeling a little bit of imposter syndrome, and they don't wanna be vocal about it. They're afraid to talk about it.
They're quiet, and the silence looks like the silence of the skeptics, but it's really something different. They're afraid to ask for help because they were the ones who were supposed to be ahead. And in many cases, they are worried about their jobs, which makes their silence even deeper.
If you only listen to who is loud, you'll miss them entirely.
[00:19:00] Let's talk about the executives and the CEOs. Many CEOs are now generating things and assuming the engineering team can run with them. The pattern, a CEO sends a finished-looking AI output and expects the engineering team to pick it up and deliver it immediately. Because they've heard so much about the technology, they've talked to their friends who are CEOs at other companies who tell them stories about doing this exact thing.
Maybe it's true, maybe it's not. But this is happening at a lot of companies, and most of us engineering leaders are not yet good at managing it. In the last podcast, I talked about those visionary CEOs who are idea factories and now have a company with all these engineers, and they're really hoping engineers can pick up some of these ideas and actually turn them into products quickly.
That was a challenge pre-AI. With AI, the CEO is now making their own [00:20:00] prototypes or maybe even what they would consider their MVP possi-possibly ready for release, and then handing that to engineering and saying, "Please ship this." If it was hard when the CEO just had lots of good ideas and you couldn't do them all, imagine when the CEO is producing code.
The hardest part of AI adoption in technology is not the engineers who are excited, it's the engineers who used to be excited and now are not.
a year ago, I thought of AI as a productivity tool. I no longer think that's the right frame. Six months ago, I thought adoption was the leadership challenge. I still think it's a real challenge, but now I think it is the part we m-most equipped to handle. We have tools for adoption. Th-those tools are called change management. The mistake is treating AI adoption as a tooling decision instead of a transformation.
If you're hoping your organization will adopt naturally because this AI stuff is in the ether and everyone's [00:21:00] talking about it and people are excited about it, maybe that's how we are in technology. The rest of the world, nah, not so much. It requires active design, and you have to think about it that way.
Today, I think the leadership skill we need most is judgment about output quality, and we're not yet training for it. Now, As I said, it's been a process for me just like every other leader I know. There are some mistakes I have made and I've learned from already.
I underestimated how much of the work shifts from writing to evaluating. I overestimated how easily existing career ladders would absorb the change. And I missed the early adopters who would fall behind. I was watching the skeptics and the eager. That middle group surprised me. As engineering teams leveled up on their adoption of AI, I was surprised how quickly that put back pressure on product as well.
So it put [00:22:00] forward pressure on reviews, put forward pressure on test, put back pressure on product in a way that I didn't expect to happen so quickly.
Some things I'm still figuring out, how to measure the right things, not just usage. We've heard about token maxing and companies focusing on how many tokens per engineer.
That Was never a good metric. It's very easy to burn tokens if you just wanna burn tokens, and if you know that the amount of tokens you're using is being looked at, you're gonna figure out ways to burn tokens, whether they're practical or useful or helpful for the company or not. I always talk about tying things to business metrics and focusing on outcome, not output.
Still figuring out what that means in this new world. Some of the old ways I've done that theoretically seem to work, but it's too early for me to say definitively, this is the way I'm doing it, and I feel pretty good about [00:23:00] it. Another thing I'm still figuring out, how to evaluate engineers when their output includes work an agent did.
How to keep the craft of engineering alive in a world where the generative part is cheap. How to make sure we're building things customers actually want, not just the things AI made it easy to build.
I'm not done changing my mind about this. I expect to change it again and again and again as the technology improves and as we learn more.
In closing, I want to reiterate AI adoption is not a technology project. It is an organizational redesign happening in real time, whether you lead it or not.
Here's something you can do this week. Ask yourself which of the four layers your organization is actually working on right now. Then ask which of the four is actually failing. If those are not the same [00:24:00] layer, that's your real work. The companies that get this right will not be the ones with the best tools.
They will be the ones whose leaders kept thinking about this after everyone declared it solved. I will probably do another episode on AI in another six months, a year. My thinking will be different, and that's the point. This is not a solved challenge. This is an evolving challenge. It's an exciting challenge. I am energized. I am excited about it. I've been lucky enough to have been in the industry for several incredibly transformative technology shifts. This one is different because of the speed and how much it affects not just the products we build or the capabilities we can add, but also how it affects just every part [00:25:00] of how our work gets done.
And for somebody like me that not only loves technology, loves building things, but also loves the pieces of it, the systems thinking aspect, the team structure aspect, the performance management aspect, AI is blowing all this stuff up simultaneously, and that's really exciting trying to figure it out and watching everyone else figure it out too.
And everyone, 'cause we're all figuring it out, I've had some of the most substantive conversations with peers at other companies talking through, " what have you tried, and how has this worked for you? And what do you think about this?" And that kind of thing, because we're all figuring it out at the same time, and we're all having to figure it out together at some level.
For me, this is just a really exciting time, and it's been a challenge for me, given how much I think about it, to not talk about it on the podcast. So I'm glad [00:26:00] I've finally done an episode on it. Six months or a year from now, I'll have learned a bunch more lessons and I'll maybe be able to share those with you and help you on this journey that you are probably also on.
I will see you next week in the newsletter. I will see you back here on the podcast in two weeks. Have a great week.
**Kevin:** [00:00:00] Hello, everybody, and welcome back to the It Depends Lessons in Technology Leadership podcast. I, as always, am your host. My name's Kevin Goldsmith. I've been in the software industry for thirty-five years, the last ten of which as a CTO. And in every episode, we talk about different aspects of technology leadership.
This week, this is gonna be an interesting one for me because this is a subject I think about all the time. It's also a subject I have avoided talking about too deeply on the podcast, and that subject is AI. It's clearly something that's completely changing our industry. It's clearly something that both me and every other technology person talks about all the time.
Why have I not talked about it on the podcast? Why have I not dedicated an episode to it? There's two reasons. Number one, a lot of people are [00:01:00] talking about it, so I didn't wanna just be another voice. I like talking about things that I have a unique approach to, meaning that even if you've heard about this subject from someone else, you'll get something new from it.
The other reason is that because I also don't like talking about things when I'm in the middle of them. I prefer to talk about things where I have the benefit of not only doing the thing, but figuring out the mistakes, or having done something multiple times and developed a little bit of experience around it, because I don't wanna give you bad advice. I don't wanna tell you something that's wrong.
So rather than talk about how you do this, to talk a little bit about here's been my process of addressing this new technology that's moving very quickly. Here's the lessons I've learned thus far that I feel relatively [00:02:00] confident about.
And here's some ideas of how you can approach this in your own company if you're also going through this transition to this new, incredibly powerful technology. And I promise you, you all are, whether you're in the middle of it or not, whether you're actively engaging with it or not, whether you've been avoiding it up till now or we're all in it.
That's what we're covering today. Now, as always, One of the ways I figure out what I want to talk about in each episode is I think back through the conversations I've been having, usually with mentees, sometimes with peers.
As I said, every conversation almost I have with a peer or a mentee involves AI for certainly the last year and a half or so. In this particular case, there's a group of alumni from my university. The school set up this sort of mentoring program. This group I've been working with for a couple of years now.[00:03:00]
It's mostly earlier career engineering leaders, people who are either moving into engineering or technology management or existing technology managers who want to move into more senior roles. So we were talking. And when the subjects came up, we started talking about AI slop.
Slop, by the way, I learned, Webster's word of the year last year, makes sense. Every one of us was addressing AI from a different angle. Obviously, none of us had completely figured it out. These are all very thoughtful, capable people. The honest version of the conversation was the version that doesn't show up in the blogs or the trade press.
Most of what gets said about AI in engineering leadership, is two different things. One is red-pilled, we love AI, very booster-ish, and I have to admit, I'm closer to that camp versus the other side, which is [00:04:00] this is a fad or the quality of this isn't ready for real production environments or, the doomer version, and I hear both.
The real thing is somewhere in between. I've been waiting to feel like I had something to say about AI. This conversation got me off the fence and really made me think it might be time to talk about I still don't feel like I have it solved, but my thinking has changed enough over the last two years that walking through the evolution might be more useful than waiting for a tidy answer.
What I can do is tell you what I've changed my mind about.
The surface question is: how do we adopt AI? Every company in the world is at some point on that process. That question, though, is a trap. It assumes there's a destination, and if you reach it, you're done.
But the technology is changing faster than any of the systems that surround it. AI is [00:05:00] changing faster than your hiring practices, faster than your review processes, faster than your testing infrastructure, faster than your team structures, faster than your career ladders. The force is pulling at every engineering leader right now.
There's pressure from above to show productivity gains and cost savings. There's pressure from below, from engineers who either want to adopt aggressively or are quietly worried about their role. Is this gonna put me out of a job? There's pressure from peers in product and design who are now using these tools and generating things they couldn't generate before.
And obviously, you're putting pressure on yourself because you don't fully know what good looks like. And if you're, like me, a CTO You're expected to be modeling this for the company. You are the head of technology. This is a new technology. It is your responsibility to make sure the company is adopting it and adopting it well.
And you're trying to do this [00:06:00] across the entire enterprise often, and you're dealing with people who don't know what good looks like. None of us know what the endpoint is gonna be like. If you wait until you have a complete answer before you act, the answer will have changed by the time you act on it.
I've talked to folks about, "Oh, we should get AI training for the enterprise." In the time it takes to generate a course, new models come out, and they completely change the game.
What's at stake when leaders get this wrong is either you over-rotate, you move too fast, and you ship slop your customers will eventually feel. Or you under-rotate, and your team falls behind a market that is moving 'cause your competitors are also figuring out how to use this technology.
Both of these have a cost, and that cost is asymmetric depending on your business. The patterns I'm seeing, I'm seeing across a wide variety of companies, and the [00:07:00] patterns are surprisingly consistent across very different environments. Large enterprises, small startups, heavily regulated and lightly regulated, engineering-led companies and business-led cultures.
A few patterns I've seen that have come up repeatedly. Adoption metrics that look great while the underlying work quality is uneven.
Review processes that started fine and quietly stopped scaling. Product and design capacity becoming the new bottleneck because dev throughput went up. We talked about this a little bit, in the systems thinking episode, where AI is completely changing the system. Senior engineers are spending more of their time evaluating output rather than producing it, often without actually being acknowledged as part of their role.
Their CEOs and executives who are now generating their own output and assuming the engineering team can run with it. The pattern across all of them, the technology adoption [00:08:00] is the easy part, relatively. The organizational adoption is where everyone is struggling. And it's because almost everyone is making it up as they go.
If you're looking for the AI transformation, don't look at the IDE, look at the org chart. Because AI is not just changing the tools, but by its very nature, it actually has to change the system that it's in, the processes of that system, the structures, the team structures of the system, all that has to change. It is that transformative.
As always, I like to give you some sort of structure or framework you can use to help. This one is the four layers of AI adoption. The further down the list you go, the harder the layer is and the more leadership it requires.
Most companies are working at layer one and ignoring two through four right now. So layer [00:09:00] one, tooling. Which tools, which models, which licenses, adoption metrics, cost measurement. This is the layer that vendors are constantly pitching you on both The big players and all the new startups that are trying to become new AI tool vendors, there's so many of them, they reach out to me constantly.
That's what most companies are focused on, just the tools.
Layer two is process. What does code review look like when one human approves and one AI approves, or two AIs approve, or one AI approves and no humans? What does testing mean when pull request volume goes up several times over? What does a sprint look like when the ticket sizes get bigger because agents handle more in one shot?
Every engineering team I know that's made some progress on AI [00:10:00] adoption starts having these problems relatively quickly.
Layer three, structure. Team shapes change. Smaller teams, different ratios of product to dev, different ratios of QA to dev. Designers and product owners are taking on work that used to live elsewhere because dev throughput has outpaced product capacity. Where does QA live when AI is starting to generate test plans and then generate the tests? There's career ladders that are built for human velocity, but not human plus agent velocity.
The last layer, layer four, is judgment. When is the output good enough to ship? When is the output slop your customers will eventually feel?
When does the output need a deterministic system underneath it because the cost of being wrong is real? This is the layer that no one is hiring for and everyone needs right [00:11:00] now. We're all figuring this one out.
So what to observe in your team right now? Where does friction show up? Is it in the tooling, the process, the structure, or the judgment? In prior episodes, we talked about tracing work through the system. This is the kind of thing that is really helpful right now.
The system is changing. The expectations around where the bottlenecks are, are being rewritten. Those expectations no longer are valid. So now you gotta start really understanding where your new bottlenecks are. For your organization, are your skeptics quiet or loud? Are your eager adopters producing more or producing more output that someone else has to clean up?
Is your review process scaling with your generation rate? These are all important questions for you to look at in your organization today and figure out the answers to these questions because that is setting the [00:12:00] conditions for how you address it.
So how do you figure out where to invest? Start at whichever layer is currently failing, not the layer that's the loudest. If your skeptics are loud, but really what's failing is your review process That's where you gotta put your energy.
If your tooling is fine, but your reviews are drowning, the problem is process, not tooling. Layer two, not layer one. If your reviews are fine, but your team shapes are wrong, your problem is in your structure, layer three.
Some common failure modes. Buying more tools to solve a process problem. This predates AI. It gets worse with AI 'cause everything's changing much faster, and plenty of people are there to sell you tools to solve each of these problems. Tools don't solve process problems.
Each of these compounds because you're [00:13:00] spending energy at the wrong layer. Talk through those four layers. Make sure you're putting your energy into the right one.
What can you do this week? Ask your team where the friction actually is, and listen for which layer they describe. They won't have heard this podcast. They're gonna talk in generalities or specifics, but you need to map to which layer and then figure out how to address it. Look at one pull request from the last week and ask whether your current review process is doing its job. Notice who in your org is doing more cleanup work and whether that role is named or invisible.
The slowest part of your team will tell you which layer to work on. It is rarely the one you wanted to work on.
Another important aspect of this, within engineering or if you're driving a larger AI transformation in your company, this is a transformation. This is a culture change. This is not a tool rollout. This is not a new HRIS system. This is a [00:14:00] massive change.
It's closer to a culture change than a software adoption. treat it the way you would treat any major change initiative, with actual change management discipline.
The good news is we have decades of practice running change initiatives, and most of those tools apply here.
Let's talk about the different people on your team and some of the reactions you're gonna get or some of the reactions you probably have right now.
You get those eager adopters. These are the ones with the most personal upside from looking like they're ahead or just really excited about new technology.
They often produce real value. Sometimes they also produce the slop because they're so excited about it and they're basically using it without having really thought it through. They're just adopting, adopting, adopting. That slop ends up getting cleaned up by someone else who now has a negative view of this technology 'cause they're cleaning up somebody else's mess.
These early adopter folks, they [00:15:00] should be rewarded. They're helping you evangelize this new technology within the team. It comes better when it comes from actual practitioners, not just leadership. However, that reward should be for quality, not just throughput.
Let's talk about the skeptics. Often, these are the folks who are the senior engineers whose craft is being challenged. These are people who have spent decades becoming exceptionally skilled at writing software, and now here's a tool that writes it almost as good as they do, or maybe even better.
These are sometimes the most thoughtful people in the room. They're very articulate about why this technology is not good. Some of them are right, and some of them are protecting their identity, and you need to tell the difference. Their concerns about deterministic outputs in financial, legal, or safety-critical use cases are usually correct. [00:16:00] So you need to hear them. You need to listen to them. You need to support them. What I've found, both in my company and others, is a lot of these folks were skeptics originally, and as they saw the technology evolve in near real-time, and they started to figure out how they could use it, and they learned how to better direct it so that they were getting higher quality output from these tools.
Those skeptics, those craftspeople have figured out, you know what? This just makes me more productive. And now that I know how to make sure the quality is where I would expect it, I'm happy. A lot of these folks I've heard about from my organization or other organizations were your early skeptics and now are your e-enthusiastic adopters. You just gotta give them [00:17:00] time.
Let's talk about the folks who are quietly worried. Some of these are folks who are not against AI, but they're worried. What's my job becoming? This is a group that often gets ignored in adoption planning. They're not gonna necessarily raise their hand because they're gonna be hearing from you, they're gonna be hearing from other leadership about how important this stuff is, and so they're afraid to be vocal about it because they know they're challenging the direction of the organization.
So they're not gonna raise their hand. You're gonna have to ask. Some of these are early adopters who fell behind. So they initially got really excited, and they started to use it. This technology is moving very, very quickly. This is one I didn't necessarily see coming In my own and other organizations, they started excited. They were evangelizing it. As the team progressed and the bar [00:18:00] moved, those senior engineers who were original skeptics and then became really excited about it and got very involved and started learning the technology and became quite skilled at it and are now leading conversations about advanced topics in AI.
Some of the earlier adopters who got excited about it as a tool could now be feeling like it's moved past them, and now maybe they're feeling like they're-- should be further along, maybe feeling a little bit of imposter syndrome, and they don't wanna be vocal about it. They're afraid to talk about it.
They're quiet, and the silence looks like the silence of the skeptics, but it's really something different. They're afraid to ask for help because they were the ones who were supposed to be ahead. And in many cases, they are worried about their jobs, which makes their silence even deeper.
If you only listen to who is loud, you'll miss them entirely.
[00:19:00] Let's talk about the executives and the CEOs. Many CEOs are now generating things and assuming the engineering team can run with them. The pattern, a CEO sends a finished-looking AI output and expects the engineering team to pick it up and deliver it immediately. Because they've heard so much about the technology, they've talked to their friends who are CEOs at other companies who tell them stories about doing this exact thing.
Maybe it's true, maybe it's not. But this is happening at a lot of companies, and most of us engineering leaders are not yet good at managing it. In the last podcast, I talked about those visionary CEOs who are idea factories and now have a company with all these engineers, and they're really hoping engineers can pick up some of these ideas and actually turn them into products quickly.
That was a challenge pre-AI. With AI, the CEO is now making their own [00:20:00] prototypes or maybe even what they would consider their MVP possi-possibly ready for release, and then handing that to engineering and saying, "Please ship this." If it was hard when the CEO just had lots of good ideas and you couldn't do them all, imagine when the CEO is producing code.
The hardest part of AI adoption in technology is not the engineers who are excited, it's the engineers who used to be excited and now are not.
a year ago, I thought of AI as a productivity tool. I no longer think that's the right frame. Six months ago, I thought adoption was the leadership challenge. I still think it's a real challenge, but now I think it is the part we m-most equipped to handle. We have tools for adoption. Th-those tools are called change management. The mistake is treating AI adoption as a tooling decision instead of a transformation.
If you're hoping your organization will adopt naturally because this AI stuff is in the ether and everyone's [00:21:00] talking about it and people are excited about it, maybe that's how we are in technology. The rest of the world, nah, not so much. It requires active design, and you have to think about it that way.
Today, I think the leadership skill we need most is judgment about output quality, and we're not yet training for it. Now, As I said, it's been a process for me just like every other leader I know. There are some mistakes I have made and I've learned from already.
I underestimated how much of the work shifts from writing to evaluating. I overestimated how easily existing career ladders would absorb the change. And I missed the early adopters who would fall behind. I was watching the skeptics and the eager. That middle group surprised me. As engineering teams leveled up on their adoption of AI, I was surprised how quickly that put back pressure on product as well.
So it put [00:22:00] forward pressure on reviews, put forward pressure on test, put back pressure on product in a way that I didn't expect to happen so quickly.
Some things I'm still figuring out, how to measure the right things, not just usage. We've heard about token maxing and companies focusing on how many tokens per engineer.
That Was never a good metric. It's very easy to burn tokens if you just wanna burn tokens, and if you know that the amount of tokens you're using is being looked at, you're gonna figure out ways to burn tokens, whether they're practical or useful or helpful for the company or not. I always talk about tying things to business metrics and focusing on outcome, not output.
Still figuring out what that means in this new world. Some of the old ways I've done that theoretically seem to work, but it's too early for me to say definitively, this is the way I'm doing it, and I feel pretty good about [00:23:00] it. Another thing I'm still figuring out, how to evaluate engineers when their output includes work an agent did.
How to keep the craft of engineering alive in a world where the generative part is cheap. How to make sure we're building things customers actually want, not just the things AI made it easy to build.
I'm not done changing my mind about this. I expect to change it again and again and again as the technology improves and as we learn more.
In closing, I want to reiterate AI adoption is not a technology project. It is an organizational redesign happening in real time, whether you lead it or not.
Here's something you can do this week. Ask yourself which of the four layers your organization is actually working on right now. Then ask which of the four is actually failing. If those are not the same [00:24:00] layer, that's your real work. The companies that get this right will not be the ones with the best tools.
They will be the ones whose leaders kept thinking about this after everyone declared it solved. I will probably do another episode on AI in another six months, a year. My thinking will be different, and that's the point. This is not a solved challenge. This is an evolving challenge. It's an exciting challenge. I am energized. I am excited about it. I've been lucky enough to have been in the industry for several incredibly transformative technology shifts. This one is different because of the speed and how much it affects not just the products we build or the capabilities we can add, but also how it affects just every part [00:25:00] of how our work gets done.
And for somebody like me that not only loves technology, loves building things, but also loves the pieces of it, the systems thinking aspect, the team structure aspect, the performance management aspect, AI is blowing all this stuff up simultaneously, and that's really exciting trying to figure it out and watching everyone else figure it out too.
And everyone, 'cause we're all figuring it out, I've had some of the most substantive conversations with peers at other companies talking through, " what have you tried, and how has this worked for you? And what do you think about this?" And that kind of thing, because we're all figuring it out at the same time, and we're all having to figure it out together at some level.
For me, this is just a really exciting time, and it's been a challenge for me, given how much I think about it, to not talk about it on the podcast. So I'm glad [00:26:00] I've finally done an episode on it. Six months or a year from now, I'll have learned a bunch more lessons and I'll maybe be able to share those with you and help you on this journey that you are probably also on.
I will see you next week in the newsletter. I will see you back here on the podcast in two weeks. Have a great week.
