Why Are Teams Only Aligned During a Crisis? | CircleCI & MongoDB

WEBVTT
00:00:08.537 --> 00:00:09.387
to Dev Interrupted.
00:00:09.526 --> 00:00:11.176
I'm your host, Ben Lloyd Pearson.
00:00:11.772 --> 00:00:13.333
And I'm your host, Andrew Ziegler.
00:00:13.583 --> 00:00:27.594
In today's news, we're talking about why startups are leaving big corporations in the dust when it comes to AI, why building a simple calculator app is actually a complex puzzle, and what to do now that AI has made your whiteboard interviews completely useless.
00:00:28.219 --> 00:00:29.689
Ben, what do you want to talk about first today?
00:00:30.257 --> 00:00:36.887
Well, I feel like since everything seems to be about AI anymore, maybe we just start with the one story that's not AI this week.
00:00:37.264 --> 00:00:39.003
Oh, I love the calculator story.
00:00:39.393 --> 00:00:46.823
So this calculator story that we're going to include in the roundup is a deep dive into something that we all take for granted every day.
00:00:47.073 --> 00:00:51.393
You know, we all open up a computer or our phone and do a simple calculation on the calculator app.
00:00:51.404 --> 00:00:54.823
But did you know that under the hood, this is actually a lot more complex than you would have thought?
00:00:55.134 --> 00:01:00.694
And this is because of how we represent numbers in computers, in the abstract ways in which we have to do so.
00:01:00.973 --> 00:01:05.679
And then when you start introducing calculations, You start sacrificing accuracy for precision.
00:01:06.328 --> 00:01:09.198
And in this article, it's a really interesting deep dive.
00:01:09.228 --> 00:01:14.198
I will say I'm not the biggest math nerd in the world, so I can't do it justice.
00:01:14.209 --> 00:01:20.629
But for those that are, I know they'll find this a fun ride behind the engineering journey and the complexity of a calculator app.
00:01:21.597 --> 00:01:30.337
Yeah, you know, this is, I wanted to include this story because it is just a fun look at something that, like you said, we take, we all take this for granted.
00:01:30.418 --> 00:01:39.778
In fact, you know, one of the biggest mistakes that my teachers taught me in high school and elementary was that I wasn't always going to have a calculator in my pocket, so I have to learn math.
00:01:40.263 --> 00:01:43.462
But, it turns out for my reality that the opposite was true.
00:01:43.462 --> 00:01:46.472
I do always have a calculator in my pocket.
00:01:46.513 --> 00:01:51.572
And actually the entire reason I got into software development was because I hate, how much I hate math.
00:01:51.582 --> 00:01:57.597
So, I'm really glad that people like this exist because it enables people like me to learn.
00:01:57.897 --> 00:02:02.727
you know, I think my distaste for math really took off when I learned about the quadratic equation.
00:02:02.768 --> 00:02:08.187
And I literally learned how to write software on my calculator that did my homework for me.
00:02:08.557 --> 00:02:14.437
And, you know, technically teachers might call that cheating, but it actually sparked a whole interest in me.
00:02:14.557 --> 00:02:18.177
you know, I would learn the formulas enough to write an application that solved it for me.
00:02:18.669 --> 00:02:22.429
But yeah, just to say, like, I don't understand it either.
00:02:23.028 --> 00:02:25.688
I'm really glad that there are people in this world who can.
00:02:26.019 --> 00:02:34.068
And if you're someone who is, like, really geeky about math, it is a pretty fascinating breakdown on how this, like, very standard technology works today.
00:02:34.824 --> 00:02:52.832
And another thing as well, on this one that was interesting to me is how when you solve a problem like a calculator app, you know, you might be able to solve 99 percent of it really, really Easily, but getting that last 1 percent so that it's perfect and putting in that blood, sweat and tears to make something foundational works incredible.
00:02:52.902 --> 00:02:54.622
That's what great technology is all about.
00:02:55.647 --> 00:03:05.807
Yeah, so I want to talk about one of the stories that I read this week, and that's about how there's this like two tier AI economy that's emerging between startups and corporations.
00:03:06.157 --> 00:03:11.608
So large organizations really kind of appear to be falling behind startups in terms of AI adoption.
00:03:12.116 --> 00:03:17.975
startups move very quickly, they're able to innovate rapidly, they try new things frequently.
00:03:18.353 --> 00:03:23.193
But, you know, big corporations, conversely, have a tendency to get stuck in like red tape.
00:03:23.596 --> 00:03:32.425
and they may have money and resources, but pouring money and resources into this AI challenge isn't really what is solving the challenges.
00:03:32.530 --> 00:03:36.139
you know, startups are experimenting really fast with these.
00:03:36.435 --> 00:03:38.375
and I think they're able to do it a lot more efficiently.
00:03:38.375 --> 00:03:45.365
So you're seeing AI adoption take off within startups, but lag behind quite a bit in larger corporations.
00:03:45.592 --> 00:03:47.193
we work for a small company.
00:03:47.252 --> 00:03:55.633
we're already using, agentic workflows, with AI to, optimize some of the stuff we do for the show, for polling and research.
00:03:55.992 --> 00:04:03.212
we're using, this new workflow we just built to collect research for newsletter articles that will be coming out over the next months and weeks.
00:04:03.712 --> 00:04:05.432
This is true from our perspective.
00:04:05.483 --> 00:04:09.752
we work at a startup and we are very rapidly innovating with AI.
00:04:10.687 --> 00:04:13.657
Indeed, and it's about being able to iterate very quickly.
00:04:13.979 --> 00:04:20.608
Sometimes if you're in a larger organization, your cycles are just so much longer naturally that can be hard to get the momentum needed.
00:04:20.879 --> 00:04:23.738
to build something that's changing so quickly, like AI.
00:04:24.108 --> 00:04:28.858
And I'm excited for what these kinds of things will unlock for us to help us work more efficiently.
00:04:29.168 --> 00:04:39.619
I also think it calls out the, real root of where the AI innovation comes from, which is, Scrappy teams making resourceful decisions with what's available to them in the time they have.
00:04:39.908 --> 00:04:57.749
And I think it's going to cause us to see a lot more diversity in the kinds of problems that get solved with technology by startups, because they have the ability to move really fast within regions and in industries that maybe are traditionally, managed by those large traditional enterprises and that can't move as quickly.
00:04:58.702 --> 00:04:58.973
Yeah.
00:04:58.973 --> 00:05:05.543
This speaking of something that we're probably going to have to iterate on very soon, the tech interview process.
00:05:05.622 --> 00:05:08.213
Tell me what you read about this week on that, Andrew.
00:05:08.838 --> 00:05:16.418
Yeah, I read this amazing article from Kane Naraway about how the AI has killed the interview process for tech folk.
00:05:16.809 --> 00:05:26.809
And it really just kind of shines a light on everything that we already understand and know about the reality of applying and interviewing for jobs in an AI world now.
00:05:27.115 --> 00:05:35.716
The idea that one, you're often tested on things that maybe are expected to be used alongside AI, and maybe the interviews are not testing.
00:05:35.940 --> 00:05:39.730
You know, your AI capabilities, which is what you'd actually be using on the job now.
00:05:40.031 --> 00:05:47.923
But on the inverse of that, there are so many opportunities and ways for AI to get between you and the interview, inject its knowledge into the process.
00:05:47.923 --> 00:05:52.853
So if someone's interviewing you, are they really interviewing your skills or are they interviewing Claude's?
00:05:53.103 --> 00:05:56.074
You know, that becomes the question of the article.
00:05:56.694 --> 00:06:01.338
And it's really shining a light on how Rehiring practices, they need to change.
00:06:01.519 --> 00:06:16.329
Just like how the day to day work of developers is evolving to be more like a manager of AI, to be in charge of these workflows and processes, to maybe not be as writing as much code, but definitely be overseeing a large technical, project or architecture.
00:06:16.559 --> 00:06:22.415
Those are the things that we should be testing and testing them alongside their ability to use, AI tools.
00:06:22.785 --> 00:06:32.144
Yeah, I mean, I feel like at a minimum, everyone should be incorporating AI questions into their interview process to figure out how candidates are using it.
00:06:32.144 --> 00:06:35.055
Like, have they experimented in positive ways with it?
00:06:35.064 --> 00:06:36.194
Have they seen success?
00:06:36.524 --> 00:06:42.185
I think in the future, like really every engineer needs to become product oriented.
00:06:42.475 --> 00:06:44.694
And I think the sooner you can do that, the better.
00:06:45.439 --> 00:06:51.519
And what I mean by that is it's not enough to just understand code and to be able to generate code anymore.
00:06:51.560 --> 00:06:57.250
You need to understand how or have an awareness of how that code impacts the end users.
00:06:57.250 --> 00:07:08.550
So even if you're making simple database updates, you should know how that database is used in real world conditions so that you're optimizing it for how it's actually going to be used.
00:07:09.050 --> 00:07:22.170
But I think beyond that, you know, some things that maybe organizations should start thinking about today are like how to lean into the human side of the experience, you know, if you can like in person interviews may actually make a lot of sense now to bring back.
00:07:22.670 --> 00:07:27.040
but even if you can't, I mean, make AI a part of the interview process.
00:07:27.040 --> 00:07:33.879
Like if you're going to do technical, challenges or technical, interviews, you should make AI a part of that.
00:07:33.889 --> 00:07:35.949
Like how much, how can they use AI?
00:07:36.230 --> 00:07:39.970
How can they leverage it to solve higher order challenges?
00:07:40.055 --> 00:07:43.915
you don't need to just Test them on their ability to produce code anymore.
00:07:44.745 --> 00:07:48.915
But it also makes me wonder, like, is LeetCode dead?
00:07:48.915 --> 00:07:55.615
Yeah,
00:07:55.757 --> 00:08:04.307
the new, the leet code of tomorrow will probably be more about testing your ability to prompt and use AI tools to solve the kinds of problems that were originally on leet code.
00:08:04.776 --> 00:08:08.817
But ultimately, what leet code is testing isn't maybe something that needs to get tested.
00:08:08.817 --> 00:08:17.944
Um, I think it really harkens back to, what's the goal of asking those technical questions in an interview of a candidate?
00:08:17.954 --> 00:08:19.533
It's because you want to see their skills.
00:08:19.533 --> 00:08:28.863
Well, the skills you want to see now are their ability to work with the tools that are going to produce that kind of content, as well as be able to understand the fundamentals behind why it's working.
00:08:29.064 --> 00:08:34.774
So really, it's about going back to the reason that we do interviews and, focus on the human elements that you can actually understand.
00:08:35.442 --> 00:08:39.452
this article, like specifically called out Max Howell, who's the creator of Homebrew.
00:08:39.763 --> 00:08:57.692
Which if you're a developer who works on Mac, there's like a 90 plus percent chance that you're using something that this guy created and how, you know, he made it, he probably can't even get hired at Google because, you know, even though most of their company probably uses the stuff he built, if he can't solve leak code, he may not get in the door.
00:08:57.692 --> 00:09:03.102
So, you know, I think in the future, people like Max Howell are the ones that are really going to Excel,
00:09:03.931 --> 00:09:04.701
Absolutely.
00:09:05.191 --> 00:09:13.230
this kind of actually touches on something very similar that's evolving and changing within the tech industry right now, especially for entry level software, developers.
00:09:13.581 --> 00:09:20.941
When they're coming on the scene, it's a lot of times the way that they've acquired their knowledge or that they start to learn on the job is, you know, by interfacing with people.
00:09:21.105 --> 00:09:26.105
Technologies like AI, they're able to get fast answers based upon the context they're working on.
00:09:26.385 --> 00:09:38.206
But when you compare that to the last decade or two, if you've become a developer, you probably spent a fair amount of time on Stack Overflow trying to find an obscure answer to your very obscure question.
00:09:38.596 --> 00:09:42.846
But as we know, now you can ask those kinds GPT even, and get a response.
00:09:42.855 --> 00:09:45.466
So, there was an article this week about how.
00:09:45.745 --> 00:09:56.375
You know, the lack of the struggle that you might go through as an individual developer to learn those things and go on Stack Overflow and find the 30 questions and answers that don't answer your question.
00:09:56.385 --> 00:09:59.806
Like, that's the struggle of understanding the fundamentals of what you're writing.
00:10:00.135 --> 00:10:10.145
Yeah, this is, I think you're referencing this article that I brought up in one of our channels about, how new junior developers appeared to possibly actually be losing the ability to code.
00:10:10.211 --> 00:10:16.351
and this kind of made me think like, Andrew, have you ever met a developer that doesn't know what Stack Overflow is?
00:10:16.918 --> 00:10:21.677
No, that's kind of what I'm thinking of immediately is it's so ubiquitous.
00:10:21.802 --> 00:10:38.192
Yeah, I mean, you can bring up something that vibes in like the stack overflow nature and any developer knows what that experience is like, where you have loads of questions that are only tangentially related and a bunch of surly people who are providing very pointed and sometimes aggressive answers.
00:10:38.697 --> 00:10:39.118
Yes.
00:10:40.052 --> 00:11:02.351
But, I mean, that's changing, you know, like, sort of grew up, for lack of a better phrase, in an environment where everyone was cutting their teeth on Stack Overflow, but now, like, when a developer encounters a problem, they can just simply paste that error into one of the many generative AI tools that are on the market and instantly get an answer that is directly relevant to them.
00:11:02.461 --> 00:11:05.652
so I'm kind of like, you know, forget calculators in your pocket.
00:11:06.366 --> 00:11:10.846
Now you've got an encyclopedia in your pocket, and not only that, you can have a conversation with it.
00:11:10.856 --> 00:11:17.380
Like, that's, it completely changes, like, the paradigm of how we solve problems, Yeah,
00:11:17.447 --> 00:11:17.957
point.
00:11:18.018 --> 00:11:29.977
I think that's the most powerful experience that I have with AI so far, is not just asking the question and getting the answer, but then continuing in a dialogue to ask other questions, to do a follow up.
00:11:29.998 --> 00:11:40.278
This is why perplexity is really Successful is because when you go to search, you know, you're not trying to magically pull up a list of things, you're trying to actually find your answer.
00:11:40.278 --> 00:11:54.227
So when you start by asking your question, and you're having a dialogue, it's a really powerful experience that becomes deeply personalized, just like how every conversation or even this one right now we're having it, you know, it's personalized, we're having our own discussion.
00:11:54.868 --> 00:11:58.488
When you are able to kind of introduce knowledge in that way, that's how we learn as people.
00:11:58.908 --> 00:12:16.376
So it's a really powerful way to kind of Build that knowledge over time, and really what it calls out to me is if you are building things with AI and you are new to technology maybe, and you're exploring how to implement things is to also balance those inquisitive moments on different levels.
00:12:16.506 --> 00:12:22.787
ChatGPT with going and doing investigations of other people's, you know, sharings and buildings online.
00:12:23.076 --> 00:12:31.006
I think that's a really great opportunity to kind of, have that communal developer experience, share knowledge, and it'll get us all out of our silos as well.
00:12:31.216 --> 00:12:41.236
And you're always going to learn something if you can saddle up to a, you know, a mentor developer who's able to kind of shine a light on why those struggles are so important to building that knowledge.
00:12:42.841 --> 00:12:48.441
it makes me, like, wonder, like, are, are we sacrificing knowledge gain for speed?
00:12:48.551 --> 00:12:50.331
So AI lets you move faster.
00:12:50.350 --> 00:12:58.145
Are we sacrificing the ability to learn So yeah, so the author of this article that I mentioned does have some great recommendations.
00:12:58.155 --> 00:13:03.395
So among other things, it's, you know, bring a learning mindset, like you should interrogate your AI.
00:13:03.576 --> 00:13:07.105
You shouldn't just accept answers that it gives you at face value.
00:13:07.535 --> 00:13:12.336
but sort of touching on, you know, the more humanistic aspects of our previous story on interviews.
00:13:12.905 --> 00:13:48.466
you know, there's a lot of human, Places or places for human discussion out there that aren't overwhelmed by bots and GPT generated stuff You know, there's lots of communities out on like discord and reddit so I think affinity groups are becoming like more important than ever like we can all learn from GPTs, but there's still a lot of value in learning from other humans as well but then there's also some recommendations like bring more conversation into code reviews, so Don't focus as much on whether or not the code works, but more on why the approach to that was taken in the first place.
00:13:48.927 --> 00:13:51.116
and then, you know, just occasionally build things from scratch.
00:13:51.116 --> 00:13:53.397
Like it's, it's fun to make stuff from scratch.
00:13:53.407 --> 00:13:56.476
I love to cook, you know, a lot of times I buy stuff from the store.
00:13:56.871 --> 00:14:03.522
But sometimes it's better to just do it yourself because you learn something along the way and you take that knowledge with you into the future.
00:14:03.522 --> 00:14:05.322
So, you know, it's not all bleak.
00:14:05.322 --> 00:14:13.981
It's not like we're gonna suddenly forget how to code, but, you know, I think we do have to take a different approach to learning in this environment.
00:14:14.342 --> 00:14:15.533
Yeah, I don't think it's bleak at all.
00:14:15.533 --> 00:14:20.462
It's just more like a, it's a new level of abstraction that will just require new skills.
00:14:20.912 --> 00:14:23.552
Yeah, So, Andrew, tell me about our guest today.
00:14:24.023 --> 00:14:30.302
after the break, we're diving into some amazing insights from a recent event that Dev Interrupted had with LinearB.
00:14:30.312 --> 00:14:35.582
This is in San Francisco and covering our 2025 Software Engineering Benchmarks Report.
00:14:36.023 --> 00:14:56.451
And so what we're going to be listening to is What happens when you get Rob Zuber, the CTO of CircleCI, and Tara Hernandez, the VP of Developer Productivity at MongoDB, together in a renovated church with a live audience and a bunch of technical experts and Well, you get magic, So stick around, after the break.
00:14:56.461 --> 00:15:01.522
We're gonna have a round table discussion with these industry leaders on dev productivity and experience.
00:15:05.570 --> 00:15:08.750
Let's talk about something that's been on every engineering leader's mind.
00:15:08.750 --> 00:15:12.350
How do you measure success beyond just Dora metrics?
00:15:13.120 --> 00:15:19.279
That's exactly what LinearB tackles in their latest guide on how top teams track developer access beyond Dora.
00:15:19.884 --> 00:15:28.484
you're relying on deployment frequency and lead time or cycle time, you're missing the full picture of how engineering actually drives business impact.
00:15:28.953 --> 00:15:31.673
This guide breaks down real world metrics that matter.
00:15:31.823 --> 00:15:37.953
Stuff like developer satisfaction, cognitive load, and engineering impact beyond just shipping fast.
00:15:38.663 --> 00:15:41.693
Because what's the point of going fast if you can't see where you're going?
00:15:42.193 --> 00:15:46.033
If you're serious about improving your team, go grab your free copy now.
00:15:46.094 --> 00:15:47.403
The link's in the show notes.
00:15:49.539 --> 00:15:59.639
We're going to get started with our first session today titled software engineering benchmarks and insights round table.
00:16:00.144 --> 00:16:04.374
Now we have a wonderful agenda for this session.
00:16:04.423 --> 00:16:08.244
We're going to look at a bunch of different insights, five of them.
00:16:08.714 --> 00:16:13.394
But before we get there, I want to introduce our amazing.
00:16:13.999 --> 00:16:20.958
Roundtable guests, starting with Tara Hernandez, VP of Dev Productivity at MongoDB.
00:16:21.428 --> 00:16:22.168
Welcome, Tara.
00:16:22.639 --> 00:16:23.139
Hello.
00:16:23.812 --> 00:16:29.472
And of course, we also have Rob Zuber, CTO of CircleCI.
00:16:30.113 --> 00:16:30.653
Welcome, Rob.
00:16:31.346 --> 00:16:32.196
Hello, everybody.
00:16:32.895 --> 00:16:34.216
It's really weird being up here.
00:16:34.255 --> 00:16:36.265
It's just like a bunch of bright lights in my face.
00:16:36.706 --> 00:16:39.375
And all I can see is glowing Linear B things.
00:16:39.456 --> 00:16:40.346
But they're very cool.
00:16:40.525 --> 00:16:42.216
And a really big pipe organ.
00:16:42.216 --> 00:16:42.296
Welcome, everyone.
00:16:43.346 --> 00:16:44.426
Yeah, can we play the organ?
00:16:45.716 --> 00:16:50.096
You know what I told them is make this look like a nineties roller rink.
00:16:50.326 --> 00:16:52.105
And I think they executed it really well.
00:16:52.105 --> 00:16:56.355
Maybe we can play the organ, but, uh, amazing to have you both here.
00:16:57.076 --> 00:16:57.745
Free bird.
00:17:02.719 --> 00:17:06.419
So for today's agenda, we're going to look at five insights.
00:17:06.439 --> 00:17:13.588
We're going to look at the PR lifecycle, the PM hygiene, Dora metrics, code quality, and predictability.
00:17:14.019 --> 00:17:15.818
This is all coming from.
00:17:16.307 --> 00:17:20.124
The 2025, engineering report that you should all have.
00:17:20.663 --> 00:17:24.773
And just a little bit of information about the report before we get started.
00:17:24.784 --> 00:17:28.773
So on the left hand side, you'll see the population size.
00:17:29.253 --> 00:17:30.784
So we looked at about 6.
00:17:30.784 --> 00:17:38.294
1 million poll requests, over 3, 000 organizations, 32 countries, you have a bunch of different calculation types in there.
00:17:38.358 --> 00:17:43.528
P50, P75, new for this year is P90 and average.
00:17:44.189 --> 00:17:47.709
This is the fourth generation of the report.
00:17:47.999 --> 00:17:51.419
And this year we have seven new metrics that we've added.
00:17:52.219 --> 00:17:56.028
The first two, approve time and merge time.
00:17:56.028 --> 00:18:01.909
That's looking a little bit more at developer experience, breaking down review time, getting a little more granular.
00:18:02.492 --> 00:18:07.893
We added PR maturity, and then we added a bunch of different ones here.
00:18:07.932 --> 00:18:20.563
Issues linked to parents, branches linked to issues, in progress issues with estimations, in progress issues with assignees, these are all what I consider predictability or predictable delivery metrics.
00:18:21.229 --> 00:18:23.808
Let's start with PR maturity.
00:18:24.159 --> 00:18:37.479
So if we think about PR maturity, think about putting up a pull request and maybe it has a hundred lines of code of change in that PR and what PR maturity looking at is the ratio of the amount of changes.
00:18:37.479 --> 00:18:39.409
So let's say that you had 20 different.
00:18:39.749 --> 00:18:42.298
Lines that were changed after the PR was open.
00:18:42.298 --> 00:18:45.545
You'd have a PR maturity, ratio of 80%.
00:18:45.951 --> 00:18:49.872
What are your thoughts on that new metric that was added PR maturity?
00:18:50.553 --> 00:18:52.212
We might talk about this for an hour.
00:18:52.799 --> 00:19:01.440
one of the things that I think about metrics in general is that there are a great way to understand.
00:19:01.765 --> 00:19:08.365
Where you should start looking, they're not going to give you all the answers to what's happening in your organization.
00:19:08.365 --> 00:19:09.785
They're going to say, Hey, that's interesting.
00:19:10.144 --> 00:19:11.795
Let's go ask a question about that.
00:19:12.365 --> 00:19:15.974
And what's really important there is what you're looking for.
00:19:15.974 --> 00:19:19.144
When you ask those questions is what is the surrounding context?
00:19:19.144 --> 00:19:21.775
Why do these numbers look the way they look?
00:19:21.775 --> 00:19:28.244
And the thing that I love about this and a few of these different metrics is you could describe a fantastic organization.
00:19:28.700 --> 00:19:32.609
And a terrible organization that has show the same numbers, right?
00:19:32.619 --> 00:19:38.410
So PR maturity, again, how much change is there after someone puts up a PR, right?
00:19:38.410 --> 00:19:42.029
The fantastic organization, you have teams pairing on code.
00:19:42.579 --> 00:19:45.069
By the time the PR goes up, two people have looked at it.
00:19:45.079 --> 00:19:48.299
They've built it together and someone goes, yep, let's put that in.
00:19:48.660 --> 00:19:49.339
No problem.
00:19:49.509 --> 00:19:55.420
You know, we've all agreed that this is the right thing in the less awesome organization.
00:19:55.759 --> 00:19:57.490
People are rubber stamping PRs.
00:19:58.609 --> 00:20:00.289
Nobody actually asks any questions.
00:20:00.289 --> 00:20:02.490
They just put them up and they're like, yeah, go for it.
00:20:02.490 --> 00:20:04.180
You know, it is like, Hey, it looks interesting.
00:20:04.180 --> 00:20:04.539
I don't know.
00:20:04.539 --> 00:20:06.230
I'm not, I don't really have time to read your PR.
00:20:06.509 --> 00:20:07.539
I got my own work to do.
00:20:08.170 --> 00:20:10.960
And like, I guess, you know, I'd always ask, which one of those do you want to work in?
00:20:11.559 --> 00:20:11.869
Right.
00:20:11.900 --> 00:20:16.799
So you can't just look at one of these metrics and say, my company is elite.
00:20:16.910 --> 00:20:18.269
My company is terrible.
00:20:18.599 --> 00:20:21.380
You can say that tells me something about what's happening.
00:20:21.380 --> 00:20:24.890
What else do I need to know to sort of triangulate?
00:20:25.460 --> 00:20:27.170
What's really happening in the organization?
00:20:28.009 --> 00:20:28.279
Yeah.
00:20:28.279 --> 00:20:40.119
You know, I was at reinvent last week'cause I think like half the living world, gosh, there was a lot of people in Vegas and there was a talk about, collaborative AI and this, this new concept that they, this the speaker was trying to hype up.
00:20:40.119 --> 00:20:48.279
and she said that really interesting, which was, you know, collaborative AI is when you recognize that AI has all of the data and knowledge that the human has the context.
00:20:48.309 --> 00:20:50.119
And I'm like, yeah, okay.
00:20:50.210 --> 00:20:51.079
I, I buy that.
00:20:51.079 --> 00:20:53.809
But that's actually true for just about anything.
00:20:53.839 --> 00:20:54.351
And I think it's.
00:20:55.140 --> 00:21:27.519
You know, certainly true for metrics, and then the example that I think I gave before that we liked is when I was at Google, I almost never made commits I was a manager, you know, don't touch the code it break something but there was a cleanup that we did We did a code refactoring It was like a million lines of code that was in the wrong spot and it was just sitting there And I'm like, I'm gonna go clean that up So I made a PR that was a negative 900 million lines of code I could assure you that it was not reviewed thoroughly, but it was the most exciting like Button press I've ever made in my life into GitHub.
00:21:27.859 --> 00:21:34.660
And you know, what would that, you know, if, if without context, that would have been like, you just deleted our, the entirety of our intellectual property, right?
00:21:34.690 --> 00:21:36.220
Like, or some other responsive thing.
00:21:36.220 --> 00:21:45.086
So I think that these metrics, PR maturity being, I think it's a good one, but it's like a piece of your Lego kit.
00:21:45.646 --> 00:21:49.196
As an engineering leader for what do you want to be true?
00:21:49.196 --> 00:21:50.457
And what do you want to value?
00:21:51.176 --> 00:21:51.527
Right.
00:21:51.807 --> 00:21:57.497
And I would argue, like, I think Rob has some good examples, but I would argue like other variations are, where are you in your career?
00:21:57.537 --> 00:22:02.527
Like as an engineering manager, you want, might want a lot of visibility on what your junior engineer is doing.
00:22:03.076 --> 00:22:10.866
And you see this long conversation in a GitHub PR about giving that person feedback, and now it's stored as part of the commit history, right?
00:22:10.886 --> 00:22:20.277
Whereas the senior engineer, you might see a couple of things cause you know, they had a. Really like deep whiteboard session at some point with the staff engineer and it came in pretty clean.
00:22:20.547 --> 00:22:30.747
So again, I think it's, you know, love the concept of it, but I think it's, you know, we really need to think about what do we do with it now that we're measuring it is the, it's kind of the important next step.
00:22:31.096 --> 00:22:33.553
Yeah, I think, well, first of all, you know, I love that story.
00:22:33.952 --> 00:22:38.242
That's a great story, but for all of these metrics and for our audience listening today.
00:22:38.633 --> 00:22:41.042
The takeaway here is context matters a lot.
00:22:41.913 --> 00:22:43.303
So you're going to look at these metrics.
00:22:43.303 --> 00:22:45.313
We're going to look at some of these insights.
00:22:45.313 --> 00:22:54.623
We're going to talk about the benchmarks, but think for yourself, if you're sitting in the audience and listening, do I know the metrics for my organization?
00:22:55.230 --> 00:22:56.339
Can I say what they are?
00:22:56.904 --> 00:23:05.654
You are the ones that understand the context of your business, your project, what you're looking to deliver, the seniority of your team, the junior ness of your team.
00:23:06.359 --> 00:23:16.680
And I think what's great about everything that we're going to talk about today is that it opens up conversation to ask the right questions of yourself, of your team, and of your business.
00:23:17.046 --> 00:23:17.425
go ahead.
00:23:17.715 --> 00:23:29.756
So I think there's another aspect of this that we've been talking about recently, and I'd love to see your reaction to this idea, which is a lot of time we think about like developer velocity metrics, like how fast your PR is going through, how long does it take you to do your code reviews?
00:23:30.105 --> 00:23:33.826
You know, how long does it take you to write your tech spec, execute, et cetera, et cetera, et cetera, right?
00:23:34.384 --> 00:23:47.365
But if you think about that in the aggregate of a team, I realized that one of the things that I think is probably one of the most interesting aspects of this is how well it shows how well that team's leader is doing, right?
00:23:47.404 --> 00:23:52.058
If the team's leader is routinely, you know, they're responsible for what their team delivers, right?
00:23:52.058 --> 00:24:00.011
So if they are missing a lot of, the team's missing a lot of deadlines, maybe their engagement survey scores aren't so great, There's quality problems, there's low test rate.
00:24:00.961 --> 00:24:07.736
Maybe that's a piece of a leadership puzzle, part of your toolkit rather than a developer piece.
00:24:08.394 --> 00:24:17.335
think a lot, talk a lot, I think, in this industry, not just at CircleCI, about kind of the delivery unit as the team, measuring velocity as a team.
00:24:17.825 --> 00:24:27.734
the way that I think about that is, to your point of the manager, I expect the team to be effective and I care about the team's delivering, but then what happens inside the team?
00:24:28.079 --> 00:24:28.289
Right.
00:24:28.289 --> 00:24:31.299
If one person is not contributing, that's a concern of the manager.
00:24:31.660 --> 00:24:35.509
And then I guess your point is like, if the whole team is not delivering, that's probably a manager issue.
00:24:35.509 --> 00:24:36.559
It's not an individual issue.
00:24:37.130 --> 00:24:39.950
of course, other dynamics and organizations we could go on forever, but
00:24:40.289 --> 00:24:41.640
everything depends, right?
00:24:41.690 --> 00:24:42.289
Exactly.
00:24:42.605 --> 00:24:52.005
But I think the resolution, like being able to see below the level of the team to me is the responsibility of the manager, meaning.
00:24:52.422 --> 00:25:00.481
I can't, I can't see that much from the outside, but from the outside it's really hard to see the dynamic that created the scenario, right?
00:25:00.751 --> 00:25:05.672
Okay, yes, this, your team is not delivering as much value as another team, or you're missing your deadlines.
00:25:05.672 --> 00:25:07.021
You're unpredictable and your delivery.
00:25:07.021 --> 00:25:12.411
You have lots of incidents, whatever you care about, but there's some dynamics inside the team that's causing that to happen.
00:25:13.155 --> 00:25:16.346
And to me, it's the manager's responsibility to figure that out.
00:25:17.256 --> 00:25:20.536
And having that signal, right, is really kind of a key indicator.
00:25:20.996 --> 00:25:25.266
We probably could have spent an hour just on this first, metric.
00:25:25.266 --> 00:25:26.131
Keep us going,
00:25:26.131 --> 00:25:26.548
Dan.
00:25:26.548 --> 00:25:27.800
I'm going to
00:25:27.978 --> 00:25:31.375
us forward and actually start looking at some of the insights.
00:25:31.375 --> 00:25:33.155
We have five or six to go over here.
00:25:33.905 --> 00:25:38.006
And the first insight is around the pull request lifecycle.
00:25:38.006 --> 00:25:40.246
So I'm going to read the key, takeaway here.
00:25:40.905 --> 00:25:49.316
PR size, so pull request size is the most significant driver of velocity across the PR lifecycle.
00:25:49.935 --> 00:25:59.145
And what the little graph is showing you here is as PRs get larger, cycle time or velocity also gets larger.
00:25:59.185 --> 00:26:06.205
And as PRs get smaller, your mirrored merge frequency increases and your cycle time decreases.
00:26:06.236 --> 00:26:07.905
Now I love this insight.
00:26:07.905 --> 00:26:11.125
It's actually when I'm working with customers, usually the first one.
00:26:11.540 --> 00:26:18.290
That we talk about around PR size and usually when I come in and work with a customer, the PR size is a little bit large.
00:26:18.766 --> 00:26:21.476
And it's not even about the PR size being large.
00:26:21.476 --> 00:26:25.596
It's about what I was saying earlier with the context of why, what is causing it.
00:26:26.175 --> 00:26:35.586
There could be a lot of different things that are causing that ranging from a, you know what, we're working in a situation where we only are releasing once every month.
00:26:35.586 --> 00:26:40.895
So developers try to get everything into a PR or something like that to make sure it goes out.
00:26:40.895 --> 00:26:43.215
Otherwise they'll mix the next release.
00:26:43.246 --> 00:26:45.395
Or it could be something that has to do with.
00:26:45.905 --> 00:26:46.665
technical debt.
00:26:46.675 --> 00:26:57.726
Every time that I try to change something within this code base, I actually have to touch five different files and change all this stuff, even though it seems like a really, really small, request of me.
00:26:57.786 --> 00:27:00.155
I'm actually having to touch a large amount of code.
00:27:00.736 --> 00:27:09.516
when you hear this takeaway of PR size is the most significant, driver of velocity across the PR lifecycle, what comes to mind for the two of you?
00:27:10.040 --> 00:27:17.127
I think the first thing is the word driver and trying to, you know, rep on behalf of my DSA team.
00:27:17.127 --> 00:27:18.657
Like, I think there's correlation.
00:27:18.657 --> 00:27:20.988
I don't know that we necessarily have causality there.
00:27:20.988 --> 00:27:25.107
Like there are feedback loops in software development that work both ways.
00:27:25.621 --> 00:27:35.020
You know, I think this is trying to say PR size leads to reduced velocity, but I would argue that reduced velocity leads to large PR size.
00:27:35.020 --> 00:27:35.181
Right.
00:27:35.181 --> 00:27:37.661
If people are slow.
00:27:38.121 --> 00:27:48.681
To review and give me feedback, or it's expensive to get a PR through the process, then I will jam more stuff into the PR and then I get this negative feedback loop.
00:27:48.681 --> 00:28:06.351
I make it bigger, people take longer, so I make it bigger, so people take longer, and now we're into month long PRs instead of like, like I've worked in teams where, you know, we would not even mention anyone's name, just write the PR, drop a link into Slack or HipChat back in the day.
00:28:07.355 --> 00:28:10.806
And someone would just review it in three minutes because they knew exactly what I was working on.
00:28:10.915 --> 00:28:12.635
They had the full context, right?
00:28:12.635 --> 00:28:14.776
So it can tell you something else about your organization.
00:28:14.776 --> 00:28:15.895
Again, everything is context, right?
00:28:16.905 --> 00:28:21.675
Are the people showing up to review saying, I don't know anything about this.
00:28:21.945 --> 00:28:27.405
Now I need to sit down and understand everything that surrounds this change, right?
00:28:27.465 --> 00:28:30.826
Therefore, I'm only going to do this on my coffee break in the morning.
00:28:31.276 --> 00:28:38.486
Versus, I could stop what I'm doing, review this, agree to it, and move on in a very short period of time.
00:28:38.875 --> 00:28:43.645
And so I think the, there's like organizational culture and structure that drives large PRs.
00:28:44.276 --> 00:28:48.905
Versus always being large PRs driving that organizational structure.
00:28:49.726 --> 00:28:51.056
100 percent agree on that one.
00:28:51.105 --> 00:29:02.885
I think that's a, it may be non obvious to some, but you know, the large PR side, it's like, what bureaucracy do you have that's causing your developer pain, right?
00:29:02.885 --> 00:29:07.675
And especially in your example, you think that developers had to re merge 14 times, now they're angry, right?
00:29:07.705 --> 00:29:09.855
So God only knows what's going into those updates.
00:29:10.724 --> 00:29:17.434
I mean, you talked about, I think a good situation where there's a shared context within the development team.
00:29:17.805 --> 00:29:20.835
But I think you said, Hey, you know, let's say that I was making a change.
00:29:20.835 --> 00:29:23.305
You already knew that I was going to be working on that.
00:29:23.305 --> 00:29:27.894
So when the PR goes up, you would say, Oh yeah, I knew Dan was doing that.
00:29:27.904 --> 00:29:29.025
I have all the context.
00:29:29.025 --> 00:29:30.565
I could review this really quick.
00:29:31.339 --> 00:29:43.759
Now, I think on the flip side in a situation where there's not a lot of context between the teams, so maybe the one team is working on many different things in parallel and also the PR size is large.
00:29:43.880 --> 00:29:46.430
That doesn't sound like a fun review to me.
00:29:46.549 --> 00:29:52.170
If I'm a developer, I might say, you know what, I'm going to wait till, I don't know, Friday to do this.
00:29:52.720 --> 00:29:55.829
That's Monday, or at least wait to the next day.
00:29:56.670 --> 00:29:57.720
I'd rather be coding.
00:29:57.720 --> 00:30:01.730
I'd rather be working on something on my own than unpack this enormous PR.
00:30:01.730 --> 00:30:02.839
So I'm not going to pick it up.
00:30:04.019 --> 00:30:05.049
Yeah, I think that's exactly right.
00:30:05.049 --> 00:30:07.579
Like I think, you know, I'm thinking whip limits, right?
00:30:07.619 --> 00:30:11.980
I mean, it's kind of nuts how quickly you get from PR size to whip limits or whatever, right?
00:30:11.980 --> 00:30:13.900
Like whatever, whatever other tool we reach for.
00:30:13.900 --> 00:30:20.365
But, and I think as, as leaders, I don't know the makeup of the audience, but I know someone's laughing at this as I say it like.
00:30:21.036 --> 00:30:22.756
We're like, well, I have six engineers.
00:30:22.756 --> 00:30:25.776
That means I could do six things at the same time, right?
00:30:26.310 --> 00:30:29.090
And someone is saying, but we pair and that's more productive.
00:30:29.181 --> 00:30:30.310
And you're like, that doesn't make any sense.
00:30:30.310 --> 00:30:32.000
Two people are doing the work of one person.
00:30:33.480 --> 00:30:35.911
This is where that matters, right?
00:30:36.431 --> 00:30:45.151
Two people driving up or even working kind of in the same area so that they have, like, they're thinking about that problem, right?
00:30:45.151 --> 00:30:46.721
You and I are working on similar things.
00:30:46.961 --> 00:30:48.050
You're thinking about the same problem.
00:30:48.050 --> 00:30:49.530
And I say, Hey, I'm going to make this change here.
00:30:49.570 --> 00:30:50.540
Like that totally makes sense.
00:30:50.540 --> 00:30:54.141
I was just thinking about that versus I haven't looked at that code base in a year.
00:30:54.800 --> 00:30:57.750
Now I need to go reread this whole chunk of the code base kind of thing.
00:30:57.750 --> 00:30:57.961
So like.
00:30:58.665 --> 00:31:23.865
The, again, to the point of it being a sign of the dynamics that are happening in your organization more than just because, because what happens a warning to all you leaders out there, what happens when you read something like this, the simple version is like, we're going to tell all of our engineers, they're not allowed to have more than 20 lines in a PR those PRs will be terrible.
00:31:24.516 --> 00:31:24.695
Right.
00:31:24.695 --> 00:31:29.875
It's going to be like every single one will be exactly 20 lines and you'll be like, what does this do?
00:31:29.875 --> 00:31:31.605
Well, there's another 20 lines coming.
00:31:32.266 --> 00:31:32.486
Right?
00:31:32.516 --> 00:31:33.955
Like, it's nonsense.
00:31:33.965 --> 00:31:36.175
I'm pretty sure there's a Dilbert cartoon about this.
00:31:36.855 --> 00:31:42.026
We all live in a Dilbert cartoon for sure, but like, you can't, you can't drive from the number, right?
00:31:42.036 --> 00:31:42.576
That's the point.
00:31:42.586 --> 00:31:43.836
This is the beginning of a conversation.
00:31:43.846 --> 00:31:45.556
You go say, okay, what are we doing?
00:31:45.556 --> 00:31:46.756
Like, where are we falling down?
00:31:46.756 --> 00:31:48.976
Why are we working on too many different things?
00:31:49.405 --> 00:31:49.766
Right?
00:31:50.026 --> 00:31:52.586
and so I, I just think there's a ton that comes out of this.
00:31:53.553 --> 00:31:56.613
what you made me think of also, if I'm a team leader.
00:31:57.208 --> 00:31:58.998
Or maybe I'm planning my sprint.
00:31:59.057 --> 00:32:04.958
Maybe we would be more inclined to say, Hey, let's tackle a particular problem together this sprint.
00:32:04.958 --> 00:32:08.067
So everyone has the same context when the PR goes up.
00:32:08.117 --> 00:32:11.788
Because the other way to do it is say, okay, I have six developers on the team.
00:32:11.788 --> 00:32:14.278
I'm going to go send you off on six different missions.
00:32:14.278 --> 00:32:18.048
Well, you know, probably your review time is going to be larger.
00:32:18.048 --> 00:32:18.444
And you
00:32:18.444 --> 00:32:21.773
better hope you don't hit it, find any buses near your team.
00:32:21.773 --> 00:32:22.050
Right.
00:32:22.050 --> 00:32:24.548
So, you know, single points of failure or anything.
00:32:25.097 --> 00:32:44.807
I think that the other thing I'm actually curious about, and I remembered it after the last, when we did the webcast, which is, we asked this question, like what, what constitute a pull request, like, so for the context, for example, of, really tight teams moving quickly, you know, how much of that code and how much of the culture is around developer documentation, right?
00:32:44.807 --> 00:32:50.387
You know, is there a markdown that's included in that and, you know, drive and maybe the, you know, there's more documentation.
00:32:50.387 --> 00:32:50.462
Yeah.
00:32:50.772 --> 00:32:53.663
Or maybe there's more documentation plus unit tests than there is code, right?
00:32:53.663 --> 00:33:11.123
So I think, again, there's another sort of contextual aspect of this metric that might be kind of interesting, to think about because in, you know, my ideal world, the PRs are of reasonable size for the, whatever it is that they're solving for, but also include some tests and some documentation and hopefully something like an architectural decision to reduce.
00:33:11.133 --> 00:33:13.432
So six months from now, when we go look at it, like, why did we do that?
00:33:13.462 --> 00:33:14.022
Oh, right.
00:33:14.633 --> 00:33:17.482
Because Rob said that we're going to do this thing.
00:33:17.813 --> 00:33:20.163
And so I think that, again.
00:33:20.667 --> 00:33:37.760
Loving the concept of, you know, how can you make this an actionable metric, you know, to drive the culture that you want to see, you know, maybe, you know, something like young developers, smaller scope problems, smaller PRs, bigger developers, probably touching more things, you're doing a big refactor, that's going to be huge, right?
00:33:37.770 --> 00:33:46.847
How can you, but then, you know, that the outcome of that is then your subsequent PRs will start to get smaller again, because now you don't have to touch, you know, at Netscape, we had this one header file, net.
00:33:47.048 --> 00:33:48.178
h, I'll never forget this.
00:33:48.522 --> 00:33:51.022
It had the world's largest global symbol table in it.
00:33:51.643 --> 00:33:57.182
It was so big, we had to have a custom compiler from SGI because it wouldn't compile in the, in the regular compiler.
00:33:57.353 --> 00:34:00.053
I'm saying this now, cause you know, everybody involved is now dead.
00:34:00.583 --> 00:34:03.393
Um, they're not dead, but they should be dead.
00:34:03.573 --> 00:34:04.103
Intrigue.
00:34:04.903 --> 00:34:06.563
It was a, it was a wretched, wretched thing.
00:34:06.563 --> 00:34:06.823
Right.
00:34:06.833 --> 00:34:09.293
And so, you know, the interns weren't allowed to touch that file.
00:34:09.793 --> 00:34:11.623
and you think, oh, well that was 30 years.
00:34:11.643 --> 00:34:12.313
Jesus Christ.
00:34:12.313 --> 00:34:13.163
That was 30 years ago.
00:34:13.612 --> 00:34:14.842
but we, we, we're humans.
00:34:14.842 --> 00:34:17.012
We keep making the same mistakes over and over again.
00:34:17.012 --> 00:34:17.273
Right.
00:34:17.322 --> 00:34:21.922
And so I think the interesting thing is like before we were leaning in on where, where did the leadership go into this?
00:34:22.362 --> 00:34:27.552
But I also wonder like, how can you make this directly actionable for that junior developer, right.
00:34:27.682 --> 00:34:29.753
For him to understand or him or her to understand what they're doing.
00:34:30.163 --> 00:34:35.862
I mean, regardless of causality or whatever, like tooling, right.
00:34:35.893 --> 00:34:39.471
Or the approaches I guess I would just share a story to say, but like.
00:34:40.340 --> 00:34:41.610
Feature flags, right?
00:34:41.681 --> 00:34:42.860
branch by abstraction.
00:34:43.030 --> 00:34:49.411
How do I make PRs that are not the whole unit of work, but allow people to keep up with my context?
00:34:49.411 --> 00:34:50.641
That's another approach, right?
00:34:50.641 --> 00:34:53.440
Like, I've got someone, maybe we're working on slightly different things.
00:34:53.440 --> 00:34:55.351
I'm like, hey, I'm thinking of doing this, I'm thinking of doing this.
00:34:55.601 --> 00:34:57.471
I'm testing it out, I'm getting feedback in production.
00:34:57.471 --> 00:34:59.001
These are all great things, right?
00:34:59.391 --> 00:35:04.411
Instead of, I need to write the whole thing, you know, it's kind of going back to the PR maturity thing.
00:35:04.755 --> 00:35:07.856
And then, oh, and then we're going to get into refactoring later.
00:35:08.195 --> 00:35:09.056
I'm so excited.
00:35:09.215 --> 00:35:10.115
Sorry, spoilers.
00:35:10.519 --> 00:35:16.179
but like, what tools do I have as a developer to actually write small PRs?
00:35:16.210 --> 00:35:25.280
Are we, coaching, are we training our younger developers on how to break down work in a way that isn't necessarily complete, but allows us to evaluate our ideas along the way?
00:35:25.280 --> 00:35:26.699
And I think there's lots of opportunity in there.
00:35:26.750 --> 00:35:27.500
Yeah, that's it.
00:35:27.922 --> 00:35:44.023
I'll push this forward, but that's exactly why I usually like starting with PR size when I go to work with a customer and for you all in the audience, like, even if you just know what your PR size is and you have a conversation with it about the team, you're going to discover something.
00:35:44.532 --> 00:35:53.452
Whether you want to make it smaller or you're okay with the size, like Rob's saying, they might say something back to you like, yeah, I'd love to have smaller PRs, but I don't trust our.
00:35:53.768 --> 00:35:56.717
Feature flagging capability, uh, whatever it is, we don't have it.
00:35:56.717 --> 00:35:57.748
I don't know how to use it.
00:35:57.757 --> 00:35:58.728
So my PRs are here.
00:35:59.463 --> 00:36:08.659
so a few of the supporting insights, if we move on, to the next slide, larger PRs wait longer to get picked up for review.
00:36:09.059 --> 00:36:10.469
I think it makes sense, right?
00:36:10.639 --> 00:36:12.909
Who wants a review in an enormous PR?
00:36:12.992 --> 00:36:13.422
no one.
00:36:14.322 --> 00:36:15.672
I'm going to save that for another day.
00:36:16.213 --> 00:36:19.652
Larger PRs have longer cycle times, probably for the same reason.
00:36:20.130 --> 00:36:22.420
Larger PRs take longer to approve.
00:36:22.650 --> 00:36:24.951
I think all of these pretty much correlate together.
00:36:25.860 --> 00:36:32.590
PRs that wait longer for the review to start also take longer from approval to merge.
00:36:33.280 --> 00:36:37.860
And larger PRs are modified more heavily during the review
00:36:38.418 --> 00:36:41.038
with my ginormous PR being an exception.
00:36:41.918 --> 00:36:42.128
Yeah.
00:36:42.128 --> 00:36:42.958
Besides you took two
00:36:42.958 --> 00:36:43.387
hours.
00:36:43.648 --> 00:36:44.427
There's an exception to
00:36:44.427 --> 00:36:44.708
every
00:36:44.708 --> 00:36:45.148
rule, but
00:36:45.177 --> 00:36:47.768
you did not build your plan on what was it?
00:36:47.768 --> 00:36:48.858
900, 000 line.
00:36:48.907 --> 00:36:50.947
Maybe it's something stupidly large.
00:36:52.469 --> 00:36:52.980
Let's move on.
00:36:53.030 --> 00:36:54.710
Oh, this one's funny.
00:36:54.849 --> 00:36:57.510
Let's move on to the next insight here.
00:36:57.510 --> 00:36:57.699
So.
00:36:58.480 --> 00:37:00.980
Project Management Hygiene.
00:37:01.300 --> 00:37:02.559
So, key takeaway.
00:37:03.440 --> 00:37:09.789
Poor project management hygiene is directly correlated with higher velocity.
00:37:09.789 --> 00:37:10.710
So, I'll say it again.
00:37:10.789 --> 00:37:16.289
Poor project management hygiene is directly correlated with higher velocity.
00:37:16.289 --> 00:37:18.610
You know what this made me think about, actually?
00:37:18.610 --> 00:37:20.449
Jira tickets are for suckers.
00:37:21.786 --> 00:37:23.476
I'm ordering my t shirt right now.
00:37:24.715 --> 00:37:25.695
Hackathons.
00:37:26.735 --> 00:37:27.326
Yeah, no.
00:37:27.876 --> 00:37:29.865
But why did it make me think about hackathons?
00:37:29.876 --> 00:37:36.505
Some of the you're usually working with a few developers that you probably, know pretty well.
00:37:37.364 --> 00:37:47.335
You're definitely, not going to JIRA and coming with your idea and saying, let me write a story before we go and work on this hackathon project.
00:37:47.684 --> 00:37:53.625
Yeah, that hackathons I've seen, developers, you know, not only come up with some of the coolest features.
00:37:54.144 --> 00:37:58.795
for the business, but move at a really rapid rate, you know, 24 hours.
00:37:58.815 --> 00:38:08.875
And I think, you know, when I think back to a hackathon, it's the situation where the developer is also the product manager in a sense.
00:38:08.875 --> 00:38:10.804
I don't, we have a shared context.
00:38:10.804 --> 00:38:12.485
I don't need to open a JIRA ticket.
00:38:12.485 --> 00:38:14.684
I don't need to explain everything that I'm doing.
00:38:15.894 --> 00:38:18.065
And therefore I move very, very quickly.
00:38:18.074 --> 00:38:19.815
Now, what's the downside?
00:38:20.355 --> 00:38:23.614
To putting hackathon code into production.
00:38:23.945 --> 00:38:26.414
Oh, that's a really good way to kill your company.
00:38:27.264 --> 00:38:28.445
Quality, right?
00:38:28.635 --> 00:38:37.594
So anyways, when I, when I saw this, I started to think about that type of hackathon mentality, but I'd love to hear what you two think about this key takeaway.
00:38:37.594 --> 00:38:37.945
I
00:38:38.358 --> 00:38:45.898
the more, you call it bureaucracy, you call it process, call it whatever, the more of it, whatever it is, it's going to slow things down.
00:38:45.907 --> 00:38:48.418
So going back to something that Rob was saying earlier.
00:38:48.913 --> 00:38:57.652
You know, how do we have a collective shared context that allows us as engineers to move forward with the lowest amount of friction, right?
00:38:58.322 --> 00:39:02.572
inevitably, as I mean, this really becomes ultimately a scale problem.
00:39:02.813 --> 00:39:05.552
A lot of people love working on the, in the tiny startups.
00:39:05.572 --> 00:39:08.472
You know, the engineering team is less than a hundred people, right?
00:39:08.483 --> 00:39:10.193
That's the greatest glory days.
00:39:10.193 --> 00:39:11.222
It's the greatest time of their life.
00:39:11.222 --> 00:39:19.492
And then if they stick with it, you know, and the company is successful, next thing you know, the engineering team is a thousand people and they're bemoaning their very existence, right?
00:39:19.492 --> 00:39:19.552
Right.
00:39:19.853 --> 00:39:20.293
why?
00:39:20.322 --> 00:39:31.092
Because the amount of thinking, you know, there's, there's probably an order of magnitude, more products and an order of magnitude, more enterprise customers that are picky and all the other things.
00:39:31.092 --> 00:39:35.643
And we got to slow down, So I would also argue, like I work for a company that makes databases.
00:39:36.393 --> 00:39:40.972
Let me tell you how often our enterprise customers want to upgrade their database.
00:39:41.463 --> 00:39:41.972
Never.
00:39:42.253 --> 00:39:42.532
Right?
00:39:42.532 --> 00:40:01.096
So that's a different kind of problem, but I do think there's a. You know, again, as you scale your organization, as you, as you grow your development team, keeping friction low, as low as you can, recognizing it will be as low as it was when you started, I think is a good goal to keep in mind.
00:40:01.106 --> 00:40:06.775
So, you don't let that tech debt get too crazy because what inevitably happens is your quality goes down.
00:40:07.110 --> 00:40:07.431
Right.
00:40:07.440 --> 00:40:14.440
And then the VPs are going, Oh, no, we got to bring in all of these scrum masters and we're going to do agile and everything will be better.
00:40:15.170 --> 00:40:15.391
Right.
00:40:15.391 --> 00:40:19.820
Sorry, I didn't mean to make you choke, but you know, that's what, you know, I mean, How long are we going to do
00:40:19.820 --> 00:40:21.701
the training for with the Agile?
00:40:22.081 --> 00:40:26.440
So, you know, not to bag on scrum masters and Agile, it's fine if you've, that's your thing.
00:40:27.177 --> 00:40:28.597
I'm not an Agile person.
00:40:29.081 --> 00:40:31.550
but I think that, you know, there's a balance.
00:40:32.101 --> 00:40:32.440
Right.
00:40:32.510 --> 00:40:37.143
What's sometimes even within a, let's say CircleCI has got this new thing.
00:40:37.153 --> 00:40:47.947
You're going to go out quickly, you know, go, go to market product testing, whatever, and then you're going to slow down as much as you have to in order to make sure that you meet your, your enterprise needs.
00:40:48.494 --> 00:40:54.224
I think there's another example that I would use similar to the hackathon is, we use this a bunch as incidents.
00:40:54.713 --> 00:41:07.210
And one of the things that I would say, like when I reflect on incidents, not that I wish them upon anyone, but I kind of like them or enjoy them anyway, but like you have clarity of purpose, right?
00:41:07.221 --> 00:41:08.420
You've shared context.
00:41:08.460 --> 00:41:11.840
Everyone knows there's one priority, right?
00:41:11.840 --> 00:41:14.371
It's what everyone says they want in their organization.
00:41:14.795 --> 00:41:19.275
But it only occurs during the most stressful time that anyone experiences in their organization.
00:41:19.356 --> 00:41:22.106
After you enter the war room together and get the context.
00:41:22.106 --> 00:41:28.045
Right, it's like, well, like, does anyone say, oh, but we also need to like worry about that typo over here?
00:41:28.056 --> 00:41:29.505
Everyone's like, why are you talking about that right now?
00:41:29.505 --> 00:41:31.206
Like we have one thing that matters, right?
00:41:31.385 --> 00:41:34.085
And that's what people kind of strive for.
00:41:34.235 --> 00:41:38.385
And so the question is like, can you create that?
00:41:39.016 --> 00:41:44.235
And I don't mean with like the caffeine fueled 72 hour stretch of insanity or whatever.
00:41:44.235 --> 00:41:44.376
But like.
00:41:45.061 --> 00:41:59.951
Can you create that quality of work, that sort of singular understanding of what matters more broadly than in those, either those hackathons, I mean, the hackathons are always funny to me because everyone's like, Oh yeah, well, no, we need to have a hackathon so we can do some innovation.
00:42:00.791 --> 00:42:04.541
I'm like, what are you doing the other 51 weeks of the year?
00:42:04.541 --> 00:42:07.320
Like we should probably just make innovation something that we do, right?
00:42:07.320 --> 00:42:09.981
Like the fact that we need exceptions.
00:42:10.630 --> 00:42:13.601
In order to be great at delivering feels weird.
00:42:14.070 --> 00:42:18.141
It feels like a problem to say, if that exists in your organization, like, please go address that problem.
00:42:19.380 --> 00:42:33.751
The other thing that I think about, about this particular metric is, I personally believe, I don't know if people are familiar with this meme, like, great project management hygiene is probably maximizing your mediocrity.
00:42:34.447 --> 00:42:40.518
Teams that are at the absolute bottom of the curve and have no idea what they're doing and are like, you know, cowboy coding or whatever.
00:42:41.074 --> 00:42:42.773
They'd probably rank poorly on this.
00:42:43.233 --> 00:42:50.204
But the best teams that I've ever worked on have not been that concerned about whether they put the right JIRA ticket number in the PR.
00:42:50.681 --> 00:42:55.630
So that everyone could follow the flow because they knew exactly what they were trying to deliver.
00:42:55.860 --> 00:42:59.880
Everybody knew exactly what they were trying to deliver and said, Hey, take a look at this real quick.
00:43:00.090 --> 00:43:00.981
You think we should fix this?
00:43:00.981 --> 00:43:01.260
Yeah.
00:43:01.260 --> 00:43:01.530
Okay.
00:43:01.530 --> 00:43:02.331
What if we moved it over there?
00:43:02.351 --> 00:43:02.800
Okay, great.
00:43:02.800 --> 00:43:03.570
Yeah, let me do that right now.
00:43:03.610 --> 00:43:05.601
Oh, now it's in production, right?
00:43:05.981 --> 00:43:13.597
And so, I think, like, the best teams capture that in a way that's hard to represent.
00:43:14.268 --> 00:43:17.697
Now, that's really tricky because I talk to my, I mean, I have this conversation all the time.
00:43:17.807 --> 00:43:23.527
People are like, How would we know from a metric perspective if we were truly like a high performing team?
00:43:23.867 --> 00:43:25.108
And I'm like, I don't know.
00:43:25.677 --> 00:43:26.688
You'll feel it.
00:43:27.166 --> 00:43:32.407
Tell me if I'm wrong, but like At some point, all the process is creating overhead, right?
00:43:32.407 --> 00:43:37.467
Like process, like we said this before, like processes to me create a floor, right?
00:43:37.487 --> 00:43:41.452
It's like, how do we raise kind of our lowest performance up to a point?
00:43:42.211 --> 00:43:47.711
But being phenomenal, I just don't think you can process your way to phenomenal.
00:43:48.324 --> 00:43:51.284
I'll give you a few, uh, examples on the other side.
00:43:51.284 --> 00:43:51.943
So I agree with you.
00:43:51.943 --> 00:43:54.534
Let's say that, you know, you're a really elite team.
00:43:54.543 --> 00:44:04.114
You've worked together a long time, maybe you're in the office together and you can easily turn around and say, Hey Rob, I'm going to just letting you know I'm going to do this.
00:44:04.503 --> 00:44:05.824
thing, you know, you have the context.
00:44:05.824 --> 00:44:14.364
Now, if I think about some large enterprises, we're in an era of distribution, we're working all around the world.
00:44:14.364 --> 00:44:20.373
Some of my teammates might actually not be sitting across, the way from me, we may be working from home.
00:44:20.373 --> 00:44:22.684
We may be a new team that just came together.
00:44:23.208 --> 00:44:31.318
And if we're working in this mode where, you know, nothing is documented, tickets don't have an associated branch.
00:44:31.588 --> 00:44:41.559
I think the flip side can also happen where you're actually producing maybe terrible code that would, not, help the business in the long run, cause bugs in production.
00:44:41.893 --> 00:44:44.414
So I think there's like a really fine, line here.
00:44:44.934 --> 00:44:53.074
the other thing that I would say is when you told me about the innovation, like with the hackathon, like why do we have to do a hackathon in order to do, innovation?
00:44:53.083 --> 00:44:58.443
Some of the companies that, I'm working with say, Hey, we want to do, more innovation.
00:44:58.574 --> 00:45:00.974
I'll say, okay, well, how much are you doing today?
00:45:01.463 --> 00:45:03.313
I'll say, I have no idea.
00:45:03.353 --> 00:45:04.454
I just need to do more.
00:45:05.003 --> 00:45:10.233
Well, a different situation can be, Hey, I actually do have a pretty good hygiene.
00:45:10.588 --> 00:45:17.898
I can tell you how much innovation effort we're putting into, you know, versus keeping the lights on versus enhancing features.
00:45:17.898 --> 00:45:22.148
And maybe we can go back to the business and say, look, we're pretty organized here.
00:45:22.239 --> 00:45:24.168
We don't get any time to do any innovation.
00:45:24.168 --> 00:45:25.829
We're doing it 1 percent of the time.
00:45:26.079 --> 00:45:28.099
Could we bake that into our normal workflow?
00:45:28.838 --> 00:45:34.259
So I think there's kind of like, it goes back to the beginning of the conversation that context matters a lot.
00:45:34.528 --> 00:45:38.179
So again, I'll reiterate, I think this is ultimately a scale problem, right?
00:45:38.188 --> 00:45:42.938
Like my team or my specific team runs from Sydney to Frankfurt.
00:45:43.516 --> 00:45:46.887
So me being in the West Coast is actually nice because I can actually take meetings.
00:45:47.297 --> 00:45:51.777
Without being up in the middle of the night like I had to do at my last job where I had a team in Bangalore.
00:45:52.197 --> 00:45:54.967
Not that there's anything wrong with Bangalore other than it's twelve and a half hours away.
00:45:55.248 --> 00:45:56.478
There's just no good meeting time.
00:45:56.969 --> 00:46:01.809
So what I always tell my team is you need as much process as helps you.
00:46:02.809 --> 00:46:08.038
If it's getting in your way, get rid of it, And you're not allowed to complain about it without saying what you're going to do to fix it.
00:46:08.757 --> 00:46:08.978
Right.
00:46:08.978 --> 00:46:10.248
And so I think that goes back.
00:46:10.527 --> 00:46:13.027
So it's a scale issue and it's also a leadership issue.
00:46:13.570 --> 00:46:20.690
I think if you let your engineering team, you know, bitch and moan about, pardon it, my, my French about, you know, Oh, this is getting in our way.
00:46:20.690 --> 00:46:21.800
We can't get any work done.
00:46:21.800 --> 00:46:23.650
I'm like, that's not my problem.
00:46:23.681 --> 00:46:24.751
Get your work done.
00:46:24.800 --> 00:46:25.050
Right.
00:46:25.050 --> 00:46:26.871
If this is getting your way, solve it.
00:46:27.320 --> 00:46:30.501
Or ignore it and then justify it because now you've made your delivery.
00:46:30.740 --> 00:46:39.360
And so, don't have a culture of learned helplessness around this, I think is the trap that often as companies scale, you know, they fall into that and that's just dark times.
00:46:39.360 --> 00:46:41.411
And then you have a lot of work to do to unravel that.
00:46:42.148 --> 00:46:43.528
Or maybe even automate it.
00:46:44.048 --> 00:46:45.318
So, okay.
00:46:45.318 --> 00:46:47.539
A few, a few supporting insights here.
00:46:48.219 --> 00:46:50.099
If we're moving over to the next slide.
00:46:50.373 --> 00:46:55.574
Hopefully up top when the percentage of branches not linked to issues is higher.
00:46:55.813 --> 00:47:02.393
That's the shorter the coding time when the percentage of branches not linked to issues is higher, the shorter the review time.
00:47:02.393 --> 00:47:05.693
I think that's a mutual context type situation.
00:47:06.253 --> 00:47:10.634
And when the percentage of branches not linked to issues is higher, the shorter the merge time.
00:47:11.153 --> 00:47:12.543
So this is our second insight.
00:47:12.869 --> 00:47:15.019
Move us forward to the third insight.
00:47:15.161 --> 00:47:16.161
Dora metrics.
00:47:16.382 --> 00:47:16.851
Do we have a key?
00:47:17.393 --> 00:47:27.063
Take away here, organizations with a longer cycle time have a higher rate of failures in production.
00:47:28.182 --> 00:47:37.503
Okay, so organizations with a longer cycle time, the longer my cycle time, the more failures I have in production, you know what this one made me think of?
00:47:38.652 --> 00:47:39.253
CICD.
00:47:40.322 --> 00:47:41.463
Made me think of you, Rob.
00:47:41.902 --> 00:47:52.762
But before I thought of you, I thought of my four year old daughter who's learning to ride a bike, because I'm observing this actually every day.
00:47:53.443 --> 00:47:58.853
As she's going slower, there's more failures in production, like falling over.
00:47:59.362 --> 00:48:02.293
This is the metaphor that I've heard lots of people use.
00:48:03.092 --> 00:48:08.592
But as she's starting to learn, be brave and pedal faster, she's steadying out.
00:48:08.893 --> 00:48:11.003
So that's the first thing that I thought about my daughter.
00:48:11.003 --> 00:48:12.682
And then I thought about you, Rob,
00:48:12.713 --> 00:48:12.932
yeah.
00:48:13.362 --> 00:48:20.492
So I'm going to pass it over to you because I think you, you know, your experience and, where you're working now, what do you think about that?
00:48:20.503 --> 00:48:20.822
Yeah.
00:48:20.822 --> 00:48:24.913
I mean, this is, this is my life, but I think it's everyone else's, right?
00:48:24.913 --> 00:48:26.893
Like the DO in DORA is for DevOps.
00:48:26.893 --> 00:48:34.583
Is it not like at some point we realize that if something is hard, you should do it more often.
00:48:35.092 --> 00:48:47.668
I will say before I got to CircleCI, uh, I started a company with the gentleman who's now the CEO of CircleCI and, um, He was actually like, he's a product person, by background, he was one who said.
00:48:47.688 --> 00:48:52.079
I think it was a 2011, you know, we should do, we should do continuous deployment.
00:48:52.509 --> 00:48:54.548
I was like, what are you talking about?
00:48:55.039 --> 00:48:58.199
And he said, Oh, what we're going to do is like, as soon as we write the code, we're going to push in production.
00:48:58.199 --> 00:49:02.208
I'm like, nah, releasing is terrifying.
00:49:02.869 --> 00:49:11.048
Releasing requires three pots of coffee and a weekend because you got to put it in production and then you got to clean up the mess, right?
00:49:11.059 --> 00:49:13.318
Like that's the life we all lived prior to that.
00:49:13.869 --> 00:49:17.876
And I remember, I actually go through this with, the kind of new hires at CircleCI.
00:49:18.085 --> 00:49:22.806
There's, if you go back and like find the original blog post, does anyone remember, IMVU?
00:49:23.045 --> 00:49:24.266
How's that pronounced, by the way?
00:49:24.746 --> 00:49:25.255
Anyone know?
00:49:25.916 --> 00:49:27.056
I don't even know I ever saw it written.
00:49:27.135 --> 00:49:28.456
Oh, Eric Ries, right?
00:49:28.456 --> 00:49:32.795
So he wrote the Lean Startup, he started that company, IMD, IMVU, I don't know, whatever.
00:49:32.885 --> 00:49:33.356
Doesn't matter.
00:49:33.856 --> 00:49:37.726
They wrote a blog post about continuous deployment.
00:49:37.746 --> 00:49:39.436
They're sort of like one of the early practitioners.
00:49:39.485 --> 00:49:41.815
Etsy, like, I don't know if anybody remembers this time.
00:49:41.826 --> 00:49:42.135
Anyway.
00:49:42.739 --> 00:49:55.818
They wrote a blog post and then he, the, the author of the blog post from ILE wrote a follow up describing the discussion on Hacker News, which is where you should get all your feedback, by the way, about this concept.
00:49:56.478 --> 00:50:00.748
And the thread was like, this company obviously hates their customers.
00:50:01.338 --> 00:50:03.869
You are not going to be in business a year from now.
00:50:03.898 --> 00:50:06.278
If you think that delivering continuously.
00:50:06.684 --> 00:50:08.634
Into production is a good idea.
00:50:10.003 --> 00:50:12.724
Fast forward is 2024 and we're, it's the gold standard.
00:50:12.733 --> 00:50:13.023
Yeah.
00:50:13.193 --> 00:50:13.523
Right.
00:50:13.673 --> 00:50:14.664
So like, thanks.
00:50:14.673 --> 00:50:15.313
Thank you.
00:50:15.333 --> 00:50:21.333
Also, if, if there are any visionaries out there who are trying things that no one's ever done before, like someone has to go do it, right.
00:50:21.333 --> 00:50:24.614
Someone has to go get the bloody nose and be like, wow, that was terrifying.
00:50:24.614 --> 00:50:26.403
But we got through it and now we're doing great stuff.
00:50:26.483 --> 00:50:29.454
This is like the most unsurprising thing, right?
00:50:29.454 --> 00:50:29.483
Like.
00:50:29.896 --> 00:50:33.226
Small units of work, fresh context, right?
00:50:33.226 --> 00:50:36.045
I mean, did I, I used to work in like telephony.
00:50:36.175 --> 00:50:39.155
Has anyone worked on like a year long release cycles?
00:50:39.615 --> 00:50:41.106
Please tell me you're not still doing that.
00:50:41.106 --> 00:50:43.295
But like someone says, Oh, this is broken.
00:50:43.295 --> 00:50:51.304
And you're like, wait, did I write, I literally have no idea what I was thinking when I wrote that 10 months ago, as opposed to like, I wrote it this morning.
00:50:51.304 --> 00:50:52.985
I put it in production this afternoon.
00:50:53.487 --> 00:50:54.967
So first of all, super low risk.
00:50:54.967 --> 00:50:55.657
I can fix it.
00:50:56.322 --> 00:51:01.302
I'm putting out releases that are like 10 lines of code change or whatever, right?
00:51:01.311 --> 00:51:08.541
Tiny PR is going straight into production instead of the last three months of work going into production.
00:51:08.552 --> 00:51:10.382
Like just the risk is so high.
00:51:10.561 --> 00:51:17.572
And so when the risk gets high, you slow down, you're like, well, we could release this thing, but we got to get a whole bunch of people together.
00:51:17.811 --> 00:51:19.302
We got to prep the war room.
00:51:19.302 --> 00:51:20.601
Cause there will be an outage.
00:51:20.601 --> 00:51:23.092
Like we don't know what outage we just know it's going to break.
00:51:23.331 --> 00:51:25.181
Like, and then everyone's so cautious.
00:51:25.711 --> 00:51:27.302
And once you feel comfortable.
00:51:27.717 --> 00:51:29.496
That you can deploy whatever you've built.
00:51:29.876 --> 00:51:32.387
Then you just go and you go and you go and you go.
00:51:32.387 --> 00:51:33.436
And it's so good.
00:51:33.916 --> 00:51:36.237
If you're not doing this right now, like, I don't know, come find me.
00:51:36.427 --> 00:51:38.606
You don't need to use my product, but my God, please do this.
00:51:39.146 --> 00:51:54.367
Well, so I have a funny story about this company will be, will remain unnamed, but we had two major engineering teams and this is when I really would have loved the Dora metrics is predated the Dora metrics and the one team was getting into continuously, let's go fast.
00:51:54.367 --> 00:51:54.887
Let's go fast.
00:51:54.896 --> 00:51:58.876
But when they started, they went super slow, as far as quality.
00:51:59.186 --> 00:52:02.786
They didn't have enough tests, they didn't have the systems, they didn't have the muscle.
00:52:03.476 --> 00:52:05.646
And the other team was inheriting their changes.
00:52:05.936 --> 00:52:15.137
And we could never convince that other team that even when the first team was actually doing successful delivery multiple times a day, anytime there was a problem, it was that team's fault.
00:52:15.666 --> 00:52:20.597
You know, and that's when I was really missing, like, we need a platform that tells us what the, you know, to prove.
00:52:20.606 --> 00:52:22.376
Like, no, it's you, it's not them.
00:52:22.597 --> 00:52:23.876
Anyway, you want to keep going.
00:52:24.056 --> 00:52:26.811
No, I know where you could find one, but Yeah, you keep going.
00:52:27.434 --> 00:52:30.184
It's just, it was a very frustrating experience.
00:52:30.943 --> 00:52:41.963
Yeah, you know, I think the other thing, to add on to what both of you are saying, especially Rob, is You know, if you have code that's sitting there for a long time, it's going to degrade.
00:52:42.764 --> 00:52:57.454
If you, are not confident to get code out in small chunks, it probably means that you don't have the right testing practices, automation practices, uh, feature flag type situation, system architecture.
00:52:57.784 --> 00:53:06.284
So I think the other context thing here is, here's an indicator, if you are able to have small releases that go out all the time.
00:53:06.679 --> 00:53:10.969
you probably have the full infrastructure and ecosystem in place to do so.
00:53:10.969 --> 00:53:15.139
And therefore, you're going to be in a much better situation in terms of quality.
00:53:15.650 --> 00:53:23.269
The one thing that I want to add, sorry, I'm going to drag this out, but this is my life, is the superpower of this is not the code quality.
00:53:24.119 --> 00:53:35.599
Like, yes, you recover from incidents, you take out the risk, but the thing that you're actually de risking when you deliver all day, every day, is not the quality of your code.
00:53:35.818 --> 00:53:36.759
It's the fact.
00:53:37.114 --> 00:53:40.724
That you are undoubtedly, I don't care what business you're in.
00:53:40.724 --> 00:53:43.054
I don't care how good your product managers are.
00:53:43.275 --> 00:53:45.204
You are building the wrong thing right now.
00:53:46.364 --> 00:53:55.664
And the longer you go on the wrong path, building the wrong thing, the more money you are wasting, not building the right thing for your customers.
00:53:55.965 --> 00:53:59.434
And if you are shipping three, four times a day.
00:53:59.900 --> 00:54:03.280
And getting customer feedback three, four times a day.
00:54:03.750 --> 00:54:04.780
We change this thing.
00:54:04.789 --> 00:54:05.769
We change this thing.
00:54:05.940 --> 00:54:08.150
No one gives a crap about this thing we just built.
00:54:08.170 --> 00:54:09.260
Let's take it back out.
00:54:09.409 --> 00:54:10.869
People familiar with painted doors?
00:54:11.018 --> 00:54:15.170
Like, we put up the button that doesn't even do anything and no one clicked on it.
00:54:15.210 --> 00:54:16.458
You know what we should not do?
00:54:16.789 --> 00:54:18.608
Build the feature behind that button.
00:54:19.219 --> 00:54:22.449
But so many teams spend six months building the feature.
00:54:23.110 --> 00:54:24.119
To put it out.
00:54:24.280 --> 00:54:28.869
And the last thing they do is put up the button to find out that nobody clicks on the button.
00:54:29.130 --> 00:54:34.199
That's six months of engineering effort that you could have put into something that your customers care about.
00:54:34.719 --> 00:54:37.918
And so, yes, you should absolutely reduce the risk of your changes.
00:54:37.918 --> 00:54:45.659
But what you're really doing is getting yourself turning towards the path of what your customers care about as quickly as possible.
00:54:46.050 --> 00:54:48.710
And that's where this really, like, this is just.
00:54:49.144 --> 00:54:58.855
Your cycle time will be lower, but it's not just that your cycle time is lower, it's that everything you deliver is high value to your customers because you're building the right thing, and that is when your business is going to take off.
00:54:59.164 --> 00:55:03.514
Which is the one thing about Agile I really like, right, which is early to customers and iterate.
00:55:03.945 --> 00:55:04.155
Right.
00:55:04.155 --> 00:55:07.914
Which was the whole DevOps thing was invented to, how do you do that?
00:55:08.385 --> 00:55:08.684
Right.
00:55:08.735 --> 00:55:11.675
And so it's like, Oh, punchline that there it is.
00:55:11.844 --> 00:55:13.315
DevOps itself is kind of the button.
00:55:13.804 --> 00:55:15.313
And then, then we broke other things.
00:55:15.313 --> 00:55:17.574
We said, Oh, we have DevOps now, so we don't need QA teams.
00:55:18.184 --> 00:55:19.954
There's, there I threw that hot take down.
00:55:20.844 --> 00:55:21.224
That company.
00:55:21.675 --> 00:55:22.614
That's a whole hour.
00:55:22.675 --> 00:55:22.875
That's
00:55:22.885 --> 00:55:23.405
for dessert.
00:55:23.664 --> 00:55:23.844
Yeah.
00:55:23.844 --> 00:55:24.144
Okay.
00:55:24.144 --> 00:55:24.594
Perfect.
00:55:24.594 --> 00:55:26.885
Let me, let, let me push us forward here.
00:55:27.405 --> 00:55:30.315
A few supporting, insights on the next slide.
00:55:30.385 --> 00:55:35.405
Just what we've already been saying, the longer the cycle time, the higher the change failure rate.
00:55:35.414 --> 00:55:36.394
There's correlation there.
00:55:36.394 --> 00:55:40.695
The longer the deploy time, the higher the change failure, failure rate correlation there.
00:55:40.695 --> 00:55:41.304
I think it makes.
00:55:41.945 --> 00:55:43.896
Insight number four.
00:55:44.085 --> 00:55:47.806
Well, we talked about this, a little bit already in the beginning, but.
00:55:48.621 --> 00:55:55.440
A higher pull request maturity ratio correlates a higher velocity.
00:55:56.101 --> 00:56:00.130
So a higher maturity ratio correlates with higher velocity.
00:56:00.510 --> 00:56:02.231
Talked about it a little bit in the beginning.
00:56:02.231 --> 00:56:03.300
Do we have more to add here?
00:56:03.708 --> 00:56:07.898
I mean, so we say that a, you know, highly mature PRs go quickly.
00:56:08.407 --> 00:56:08.918
Yes.
00:56:08.978 --> 00:56:09.288
Right?
00:56:09.898 --> 00:56:13.427
I struggle with this one to, to be honest, because we'll go back to the hackathon example.
00:56:13.427 --> 00:56:18.038
Yes, if we have no process, no Jira tickets, no product planning, no PRQs, whatever, you can move fast.
00:56:18.077 --> 00:56:22.157
You can move really fast, but you're producing unusable garbage from a business perspective.
00:56:22.958 --> 00:56:35.927
that's sort of a harsh take, but you know, that's an extreme example, but yes, I think that if all of those other things have lined up, right, that the culture is good, the planning is good, the modularity of your system architecture is sufficient.
00:56:36.407 --> 00:56:39.467
Et cetera, et cetera, et cetera, then absolutely right.
00:56:39.467 --> 00:56:45.818
But I think the quality bar has to be correlated to the previous thing, which is what's your failure rate, right?
00:56:46.121 --> 00:56:47.260
and then your resolution rate.
00:56:47.581 --> 00:56:51.300
So I feel like I'm in the right room to make a request for next year's report.
00:56:51.757 --> 00:57:02.331
I'd like, I'd like to see this over time because velocity over time is actually, I mean, lowercase L velocity, not like story points.
00:57:02.331 --> 00:57:04.342
I couldn't even remember what they were called for a second, their story points, whatever.
00:57:04.342 --> 00:57:14.192
But like, how fast am I delivering business value and how are my approaches to software engineering impacting that over time?
00:57:14.192 --> 00:57:22.981
Because I think one of the most common things that happens, you know, we take your hackathon example is we're fast at the beginning, and then we just.
00:57:23.882 --> 00:57:38.012
Like, it degrades, right, because we didn't have the maturity, the process, whatever that is, and it would be really interesting to see some of these metrics, how they evolved over the course of like 12, 24 months, because you have all the data.
00:57:38.012 --> 00:57:38.882
So that's awesome.
00:57:39.032 --> 00:57:41.831
because I think that would be really telling, right?
00:57:41.831 --> 00:57:47.081
Like what I care about as an engineering leader is not how fast did you move this week or this month or this quarter.
00:57:47.476 --> 00:57:54.246
But can we sustain that for an extended period and consistently predictably deliver value?
00:57:54.806 --> 00:57:57.666
and all the times we're like, we're really fast right now.
00:57:58.016 --> 00:58:03.646
We know we're like, gonna hate ourselves in about six months for whatever it was that we did.
00:58:04.097 --> 00:58:11.047
Yeah, I mean, I think, you know, what's the definition of, it's quality, time, feature set, pick any two, right?
00:58:11.496 --> 00:58:15.416
Because what do we know for a project that has a certain longevity?
00:58:15.702 --> 00:58:18.311
Yeah, you get to the point where it's like, okay, it's basically there.
00:58:18.311 --> 00:58:22.318
And then what you want to actually see is the quality correlates to the churn rate, right?
00:58:22.318 --> 00:58:30.318
So you know that the successful, mostly complete project, the pull request maturity might be good, but the velocity should just go through the floor because you're not touching it anymore.
00:58:30.538 --> 00:58:34.327
You're moving on to the next thing, the thing that's going to make you money a year from now or six months from now.
00:58:34.992 --> 00:58:35.222
Right.
00:58:35.222 --> 00:58:40.132
So that I think there's some really interesting ways and I plus one, I would love to see this over time.
00:58:40.672 --> 00:58:40.913
Yep.
00:58:40.913 --> 00:58:44.280
So, a new, uh, metric that was added, this year.
00:58:44.280 --> 00:58:47.239
So I think, a fifth generation will be even better here.
00:58:47.250 --> 00:58:53.166
I'm going to push us forward Here's the last, Insight that we have, and it's around predictability.
00:58:53.777 --> 00:58:57.456
So the key takeaway is around capacity, accuracy.
00:58:57.646 --> 00:59:03.916
So what it's saying here is over half of engineering projects under commit against their goal.
00:59:04.353 --> 00:59:09.474
And less than 25 percent of projects fell within the ideal range.
00:59:09.494 --> 00:59:14.094
So when we're thinking about under committing, what it really means is.
00:59:14.648 --> 00:59:31.018
Hey, I can really deliver this amount of story points, but I'm only going to say that I can deliver maybe 80 percent of them, but I'm under promising so that I can over deliver in order to hit my, maybe my predictability goal or to be on time.
00:59:31.704 --> 00:59:32.724
This one blew my mind.
00:59:32.724 --> 00:59:38.795
I have to say, because in, I've been doing this for 30 years, every engineer, I can't think of an engineer that did not.
00:59:39.219 --> 00:59:39.789
Overcommit.
00:59:40.480 --> 00:59:41.639
I cannot think of one.
00:59:41.920 --> 00:59:49.630
In fact, I have this system where depending on who you are, you get a multiplier, like no matter what you say, I'm going to multiply it by three because I just know.
00:59:50.469 --> 00:59:50.590
So
00:59:50.590 --> 00:59:52.099
this one is fascinating to me.
00:59:52.099 --> 00:59:52.934
I don't know your experience.
00:59:52.934 --> 00:59:53.360
You have
00:59:53.360 --> 00:59:55.065
a Monte Carlo simulation for all your engineers.
00:59:56.025 --> 01:00:00.344
Um, well, two, two things that I think are interesting here, a hundred percent agree with that.
01:00:00.344 --> 01:00:03.744
I'm just like, wait, there are engineers who actually undercommit.
01:00:03.844 --> 01:00:04.585
That's cool.
01:00:04.974 --> 01:00:06.275
they're very smart, by the way.
01:00:06.275 --> 01:00:06.655
I agree
01:00:06.815 --> 01:00:07.155
with that.
01:00:07.264 --> 01:00:11.195
So one, one thing that I read between the last time we talked about this and now.
01:00:12.574 --> 01:00:18.585
Well, I don't know if people are familiar with Kent Beck, I love his latest book, Tidy First, but I started following his sub stack.
01:00:19.034 --> 01:00:22.025
He published this thing called, Forest in the Desert.
01:00:22.034 --> 01:00:22.914
Did anybody read this?
01:00:23.144 --> 01:00:27.855
So he talks about different kinds of projects, and, uh, the feeling, right?
01:00:27.855 --> 01:00:30.054
Again, kind of stepping away from the metric, but what it feels like.
01:00:30.344 --> 01:00:44.675
And in his forest model, he has roots, and he presented this with someone else, and I forget who it was, and I apologize, but, And one of the roots are kind of like the core things that have to be true in order to get to a great project or a great team or however you want to think about it.
01:00:45.065 --> 01:00:51.335
And one of the things in his list of roots was that you never commit to more than 50 percent of what you can get done.
01:00:51.699 --> 01:00:56.119
And that's, that's, that's like, so that's really interesting in this metric.
01:00:56.329 --> 01:01:02.489
And his whole point is you set yourself up, you, put the time and place to do things well, right?
01:01:02.518 --> 01:01:04.139
Again, to that sustained velocity.
01:01:04.619 --> 01:01:06.579
And then he talks about the feeling, right?
01:01:06.579 --> 01:01:08.059
What does it feel like then to be there?
01:01:08.068 --> 01:01:11.009
The feeling is like someone shows up and says, we're going to do a thing.
01:01:11.009 --> 01:01:12.208
We know exactly how to do it.
01:01:12.648 --> 01:01:13.918
We know exactly where it goes.
01:01:14.398 --> 01:01:16.498
We know what it would take to do it.
01:01:16.509 --> 01:01:21.289
Like all of those things that it just feels like in so many projects, you're like, I have no idea.
01:01:21.289 --> 01:01:24.239
Let's just pick a number and pretend we could get it done in that amount of time.
01:01:24.728 --> 01:01:28.764
Um, and so I think leaving space kind of caught it to your point.
01:01:28.775 --> 01:01:30.994
Like everyone's overcommitted, right?
01:01:31.014 --> 01:01:34.155
Everyone underestimates what it really takes to get something done.
01:01:35.119 --> 01:02:22.320
And then they get pushed up against the wall, and they're like, well, we said we'd get this stuff done, and maybe if we just cut some corners, then we will get it done, so now they got it done, and the next time, their estimate is even worse, because they just made the system harder to work on, and there's this kind of negative feedback loop, I just think we, as engineers, tend to put ourselves, and as a leader, I'll say it's probably my fault, I don't know why yet, but it probably is, but like, put ourselves in that situation where we believe something is achievable, we feel we're smart, we know how to get stuff done, and then we create that negative scenario where we're like, well, I said I could get it done, so now I am gonna get it done, by like, not sleeping, and cutting corners, and all these kinds of things, and that would make it worse for the next round, and the next round, and the next round, whereas if you leave yourself the space, I think that you can So, yeah.
01:02:22.320 --> 01:02:22.329
Yeah.
01:02:22.989 --> 01:02:29.244
and then the other thing that I was going to say about this, I can't remember, are people familiar with Marty Kagan, Silicon Valley product group?
01:02:29.373 --> 01:02:32.123
Anyway, it's got like a very large number of books.
01:02:32.123 --> 01:02:40.853
I don't know how he also does his job and writes all these books, but, he reads his own audio books to props to Marty Kagan.
01:02:40.853 --> 01:02:46.014
Anyway, he talks about predictability versus innovation.
01:02:47.023 --> 01:02:49.523
just since we talked about innovation earlier and like that.
01:02:50.034 --> 01:03:03.273
That many R& D leaders are focused on predictability as the most important thing, but, I can't remember who he's quoting when he says like 100 percent predictability is 0 percent innovation, right?
01:03:03.273 --> 01:03:10.304
Like when we are perfect at understanding how long everything is going to take, that means we have not taking any risks.
01:03:10.733 --> 01:03:13.153
We're not trying anything novel or anything new.
01:03:13.503 --> 01:03:17.443
And so, you know, I think we need to, predictability tells you a lot.
01:03:17.869 --> 01:03:21.789
About whether you understand your system, whether you're capable of doing the thing that you need to get done.
01:03:22.358 --> 01:03:27.759
But if all you're doing is just, we ticketed out the work and we delivered the work exactly as was planned.
01:03:28.353 --> 01:03:32.494
Nobody's pushing the envelope, no one's taking risk, no one's innovating.
01:03:32.494 --> 01:03:34.534
And as a business, that's not going to be a good outcome.
01:03:34.543 --> 01:03:38.603
One thing that I will say, and when you look at the report, you know, there's two different metrics here.
01:03:38.603 --> 01:03:40.583
One of this is capacity accuracy.
01:03:40.583 --> 01:03:42.804
There's another one that's planning accuracy.
01:03:42.824 --> 01:03:51.983
So when you think about capacity accuracy, it's kind of like the know thyself as a team, which means like know how much work you can get done in a sprint.
01:03:52.213 --> 01:03:54.653
Now that's different than planning accuracy.
01:03:55.434 --> 01:03:57.614
Planning accuracy is more so saying.
01:03:58.014 --> 01:04:04.684
I'm coming to you, Rob and saying, Hey, I think I can get this feature delivered on this date.
01:04:05.313 --> 01:04:11.293
and whether I, or let's say within a sprint, I think I'm going to do these exact 10 stories.
01:04:11.534 --> 01:04:19.713
did I actually do those exact 10 stories or did I have bugs come in from production or actually wanted to do some innovation stuff?
01:04:20.014 --> 01:04:25.244
And I ended up doing 10 things, but it wasn't what I actually said I was going to do.
01:04:25.454 --> 01:04:28.414
So just a little bit of a distinction there.
01:04:28.824 --> 01:04:31.833
but yeah, all in the report, interesting stuff.
01:04:32.289 --> 01:04:32.760
Okay.
01:04:33.289 --> 01:04:35.670
So just the, a few takeaways here.
01:04:35.670 --> 01:04:38.474
So all of you do have, the benchmarks report.
01:04:38.994 --> 01:04:55.025
highly recommend you check it out, highly recommend that, you know, for your own, own organization, what your data points are, how you, uh, match up against the benchmark so that you can have the right conversation, kind of like how we did up here live, with your teams.
01:04:55.525 --> 01:05:02.824
And I wanted to give a big, round of applause and thank you to Tara and Rob, for being our panelists today.
01:05:02.864 --> 01:05:03.434
Great job.
01:05:10.695 --> 01:05:12.034
So, hello everyone.
01:05:12.034 --> 01:05:13.005
I'm Ben Lloyd Pearson.
01:05:13.025 --> 01:05:16.704
I'm one of the hosts of Dev Interrupted, and I'll be emceeing some of the event tonight.
01:05:16.755 --> 01:05:19.034
we've got a few minutes before dinner is ready.
01:05:19.034 --> 01:05:22.275
We have time, I think, for three questions.
01:05:22.695 --> 01:05:27.164
So if you have a question, we have a microphone that Andrew will bring around to you.
01:05:28.329 --> 01:05:46.130
love the example of the hackathon and project management and correlation over there, but a factor how many of these hackathon projects actually become features is a question because quality, a clear understanding of the intent, there's a lot more that happens, right?
01:05:46.139 --> 01:05:50.849
For the product side, speed is, is the primary factor in the hackathon.
01:05:51.579 --> 01:05:54.789
Uh, but there are other things that that make a product market fit.
01:05:54.820 --> 01:05:56.909
So any thoughts on that?
01:05:57.681 --> 01:05:58.420
I certainly do.
01:05:58.440 --> 01:06:11.590
I've seen a bunch of different companies do hackathons and I think the ones that are the most successful are the ones where, the senior leadership in engineering and also the product team is heavily involved in either seeding the themes, right.
01:06:11.621 --> 01:06:19.400
And then committing to identifying and then executing and making space for execution that creates a really positive feedback loop.
01:06:19.690 --> 01:06:20.681
Like here is a customer problem.
01:06:20.690 --> 01:06:28.150
We don't have a good solution for, or here's a set of challenges and then the engineering team sees, Oh, we actually took a couple of those and turn those into actual features.
01:06:28.360 --> 01:06:30.990
We invest in that time and it was an enormous success.
01:06:31.420 --> 01:06:39.590
I think where you don't see this success where it's, you know, very kind of low key, it's mostly a reason to hang out and, you know, code from the bean bag in the office and eat free pizza.
01:06:39.990 --> 01:06:41.090
You don't get the value add.
01:06:41.090 --> 01:06:44.541
And then at some point somebody realizes, wow, this is a really expensive prospect.
01:06:44.541 --> 01:06:45.800
We've got 500 engineers who are.
01:06:46.135 --> 01:06:48.815
Basically just fucking off for 24 hours.
01:06:49.025 --> 01:06:49.715
Let's stop.
01:06:49.746 --> 01:06:51.545
And now we don't have hackathons anymore.
01:06:52.434 --> 01:06:54.315
how many things can I plus one in there?
01:06:54.525 --> 01:06:55.235
All of them.
01:06:55.344 --> 01:06:56.514
I even keep track.
01:06:56.875 --> 01:06:57.855
10 plus ones.
01:06:57.974 --> 01:06:58.644
Is that plus 10?
01:07:00.675 --> 01:07:01.114
Yeah.
01:07:02.045 --> 01:07:06.184
I worry less about the quality.
01:07:06.925 --> 01:07:09.335
I mean, I worry about quality, but that's not the thing about hackathons.
01:07:09.344 --> 01:07:13.485
It's more the like, is it aligned with what we're actually trying to achieve?
01:07:14.085 --> 01:07:14.855
and I think.
01:07:15.409 --> 01:07:26.530
What I would encourage, and I sort of got into this on the DORA metrics thing, right, like, if you can get real feedback from your customers by delivering something in the same day, life feels like a hackathon.
01:07:27.110 --> 01:07:27.420
Right?
01:07:27.420 --> 01:07:42.512
Like, I think that a lot of that is spawned, from my life is drudgery, my project management hygiene is super high, and all I do is, like, execute the tickets that were, you know, Perfectly framed for me,
01:07:42.641 --> 01:07:43.311
kill me now
01:07:43.512 --> 01:07:55.175
and involving your engineers, even if it's just a more senior, like your tech leads or whatever that might be like involving them in the product discovery is kind of the ideal for me.
01:07:55.175 --> 01:08:04.425
Like if it feels like I need to take a week off and sit in the bean bag and eat the pizza and like make whatever I want, that's often because my job doesn't feel particularly fulfilling.
01:08:04.664 --> 01:08:10.184
So I would encourage like a deep think about what it is to get your folks engaged all the time.
01:08:10.905 --> 01:08:14.594
So it doesn't feel like a hackathon is like this special treat where I get to work on stuff.
01:08:14.594 --> 01:08:17.015
I like, uh, that sounds terrible.
01:08:17.885 --> 01:08:19.845
One variation I've seen, it's been kind of interesting.
01:08:19.854 --> 01:08:29.064
I think this is like depends on the company, but like rather than hackathons, they would do rotations of engineers into sales engineering or technical services, right?
01:08:29.064 --> 01:08:31.345
So it's like now you're getting the customer empathy.
01:08:31.359 --> 01:08:52.069
So I think there's a lot of different ways that you could approach this theme right around getting ultimately what you want, which is engineering, a sense of engineering, ownership, and accountability in the success of the customer experience.
01:08:52.979 --> 01:08:54.564
Hello, I like this talk.
01:08:54.583 --> 01:08:55.533
I was, I want to hack it.
01:08:55.663 --> 01:08:57.493
That one terror was working at Mongo.
01:08:57.884 --> 01:08:58.673
Do you, I mean, I was working on that.
01:08:58.684 --> 01:08:59.184
My God, is
01:08:59.184 --> 01:08:59.823
that Marcus?
01:09:00.123 --> 01:09:00.394
Is it?
01:09:00.684 --> 01:09:04.984
If it's, hi Tara, I just sent you an email, but it's irrelevant.
01:09:05.186 --> 01:09:09.269
there's a lot of talk right now about, there's a ton of engineers.
01:09:09.269 --> 01:09:18.358
Most engineers, even at these larger companies, where the vast majority of them just do the bare minimum.
01:09:20.217 --> 01:09:26.377
And maybe that's reflected in this mythical under committed metric that I never would have guessed.
01:09:26.931 --> 01:09:38.542
What do you do about, a situation, a team where one or two engineers are really outperforming, but everybody else is, they're fine.
01:09:38.551 --> 01:09:40.792
They're not like, no, one's terrible.
01:09:40.792 --> 01:09:41.612
So it's not a situation.
01:09:41.641 --> 01:09:46.152
It's not a situation like you're talking about earlier with there's a leadership puzzle piece.
01:09:46.567 --> 01:09:51.895
To address, but like everyone's doing okay, but then some people seem to be fantastic.
01:09:51.895 --> 01:09:55.525
Is that normal or do you have something else to investigate?
01:09:55.855 --> 01:09:56.996
Is there more to get out of people?
01:09:56.996 --> 01:10:00.805
Is there a platform issue, tooling issue or culture issue?
01:10:00.805 --> 01:10:08.275
I'm just curious what you all think about this notion of people are just barely doing any work.
01:10:09.136 --> 01:10:09.555
I've seen this.
01:10:09.555 --> 01:10:11.225
It's the 10 times employee, right?
01:10:11.256 --> 01:10:11.716
I know, do you?
01:10:11.796 --> 01:10:12.676
I went first last time.
01:10:13.185 --> 01:10:16.000
Oh man, You know, you listed some ideas there.
01:10:16.289 --> 01:10:17.560
And it's a really interesting question.
01:10:17.579 --> 01:10:19.890
Listen, my idea, like I said, platform problem is a tooling problem.
01:10:19.890 --> 01:10:20.680
Is it a culture problem?
01:10:20.680 --> 01:10:22.649
Like the answer is yes to one of those.
01:10:23.229 --> 01:10:28.130
And this is where like the whole theme of metrics are the start of a conversation.
01:10:28.140 --> 01:10:28.460
Right?
01:10:28.479 --> 01:10:39.239
Like, okay, I can see that a couple of people on my team are, you know, whatever your measure might be, they're just delivering a lot more, they're writing better code, they're super engaged while other people are a little checked out.
01:10:39.762 --> 01:10:42.252
That's your manager's job, right?
01:10:42.292 --> 01:10:43.443
Is to figure out.
01:10:44.007 --> 01:10:46.448
Do these people have, do they have the skills?
01:10:46.768 --> 01:10:48.578
Are they engaged in the problem we're solving?
01:10:48.917 --> 01:10:53.127
Are they, you know, 5 percent of the time, I have no percentage.
01:10:53.497 --> 01:10:57.018
Some percentage of the time, that person just has some shit going on in their life.
01:10:57.768 --> 01:11:02.257
And they were actually great three months ago, and they're gonna be great in three months, and they're just going through some stuff.
01:11:03.108 --> 01:11:10.467
And sometimes they are absolutely terrified to show up at work every day because they feel like they have no idea what they're doing, and someone's gonna find out.
01:11:11.127 --> 01:11:14.837
I cannot tell you which of those scenarios it is from looking at a dashboard, right?
01:11:15.207 --> 01:11:25.868
And so it's absolutely telling, if some people are able to execute really effectively in that environment, then you have an interesting baseline.
01:11:26.658 --> 01:11:29.268
Some people are executing here and some people are executing here.
01:11:29.268 --> 01:11:32.478
So it's not the environment holistically, right?
01:11:32.507 --> 01:11:36.938
We talk about this with like twins raised in the same environment, same genetics.
01:11:36.938 --> 01:11:38.358
Okay, but they end up different.
01:11:38.358 --> 01:11:39.068
Okay, what's it?
01:11:39.068 --> 01:11:39.917
What do we learned?
01:11:40.557 --> 01:11:44.287
I can't tell you what the answer is, but I can tell you there's something interesting there.
01:11:44.528 --> 01:11:46.938
And I would want a manager to be going and figuring that out.
01:11:46.948 --> 01:11:53.108
Like if you, you know, there was kind of open with the like, what do you think about people kind of drifting by and not, you know, just doing the bare minimum?
01:11:53.677 --> 01:11:54.627
It's not the ideal.
01:11:54.997 --> 01:11:56.828
It's not what I want in my organization.
01:11:56.908 --> 01:12:00.627
I understand it happens, but the first step is just understanding why, right?
01:12:01.177 --> 01:12:03.898
And some people might just be burned out on your organization.
01:12:04.047 --> 01:12:05.028
maybe they need to change.
01:12:05.028 --> 01:12:06.408
That's totally fine, right?
01:12:06.408 --> 01:12:19.211
But if you're not even having that conversation, What will happen in that scenario, again, I'll call it 80 percent of the time, I'm making up random percentages, is those two people that are crushing it, probably gonna leave.
01:12:19.930 --> 01:12:26.640
Cause they're like, wait a second, I'm carrying the team, and everyone else is making the money I'm making, but I'm doing all the work, right?
01:12:26.650 --> 01:12:38.860
So it's important to address, there's plenty of human factors and it's just, it's complex for sure, but if you don't address it, what you're gonna get is just a team of people who are just phoning it in.
01:12:39.461 --> 01:12:40.371
That much I guarantee.
01:12:41.020 --> 01:12:59.631
Yeah, just added, I think it totally goes back to the team leader and to the manager, each one of the developers on the team is going to have different aspirations, what motivates them, what a good experience looks like, you can probably ask three to four questions and find out, Hey, do you like the product that you're working on?
01:13:00.100 --> 01:13:00.770
Yes or no?
01:13:00.810 --> 01:13:03.461
Are you inspired by the technology that you're using?
01:13:03.905 --> 01:13:07.296
Do you feel that you have the right career path here?
01:13:07.735 --> 01:13:10.475
And are you motivated, if I'm the team leader, by me?
01:13:10.475 --> 01:13:11.855
Am I giving you inspiration?
01:13:11.855 --> 01:13:14.126
You ask a few questions, you'll probably find out.
01:13:14.386 --> 01:13:18.905
The two that are performing really well probably have good alignment on most of those.
01:13:19.565 --> 01:13:22.595
The ones that aren't performing as well, there's going to be a major gap.
01:13:23.456 --> 01:13:34.525
So yes, to all that, I think there's another thing, and I've had some success with this, which is, you know, how, how intentional are you as a leader about your culture kind of in the broad sense?
01:13:34.525 --> 01:13:34.685
Right?
01:13:34.685 --> 01:13:35.105
Well, one.
01:13:35.485 --> 01:13:40.036
First of all, humans aren't fungible and don't treat them as if they are, because that way lies madness and chaos.
01:13:40.466 --> 01:13:43.619
but yeah, bench management, to Rob's point, super critical.
01:13:43.649 --> 01:13:45.880
Don't take your top performers for granted.
01:13:46.189 --> 01:13:49.460
They think about, how do you incentivize the type of culture you want?
01:13:49.470 --> 01:13:54.789
You know, we say as leaders that we want employees that, have a learning attitude, a learning mindset, right?
01:13:54.789 --> 01:13:57.170
Well, how do you know that and how do you incentivize it?
01:13:57.439 --> 01:14:05.470
So there was a company I was working at that, was kind of migrating out of a more stodgy, infrequent deployment, and wanted to get into the cloud.
01:14:05.840 --> 01:14:07.739
Uh, wanted to, you know, go faster.
01:14:07.750 --> 01:14:09.579
And I'm like, alright, well, what's the test story?
01:14:09.609 --> 01:14:13.250
Well, we don't have very many tests, automated tests, but we have a good QA team.
01:14:13.250 --> 01:14:16.350
Like, okay, well, they need to focus on the stuff that's hard to automate.
01:14:16.750 --> 01:14:17.890
We need the developers to write tests.
01:14:17.890 --> 01:14:19.520
Well, developers don't write tests, they're not QA.
01:14:19.649 --> 01:14:21.789
Now, this is the opposite side of the problem that we have now.
01:14:22.314 --> 01:14:28.335
Which developers can't write negative tests, but in any case so I we this is we're on the office We have screens everywhere.
01:14:28.505 --> 01:14:41.680
I wrote some code that would call stuff out of we were using bamboo, which is a terrible CI system Don't use it You know, we're using Jira like okay top Top build breaker, you would get like a negative Chivo, like top, you know, bug generator.
01:14:41.680 --> 01:14:43.470
You get a negative Chivo, top bug fixer.
01:14:43.470 --> 01:14:49.060
Oh, you get happiness and light and it was a totally Mario Kart sort of experience all over the engineer organization.
01:14:49.060 --> 01:14:49.810
What do you know?
01:14:51.380 --> 01:14:55.770
The number of tests went up, the number of failures went down because we made a game out of it, right?
01:14:55.770 --> 01:14:57.949
So you know that's not gonna work for every organization.
01:14:57.949 --> 01:14:59.649
It's not gonna work for every product team, right?
01:14:59.670 --> 01:15:04.279
But you know again goes back to these metrics are critical tools.
01:15:05.619 --> 01:15:08.010
Not outcomes in and of themselves, right?
01:15:08.029 --> 01:15:16.880
that is the biggest thing what you get the data Make sure you it's accurate as possible and now do something with it Don't just have a dashboard that you kind of look at every once in a while and think huh, that's interesting
01:15:17.470 --> 01:15:17.760
All right.
01:15:17.930 --> 01:15:18.270
One more.
01:15:18.649 --> 01:15:19.369
Just one more.
01:15:19.552 --> 01:15:29.930
So during the discussion about CI CD and Tara you dropped a little grenade about companies throwing out their QA processes, in, in sort of the search for CD nirvana.
01:15:29.951 --> 01:15:43.530
And you touched on just briefly, but I want to know what, from your perspective as a leader who's seen kind of the adoption of CICD, what's to be the best approach for folding in an existing QA process into our migration to CICD?
01:15:44.671 --> 01:15:50.610
So, I mean, to me, the, the value of qa, like a QA engineer, their value is that they understand how to break things.
01:15:51.836 --> 01:15:55.036
they're multipliers on the robustness and reliability of your system.
01:15:55.466 --> 01:16:09.706
Developers know how to write statistically speaking, I'm not saying absolutes, but developers statistically speaking are really good at writing tests that prove that what they wrote worked, not how to prove that it doesn't break under different circumstances.
01:16:09.706 --> 01:16:17.653
And so the domain of QA, I think that we as an industry have too often blithely kind of tossed has lost that art.
01:16:17.662 --> 01:16:17.932
Now.
01:16:19.108 --> 01:16:39.108
You could argue, like, used to be, like, you could, you know, you could automate, like, back end stuff that's all APIs, but you couldn't automate front end now, and then SonarQube and, and other companies came out, well, now you can automate your front end, but that still doesn't solve, I think, having the domain expertise to guide the negative aspects, right, the stress testing, what have you, and what we've turned it into is incidence management.
01:16:39.118 --> 01:16:44.417
It's like, oh, well, it got out to production, and now we can, if we can fix it fast, we're okay, and maybe that's the right answer, right?
01:16:44.417 --> 01:16:46.188
I'm not, like, an oracle of truth here.
01:16:46.568 --> 01:16:52.627
But I do think that as an industry, we've lost something there and it, and I think it'll be interesting and I suspect it will actually come back around.
01:16:52.627 --> 01:16:55.318
Everything in the internet is cyclical in my experience.
01:16:55.318 --> 01:16:57.877
So I think people are identifying, they still have QA teams.
01:16:57.877 --> 01:16:59.078
They probably are still getting value.
01:16:59.448 --> 01:17:04.507
If they're struggling in a lot for quality issues, they're probably thinking, huh, maybe we need some QA experts.
01:17:04.648 --> 01:17:11.648
You know, hopefully they don't still exist, um, to come in and help guide how we do production operations, how we do CI/CD in interesting ways.
01:17:11.887 --> 01:17:14.568
I think it's an interesting thought exercise, honestly.
01:17:15.773 --> 01:17:35.979
Yeah, I mean, I definitely agree with developers testing the assumptions that they've made, proving that the assumptions they made work as opposed to questioning the assumptions they've made right at QA, the best sort of QA folks that I've known in my life were that they're like, what happens if I type this crazy string in here?
01:17:35.979 --> 01:17:37.840
And you're like, well, you just crashed the entire site.
01:17:37.869 --> 01:17:38.680
That's pretty impressive.
01:17:39.039 --> 01:17:39.859
How'd you think of that?
01:17:39.899 --> 01:17:40.760
I don't know.
01:17:40.760 --> 01:17:41.930
It just made sense to me, right?
01:17:41.930 --> 01:17:42.130
Like, yeah.
01:17:42.430 --> 01:17:44.350
And I think that that's missing.
01:17:44.859 --> 01:17:50.720
I will say at CircleCI, I would, we do not have anyone with the title quality QA, anything like that.
01:17:50.720 --> 01:17:52.310
I mean, that's been true since I got there.
01:17:52.659 --> 01:18:00.020
and we have built a large amount of tooling for ourselves and unsurprisingly for our customers to allow you to, you know, to mitigate risk.
01:18:00.529 --> 01:18:06.579
In production deployment to make it so that, you know, if something does go wrong, it impacts a very small number of your customers.
01:18:06.579 --> 01:18:07.979
It's easy to remediate, et cetera.
01:18:08.590 --> 01:18:17.289
so I think there's many strategies, but I do think that's an interesting piece that we've, you know, I, does anyone have a replacement for the baby bathwater metaphor?
01:18:17.329 --> 01:18:18.920
Because it kind of freaks me out, but like.
01:18:19.734 --> 01:18:21.225
That's the one you would use, right?
01:18:21.225 --> 01:18:23.395
Like we got, we're like, Oh, we have automated testing.
01:18:23.484 --> 01:18:31.465
But we threw out with that the ability to reason about the weird edge cases that are going to break a system because you didn't think about them as you were implementing.
01:18:32.028 --> 01:18:35.108
but I think the question was really about making the transition.
01:18:35.207 --> 01:18:36.747
Take it in small increments.
01:18:36.757 --> 01:18:39.167
Find a way to test a certain part of your platform.
01:18:39.347 --> 01:18:43.747
Take the things that are fairly stable and put in regression tests, you know, those sorts of things.
01:18:44.097 --> 01:18:55.787
I think a lot of people, when they try to make big transitions to anything, any technology or whatever, try to do something all in, and, like, spoiler, I don't even know what you're thinking about right now, but that's gonna fail.
01:18:56.387 --> 01:18:59.698
Right, doing anything all at once is just guaranteed to fail.
01:18:59.698 --> 01:19:06.877
It's gonna be, like, way bigger than you ever imagined, you're never gonna get there, but if you can find a way to do it in small increments You can make any transition over time.
01:19:07.047 --> 01:19:07.217
Great.
01:19:07.228 --> 01:19:21.167
And I think, you know, just to close it out, it's, I think one of the main reasons that, as a DevOps aficionado, I kind of mourn the fact that DevOps contributed to this concept because you can't have manual QA gate keeping as part of a continuous delivery or continuous deployment mechanism.
01:19:21.167 --> 01:19:21.427
Right.
01:19:21.828 --> 01:19:26.778
But I, I assert that there are probably different ways that we can approach this asynchronous as a thing, right.
01:19:26.797 --> 01:19:28.908
That, you know, where do you insert them?
01:19:28.908 --> 01:19:29.898
Is it in the design phase?
01:19:29.908 --> 01:19:34.108
Like there's a lot of things there, but I think what you have to think about as a business is where are you struggling?
01:19:34.632 --> 01:19:41.036
And, you know, can, can a different type of, of domain expertise help and then figure out what that means.
01:19:42.033 --> 01:19:42.443
Awesome.
01:19:42.483 --> 01:19:44.393
Well, let's hear it one more time for our guests.