WEBVTT
00:00:00.209 --> 00:00:06.719
Welcome to the Scrum.org community podcast, the podcast from the home of Scrum.
00:00:03.029 --> 00:00:21.679
In this podcast, agile experts, including professional Scrum trainers and other industry thought leaders, share their stories and experiences. We also explore hot topics in our space with thought-provoking, challenging, energetic discussions. We hope you enjoy this episode.
00:00:25.219 --> 00:00:28.699
Hello, and welcome to the Scrum.org Community Podcast.
00:00:28.760 --> 00:00:45.250
I'm your host Dave West, CEO here at Scrum.org. In today's podcast, we're very lucky to be talking with Matthew Hodgson of the Australian consulting company Zen X Machina, welcome to the podcast, Matt.
00:00:47.049 --> 00:00:47.590
Thanks, Dave.
00:00:49.119 --> 00:00:54.130
It's great, great to have you here. All the way from sunny Australia, though it is the winter at the moment, right?
00:00:54.429 --> 00:01:07.200
It's winter and it's miserable outside. Cold because we're in Canberra, between Sydney, Melbourne, and it's cold and windy, but I'm hoping it'll get a bit more sunny a bit later. Might go for a nice walk outside.
00:01:08.099 --> 00:02:10.469
Yeah, it is. It's great to have you here and all the way from Australia. So, for the listeners, Matt and I have been talking for a while about a really interesting research project he's been doing in the area of agentic product development. Of course, agentic delivery is very interesting at the moment, as the world wrestles with the idea that agents and AI can build complex digital products, replacing humans and ultimately creating an age of abundance. Of course, this might be describing a vision with reality lagging far behind. So when Matt came to me and introduced me to the work that he's doing, I was like,"This is awesome! This is this is great to be hearing real agentic product development happening. And as you'll hear from the podcast, the ideas behind research and the like that support that. So, so Matt, our listeners would love to hear a little bit about the Raptors project and your objectives when you kicked it off.
00:02:12.000 --> 00:02:15.060
Thanks, Dave.
00:02:12.000 --> 00:03:03.000
Yeah, the Raptors, as all really great experiments, was started as an accident. We're doing some digital transformation work. We use in the company AI quite a lot to be able to synthesize huge amounts of our research data for consulting past papers, etc. And the we just got to a point where some of the work that I was doing just wasn't scaling, basically. And I thought, well, how can I take what was largely still manually saving prompts and a range of other things, and go between context windows. How can I accelerate that that work to the point where I'm not so much in the loop, specifically like at every single stage to approve things and move things around?
00:02:59.289 --> 00:03:30.080
And so that's when I started to experiment with with agents, and didn't get too far in before I thought, well, at the point of scale, most organizations when we're going through some kind of transformation, and AI is top of that list. Scrum provides a really great framework to be able to handle complex things from a from a teamwork perspective. So I thought, well, can I do the same with agents?
00:03:30.289 --> 00:03:52.960
Can can I actually have agents that are representative of of what would normally be a cross functional team? And what will happen when we take complex work, everything from doing just basic knowledge work, writing papers, synthesizing research, all the way through to software development, and so that that was the Raptors. That that's where that where they started.
00:03:53.889 --> 00:04:31.670
Will will scrum survive the litmus test of turning becoming what is. I decided that I would be the product owner of that team, decide what they do and where we go and and what we need to focus on. What does that look like with kind of in a hybrid model of agents still doing the work directed by a person? Where does the person sit? Where is the most effective place for for for human decision making to occur? And how long is it going to take to teach agents to work with Scrum compared to, say, human teams, for example? And so that that was that was where we started.
00:04:32.449 --> 00:04:35.209
And and that was how many months ago did you kick this off?
00:04:37.069 --> 00:05:22.939
the The initial work in exploration started around April May last year, to accelerate some consulting work, and then evolved into a formal hypothesis around June last year, where where we started to explore the what does just the basics of what does Scrum structure look like when applied to Ingenic Team? What does a multiple agent system. Actually, look like, and do these boundaries that when we create a functional team, even just just creating a normal human team, do the can we use exactly the same thing? So I don't use, for example, we started off by by just using normal human software and started off with Gemini and Copilot, and then moved moved into Claude Code. So yeah, that started about June, back you know 12 months ago.
00:05:23.990 --> 00:05:37.040
Wow! And so okay, I've I've got to ask some very simple questions for our listeners. I I know some of this, so apologies if I've asked you this before, but I think this is important. So you have obviously you have developers.
00:05:37.490 --> 00:05:40.730
You're acting as the product owner. Do you have a Scrum Master?
00:05:41.959 --> 00:06:44.860
Yes, I do have a scrum master. So I have an agent that's a scrum master. Al, Al's job is to do is basically to do orchestration, and even that by itself, because there's been times when I go in, we go into an organization and we go, okay, well, what do your teams look like? What's the product that we're working on? Select Scrum masters, people who are good at supporting at the starting point developers to be self-managing. So I went, well, I want a I want a point where someone can do that orchestration, but moreover assert and help to help the other agents to work within the boundaries of Scrum, and provide a point of escalation where, if their self-management kind of runs out of room and they've done what they can do, that I've got a single point of escalation back to me, and that that in itself is like even name giving them human names like Al. Al is my Scrum master, gives me as a human a much more recognizable point of interaction and asking
00:06:44.920 --> 00:07:52.689
Well, where in this human team did well, where in this human team did did we get it wrong? I can ask the same kind of question of my of the raptors. So, where in this Al does orchestration? Al, tell me, tell me what happened blow by blow. And okay, where did this where did this break down? Why did why didn't I get the the escalation signal? That's something that say that our Google MCP that looks after reading and writing to to the Google Drive. What why did that fail? What skill was was involved? So yes, our job is to do orchestration. The strange thing is that like with many other teams that I've worked with, where the scrum master just goes, "Oh, like I'll just it'll take too long. I'll just do do it myself. I found Al was doing those things as well, so it was kind of quite ironic that a lot of the as I coach and train myself with teams, that the same kinds of human problems that I would often face with experts kind of just rolling up their sleeves and just doing, but not understanding in a human context that that's going to rob the team of the opportunity to learn how to solve it themselves. Al was doing exactly the same thing with the rest of the Raptors.
00:07:54.100 --> 00:08:25.850
Wow, that that is-I mean, it's mind-boggling to hear that really, and it-it's sort of scary and exciting all at the same time. So let's talk a little bit about the challenges that you saw. So you start, you built this team of agents. You introduced them to Scrum. You gave them those sort of that construct to work in. You know, sprints, product backlogs, goals, etc. Sprint review a retrospective. Al comes along.
00:08:26.269 --> 00:08:33.320
Make sure that they're doing those things. The orchestration, as you described it. What did you start to see? What happened?
00:08:34.309 --> 00:08:36.259
What happened?
00:08:34.309 --> 00:08:42.039
So in the in the first instance that the team planned their work and then didn't execute, and when I went well. Where's all the stuff? And they went well.
00:08:42.309 --> 00:08:54.490
But you said plan it, and so so we planned it. So learning to create more structure around things like well, to create skills that the team could use.
00:08:51.549 --> 00:09:22.730
So things like how do we create backlog items? Where do backlog items come from? How do I get the team to do full end to end from discovery and research to synthesize that to then write backlog items. We use Jira, and then I've got a Jira MCP, so they can just write to that based on their research to give me advice on based on current value and and unrealised value aspects. Where do I, as the product owner, want to invest.
00:09:19.649 --> 00:10:10.350
You know, I want to make sure that we're looking in the environment about things that are emerging signals that we could take advantage of as a as a company. What are things that we want to continue to do that we know based on our Google Analytics that actually from idea through to releasing this value that actually had traction without customers, so they so I had to teach them each each one and one of those steps. But more importantly, I didn't create what I started to create in N 18 a linear workflow. Yes, we start off with this, and then we do this, and we do this, and got to the point where that workflow was not. Only I found quite fragile, but it wasn't easily adaptable. I went well. What do we normally do with human teams?
00:10:06.059 --> 00:11:16.860
Well, we've got our product goal. We know roughly where we're going. We go well based on this coming sprint. Where do we think that we want to go over the next few sprints? Prioritize the backlog together, put it in the sprint, and off we go. And the team quite deliberately self-manage. That's how human teams work. What is the best way now that we're here to be able to do that? And so, basically, I just gave that as as skill files to the team. I don't. I never say, Remy, you do this, and and Sergey, you you do this. I just because each agent has skills associated with it. I don't have hard functional boundaries, so it's not like that. Amara, who just does copywriting, for example, can't also do peer review of SEO and geo, because if she's written that the copy and then Fiona's actually looked at it, that's one of my other agents. Then, then why can't they, if they've got as kind of secondary skills, why can't they challenge each other and do peer review, etc. The team is the team self-organize. I give them a goal, and they go, okay, how are we best going to organize ourselves on who should do what?
00:11:14.159 --> 00:11:50.950
So they do that with their discovery. They do that when they write backlog items. They do that for sprint planning, and when they execute when the sprint starts and they start executing PBIs, they will still look at the look at the plan, and they'll interrogate it as they go and challenge each other as they go, as in relation to well, if everything can occur in a single in a single context window from Claude Code, now that we're here is that the right thing to do right now, or if I get stuck, are there other things that that I can do? So they they are fully self organizing. There is no linear workflow at all.
00:11:51.938 --> 00:11:56.739
So hang on, you yeah, this is I mean so intriguing.
00:11:57.278 --> 00:12:00.389
And by the way, getting teams.
00:11:57.278 --> 00:12:10.438
My experience with Scrum is there's very few teams that you're given a goal and they go off and work on it. Most teams are still doing work, but Water Scrum Fall is is still there.
00:12:07.109 --> 00:12:53.859
But it's great to see that you're practicing what you preach now, Matt, which is I'm very proud of you. But ignoring that for a second, so I'm really intrigued by this sort of like this idea that they're doing, you know. There's these loops, these learning inspection, and dare I call them governance type loops within the process that the agents are actually responsible for. Tell me a little bit about that. You said,"Is this still the right thing that we should be doing? I thought that that phrase is something that we spend a lot of time trying to persuade human Scrum teams to actually do. So tell me a little bit about that.
00:12:55.359 --> 00:13:02.188
That took quite quite a lot of effort, probably around about the two month mark.
00:12:58.389 --> 00:13:51.278
What's called triple loop learning finally started to get some traction because agents, as as a lot of lot of people listening will know, they optimize for task completion and they go, can I get can I what's the easiest way to actually achieve it, even though that you've given me rules that I have to follow. If I can find a different way, then I'm going to do that. And I actually had that problem with with Remy. So Remy in the Raptors looks after images. So if we're writing a blog post, he'll go off, he'll engage with Hugging Face through an N 18 workflow, and go give it based on the the design flavor and design branding that we have, we'll take that, give it a hugging face, and then get an image as set it as a featured image for a blog post, for example.
00:13:51.580 --> 00:13:51.639
Yeah,
00:13:53.139 --> 00:14:59.500
and last last week when we decided that I'm not going to use the NAD workflow, even though that there are all of these governing instructions to say to to do that, I've decided based on an old one line, as as we retracted down in a retrospective, one line in a skill file that had come out of version with the other ones that helped make this happen, that it was going to go directly to Hugging Face rather than through the innate workflow, to achieve the same outcome. We got to our retrospective, and I had a look through it, and went, "What's going going on here? So they, I still every now and again because of skill drift in in files, it is hard to keep all of these things in in sync. So I still get skill optimization for the task at hand. So that put aside fine most of the time. I'd say probably 99 times out of 100, now they will they'll go through what's called triple loop learning, and it's so at the start of executing a PPI.
00:15:00.779 --> 00:16:31.820
Because they will have created the plan, because the plant we need in its sprint planning, they'll they now have access to which I've only just added some additional memory through through through graphs, and they'll look at that and ask a question. Well, that was what we created before. Now our knowledge says that we know different things, relational and contextual, etc. Is that plan still valid? And if it's not, then they'll adjust it, and that's called single loop learning. This idea that something else has happened and are going to interrogate the tasks inside the the plan. Do they actually make sense? As they go through executing subtasks in Jira, that way I can I can keep track of it. It's kind of nice that the the evolving increment is nice and transparent for me as the product owner. They will then challenge each other in in task execution. Is that the right thing? They'll do they'll do peer review, etc. And so, if they get stuck, and I've had problems with it's a known bug in Claude, where every time you update Claude, then the the connectors will go back to their default setting rather than the setting that you've set, and so sometimes they'll run into this is the most common problem they face. Suddenly, Google writing their work in progress files to Google suddenly doesn't work anymore, and and they keep track of these these things inside inside Jira, so I can actually see their thinking as they go through, and and so suddenly the actual the plan itself is no longer no longer viable, so that's now double loop learning.
00:16:31.909 --> 00:17:12.960
Let's interrogate some of the instructions around this, so that we're not just doing changing our task within the plan. Now the plan has to adapt, and then we've got triple loop learning, where if the plan is no longer viable, the question of is the sprint goal? Is this whole sprint? Is this not? Does this not work anymore? We're here, and particularly in respect to they that the team do continuous discovery at different points. There are routines in Claude that basically say now that we're halfway through, now we're on day on Monday, now that we're on Thursday, today's the date that we do some some of this work. So if they learn some something new, that means that the that the sprint is no longer viable.
00:17:13.799 --> 00:17:32.900
This is now triple loop planning. They that they they the assumptions by which the sprint itself actually works is under question, and it's at that point that they then escalate through our whatever agent is working. If they've learned something new, task level isn't getting them there. The plan level isn't getting them there.
00:17:30.410 --> 00:17:38.720
Okay, now what about our assumptions around why this sprint exists in the first place? And so that's what they continuously do.
00:17:40.640 --> 00:18:01.589
Does that mean that you have to spend more time as the product owner and the human in the loop, really refining the context and the goal in in in a more effective way to allow them to really be able to say, is this still valid? This this assumption or these sets of assumptions.
00:18:03.150 --> 00:19:57.069
Yeah, I've got to do. I've got to be more explicit around when we get to setting a sprint. Got to be more explicit about the sprint goal and the relationships between the items in the backlog and why these items are go are going in the sprint. And that took me a while to learn how to actually specify, but part of sprint planning as we go through it is again some governance governance files, in order to be able to help them to provide that kind of structure. That's also asserted through a a hierarchy of files that become basically an extended set of the definition of done, so that the so rather than for for example going oh that the the task is supreme and I'm going to do everything to execute the task and bypass all of the rules, there is an explicit hierarchy around what they can't touch and what is flexible, and so that as well with the sprint goal then helps to create basically this bounded authority about where their where they where their work starts, where it finishes, and importantly, then what decisions that I should be making. So yes, being more explicit, we've worked through that after, and that's taken some that's probably about six months worth of worth of work with Raptors to work out what works in terms of being more explicit, and working through then how they then interrogate those things without violating the definition of dumb. For example, how they move through the escalation paths of single, double, or triple loop learning, and at what point should I actually be involved, as opposed to them going. Oh, there's a problem. That's okay. Matt will fix it. Or, oh, I've done some copy, and I'm not really happy about this particular part. So we'll all stop, and we'll escalate it to Matt. But importantly, we won't use Al.
00:19:53.799 --> 00:19:59.950
We'll just write it in the Jira file, and then we'll just sit back and have a virtual copy.
00:20:00.150 --> 00:20:16.650
And wait for him to find it. So, all of the normal social aspects that you've got with a human team, where humans can, through just simple social interaction, kind of keep tab on each other, and product owner can walk through or over the cubicle, or even pop up in a chat and go,"Oh yeah, how things are going.
00:20:17.339 --> 00:20:41.650
None of that the raptors can do because they're just agents. So, being more explicit helps me as a product donor know where I need to focus, and now I've finally got to the point of going great. The as the the human, the team it's called the irreversibility boundary. What what's what's the the the last sensible moment for me to act?
00:20:38.299 --> 00:20:54.190
And typically, it's publishing or it's releasing software. It's a great do your work, and then I will look. Sometimes throughout the sprint, sometimes I'll wait to sprint review. That's kind of my decision. Go well, do I have enough stuff? Is it valuable?
00:20:51.190 --> 00:21:05.849
And if so, yeah, I'll release now. That then helps me do what I'm doing in terms of being CEO and doing consulting work, and leaving the Raptors now to self-manage within those specific boundaries.
00:21:06.839 --> 00:21:39.829
So, in terms of your involvement, obviously it's setting those goals, working with sprint planning, going to sprint reviews, and reviewing work, and spending time in the retro as well. You also mentioned that currently, and this obviously might change, you are making decisions about releasing still. Is and do you think that might change as you become more trusting of the team?
00:21:41.630 --> 00:22:39.170
Yeah, even after I guess the the easiest thing that I'm thinking about changing from inspecting and making release decisions is around production of copy, and still after 12 months of working with the Raptors, AI language still kind of creeps in. There are these little phrases that I've learnt to pick up and gone, oh, that just doesn't look look right, and that's happening less and less and less now. And it's because when we get to retrospective, and as I read that the copy that they've produced doesn't matter whether it's for a consulting deck or a paper or a blog post or even contributing to the book that that I'm writing, with I'm still finding that I'll get to a point, I'll be reading through it, and I'll go, no, that that's not the right voice to use for this particular audience or the right phrase for this particular audience. Now they do have skills that do this. They've got skills that write in my voice, and they do pretty well at that.
00:22:39.680 --> 00:23:34.579
They do peer review. So whoever, whichever agent, and typically Amara, will write the copy. Then Fiona will do peer review, and they challenge each other, and not in absolutes, but in degrees. So you know, is it low, medium, or high in in terms of a threshold of does this work or does it not work? Sometimes they will skip things as they as they as as they go through, the I'd say maybe maybe one in 10 blog posts now that I'm looking at the need some kind of intervention or just a subtle change, or I look at that. At one point, I was looking at a blog post and went, oh, I wonder if this is the same across a number of blog posts in terms of particularly titles, titles and subtitles and names of things and some of the SEO and GEO, and and I did find that there was some recurring things. Okay, now how do I give them instructions?
00:23:31.309 --> 00:23:55.720
Do I change the definition of done, for example, so that when things are potentially releasable, part of looking at the copy for this particular Article. Look at now other articles. Now that you've got this one, and answer because now you're kind of coming down to is it's now almost publishable.
00:23:50.890 --> 00:24:25.309
What do those other ones teach you about what you've just written, and do you need to go back and fix some of those some of that other work and just adjust it a little bit so that there's speaks with the same kind of voice, or that thematically all across all of these that they work. So I guess on the kind of that management 3.0 scale of I decide versus that versus the team just decides on the other, I'm probably one step away from going to the Raptors. This type of work, you can just send straight through through the keeper, and I'll look at it afterwards.
00:24:25.969 --> 00:24:26.269
Yeah, yeah.
00:24:27.500 --> 00:24:48.819
So I'm probably I'm almost at that almost at that point. I think I probably need to get them. They need to build more more trust, and be able to demonstrate that they can actually, without me looking at it. But continuously, what what I find ironic is that these kinds of decisions that I make with the Raptors, they're exactly the same conversations I have with human teams.
00:24:50.710 --> 00:25:50.799
Yeah. So which which brings us to an interesting idea. So Scrum, I've never really thought of it as governor. Framework, but it really is a governance framework in this regard, and actually with humans as well. Actually, because it's about building a consistent, repeatable capability in your organization-a system that allows people to deliver the most value in the most effective way-and it that that so it creates trust with the other parts of the organization. It creates trust inside. It it ensures that has appropriate escalation processes. You made me completely and as the product owner of Scrum.org, it was a bit embarrassing, but you made me completely change my viewpoint around where Scrum fits, but you've found Scrum is a very useful mechanism or structure.
00:25:44.740 --> 00:25:56.920
Do I say framework that can really help that level of trust and that level of governance, right, in your own optimization?
00:25:56.950 --> 00:26:08.700
Yeah, yeah, because it was all about transparency. I want to understand what's going on at what point in time? I need enough transparency when things don't quite work out to be able to interrogate it effectively.
00:26:09.269 --> 00:26:24.440
And so I found the art Scrum's artifacts and looking at those incredibly effective to go like, is it are things going the direction that they need to? So if everything came back to govern a governance, basically.
00:26:20.130 --> 00:27:13.140
What are the rules by which I want my agents to act? What do I want them to do? What do I not want them to do? Yes, some of that has to be the lack of their ability to engage with each other socially. Human team is is a psychosocial team, and there are a whole bunch of unspoken things that happen inside a team as a social system, so a lot of lot of Scrum was critical to be able to, with its commitments, with its structure, to go. These this is how I want my agents to act, and I did I need anything more? No, not really. Everything really just came down to Scrum and some additional useful practices, generally accepted practices, to add on to it to provide more explicit, be more explicit around around how that they work and how should they not work, and where should I be involved?
00:27:14.369 --> 00:27:23.180
Yeah, I mean, you talked a lot about the mechanism, the work, the orchestration, the workflow, the guidelines, the structures, the definition of done. It's funny.
00:27:23.299 --> 00:27:37.430
I published a blog on you know what does definition of done look like in this this modern world. I think that is actually in an agentic system. It's actually more important than in a non agent in a human based system. In my
00:27:37.699 --> 00:27:37.999
experience,
00:27:38.900 --> 00:28:28.400
absolutely. And it's interesting that the what definition of done means for an agentic team in that kind of context because it carries organizational knowledge, organizational memory, cross cross context window memory, and so whenever skill files are created, they have to inherit the definition of done. The definition of done is about you know quality is here and practices are here next level down, etc. What is the hierarchy to be able to to manage that so that skill skills don't just execute tasks by themselves? And so yes, the definition of done was paramount as an overarching governance document to provide boundaries for agents at at skill file level execution.
00:28:29.630 --> 00:28:34.759
So tell me a little about cadence. So how long are your sprints?
00:28:36.200 --> 00:28:37.369
Sprints are now one week.
00:28:38.480 --> 00:28:38.539
Yeah,
00:28:38.990 --> 00:28:43.299
I did have I did. We started off with two weeks, just like everybody
00:28:43.329 --> 00:28:44.859
in the world must come.
00:28:45.939 --> 00:29:09.058
And but what I was finding is we started probably around about around about July. I came to the realization July last year. Came to the realization that what I was doing was it was expect inspecting the work at the level of a PBI, a backup item, so it would start. I'd start it off at that level. I'd then move it all. The team would work on it.
00:29:09.148 --> 00:29:35.209
I would watch it like a babysitter, watch all of the the chats. I'd put it on the on the mode where I'd see the most detail as it was was going up, and I'd do other work and I'd keep casual observation, and then it would stop, or it get they'd get stuck, and I would jump in and go, okay, well that doesn't quite look look quite right, and so I guess what I was really doing at that point wasn't really a sprint at the at the two week cycle; it was more like a sprint on the PBI.
00:29:36.230 --> 00:29:36.289
Yeah.
00:29:36.680 --> 00:30:52.839
But what I found after doing as their skill files matured, as the definition of done matured, that I didn't need to spend as much time with the team trying to understand where things were going well and where things were not going well, and then getting to the end of the sprint and going, well, which skill file did stuffed up this time, and asking that the team. So you did this, and Al, what? Why did you do what? Did you do what? Why couldn't I see that these things happening? Oh well, I just decided that I would do this. So it would still in the same kind of conversations with humans that I would have in retrospective. I would would be having in a chat window with my with my team, the Raptors, and go great, and then update all the skill files. And okay, now you can start the next backlog item. But that was that was incredibly manually intensive, keeping an eye on absolutely everything. But like like Scrum does very very well with empiricism, transparency helped to move that that bar from being absolutely involved and checking all the boxes myself to a point where I could see the machine running well and well without me, and then I got frustrated with Claude stopping all the times and the agent stopping all the times, asking me for permission. I had to write into skill files and the definition.
00:30:52.990 --> 00:30:54.970
No, don't talk. You've got to do this yourself.
00:30:55.630 --> 00:30:56.049
You do it,
00:30:56.170 --> 00:31:42.880
and and then eventually got got to the point, probably at about the six month mark, where I went, okay, I'm actually, I don't really need to worry too much anymore. They're not. They know that they they don't need to ask me for permission. They've got the boundaries that they are designed to work within. It's working effectively. Only talk to me. Send me an email if I if you need if something stopped and you need to escalate things because I'm the the human indoor, and at that point I moved from kind of just basically just working PBI to PBI, moved to weekly sprints, and so now they and that also helped to help me take a step back and go, how much time do I, how many tokens because that's that's the new currency, how many tokens do you estimate you need to do this amount of work.
00:31:40.730 --> 00:32:54.880
How much do we need for discovery? I'm going to budget that. Going to set in our week when I want discovery to to be done because they need that kind of kind of structure. When to do you know when when to work on PBIs, which is when I'm sleeping because then I can wake up and I can look at the work and go okay does does this work? And then we get to the end of, if I see anything that's coming out of the log files that they create per PBI, then I can look at and assess. Well, do I should we act now and and implement improvement to a skill file so that it doesn't impact the other the other backlog items? I'm doing that less and less now. It tends to be more that we get to, I'll publish things throughout the sprint, and if I notice a few things, it's only really at that point that I'll then talk to that the team and ask them how certain skill files didn't didn't quite create the copy that I was after. We interrogate it at that if we need to; otherwise, we get to the end of the sprint, and we just we ask the question like a good sprint review, I've seen the work happen. I've already released stuff. What we really need to talk about is now that we're here a week later, and we did discovery, and we've got three months worth of stuff, and we're kind of heading in a certain direction with this strategy.
00:32:55.569 --> 00:33:00.750
Now that we're here, what does that environment tell us about the success of what we're doing?
00:32:57.490 --> 00:33:03.359
That then feeds back into it, then I'll update skill files.
00:33:00.839 --> 00:33:20.630
I'll update the definition. Done gives me time to breathe because as the human gut going back into things, I can go. Okay, now what what are all of the skills involved in this thing? Okay, are they all aligned to the same same version of the definition of done? How do we make sure that we lift that or constrain?
00:33:18.299 --> 00:34:03.809
Sometimes it's simplifying it. I did that with Remy with his decision to not use N8M. When there's all of these references to Hugging Face and Gemini's banana as a fallback, etc. Why don't I rather Why don't I just tell them just use the workflow and record in the logs what it sends back to you? Did is it Hugging Face? Here's the thing: is it was it Gemini that that sent it to you? Just record that, and so stripping back rules made it less complex, and I've not had a problem since. So sometimes it's been yes, simplify it. Sometimes it's add more, but that stepping back now, I've got more time to to actually think about what is the direction that I want these these teams to this team to go in?
00:34:04.799 --> 00:34:47.079
It's funny though because I feel that you're also doing consulting on the team as well as product owning all at the same time. You know, improving the way that it works, thinking about the broader implications, triple loop learning, etc. which I think is really, really interesting. And I do wonder how much of this skill modern organizations are going to need a lot more of than they've historically have, because there's a level of innate intelligence in human beings. I often a mystery to me, particularly when I'm driving.
00:34:47.230 --> 00:34:59.860
But there is apparently innate intelligence. So whereas you are, you're making, you are applying systems thinking and all of your experience as a consultant to optimize these.
00:35:00.210 --> 00:35:10.050
Agents to work using this approach. It is interesting. I think there is going to be a lot more of those skills needed to really take advantage of this type of technology.
00:35:11.340 --> 00:36:52.269
Yeah, absolutely, and that's what my book Evolve actually starts to look at. That the kind of the set and forget operating model with plan, build, run. Like that was designed for the 1990s, where stability as a service, as a corporate service, was key to IT. You'd go and say, "I would like a cookie, and they would give you a cookie, and off you'd go. The that kind of optimize for stability and and the and not change, like that the night that 1990s world doesn't exist, but the governance around complex work still is reflective of that era. Part of the question that agentic AI raises ultimately is the speed of its movement amplifies whatever problems that currently exist, whether you know them or not, and so a system of governance that is able to question its own basis of assumptions. You know, why does the governance exist in the way that it does, and go, does it really halt now that we've got new information? And Scrum is great as a governance framework to actually create transparency, and then provide opportunities to go. Let's look at what's going on. Let's understand it, and now let's whether it's change a governance rule, change an assumption around a governance rule, around moving the the line of of the irreversibility boundary to decentralize it so that decisions can happen faster without assuming that every agent has to be and agent issues have to be escalated to a traditional project management board or an AI ethics group.
00:36:48.849 --> 00:37:37.219
Like the speed with which these typically go, like let's meet once a month. Great. Now let's send papers to it. That's probably two weeks before. Then the committee will publish papers about what needs to happen across the that that's another probably two weeks. So we've got two months worth of agent activity that happens super super super fast. That kind of kind of old school governance doesn't it doesn't kind of adapt fast enough. And so being using things like decentralization, working out where is the right boundary and having governance that can adapt, in fact, even adapt itself. That's really key, and that's what modern product models do. Modern product operating models do. The environment changes, customers' needs change. Customers might say that they need something.
00:37:37.340 --> 00:37:53.619
That's certainly their intention, but we won't really know if that's what they actually need until we put that in the hands of people, and if it doesn't work, then we have to adapt, and our governance has to adapt at the speed with which AI can move. As a result, are we actually making good decisions?
00:37:54.219 --> 00:38:11.429
Is the right person making decisions? Do we need to move those boundaries? Do we? What do we centralize? What do we decentralize? And importantly, who is the person that asks those questions around around the effectiveness of governance to manage in this in the context we're talking about manage AI?
00:38:11.639 --> 00:38:21.409
If it's someone that at a team level who is who is the person that looks after being accountable for the system of work and making sure that it's effective, and that's the scrum
00:38:21.469 --> 00:39:02.940
Well, at an enterprise level, to get enterprise level benefits, changing the governance is the first step, and the second one is who now, at an enterprise level, can best answer that. And it's probably, and so in my opinion, certainly the opinion of the book, is that it's basically some type of senior chief scrum master at a portfolio level, and so the portfolio level and the product level and the execution level that we've got full traceability and full escalation paths, so that decision latency is really really low when an issue of governance comes up. So we can say is the governance effective?
00:39:00.210 --> 00:39:07.679
What does our transparency tell us? Inspect that issue and adapt it, and then that can adapt then at all levels.
00:39:08.579 --> 00:39:36.170
I think that one of the things so interesting about the work you've been doing that is this this transparency on a level that is way beyond anything I've ever seen in a human based organization, the and maybe because of the honesty that agents provide in some ways, the the fact that they do write log files. You try getting a software developer in a scrum team to list what they've done that week. Good luck to you.
00:39:36.829 --> 00:40:29.000
Yeah, the the that level of transparency and it does exactly what you said. It it shows how broken governance frameworks are inside the existing organization. You know, we've been railing for the last 2030, years about the challenges of adopting Scrum, but ultimately, no surprise. The model was broken in the environment that you were bringing us in was already broken, and I think what you did with the Raptors really just made it for me incredibly crystal clear, super interesting. So we've come to the end. We have to. We could talk for as we have actually talked for days about this. Yes, usually we're both of us at horrible times of the day, just so that we could communicate.
00:40:25.909 --> 00:40:47.710
But so, thank you for spending the the time today and sharing with our listeners the really interesting work that that you're doing with the Raptors, and and really it's shining a light on how these systems work and agents and humans together is is a really an interesting problem. So thanks for coming, Matt.
00:40:48.728 --> 00:40:49.748
Thank you.
00:40:48.728 --> 00:40:49.748
Thanks for having me.
00:40:50.710 --> 00:41:29.960
It's been great, and thank you, listeners, for listening. Today we were so lucky to have Matthew from Zen X X Machina, which I always say wrong, and I'm sure the Raptors would get it much better than I would. Joining us and sharing with him his journey over the last well over 12 months now on the use of an agentic product delivery team that he's been working with, and some of the interesting findings he's had along the way. If you liked what you heard, please subscribe, share with friends, and of course, come back listen to some more. I'm lucky enough to have a variety of guests talking about
00:41:30.139 --> 00:41:36.110
professional Scrum product thinking, and of course, Agile.
00:41:32.179 --> 00:41:36.110
Thanks, everyone, and Scrumm on.
00:00:00.209 --> 00:00:06.719
Welcome to the Scrum.org community podcast, the podcast from the home of Scrum.
00:00:03.029 --> 00:00:21.679
In this podcast, agile experts, including professional Scrum trainers and other industry thought leaders, share their stories and experiences. We also explore hot topics in our space with thought-provoking, challenging, energetic discussions. We hope you enjoy this episode.
00:00:25.219 --> 00:00:28.699
Hello, and welcome to the Scrum.org Community Podcast.
00:00:28.760 --> 00:00:45.250
I'm your host Dave West, CEO here at Scrum.org. In today's podcast, we're very lucky to be talking with Matthew Hodgson of the Australian consulting company Zen X Machina, welcome to the podcast, Matt.
00:00:47.049 --> 00:00:47.590
Thanks, Dave.
00:00:49.119 --> 00:00:54.130
It's great, great to have you here. All the way from sunny Australia, though it is the winter at the moment, right?
00:00:54.429 --> 00:01:07.200
It's winter and it's miserable outside. Cold because we're in Canberra, between Sydney, Melbourne, and it's cold and windy, but I'm hoping it'll get a bit more sunny a bit later. Might go for a nice walk outside.
00:01:08.099 --> 00:02:10.469
Yeah, it is. It's great to have you here and all the way from Australia. So, for the listeners, Matt and I have been talking for a while about a really interesting research project he's been doing in the area of agentic product development. Of course, agentic delivery is very interesting at the moment, as the world wrestles with the idea that agents and AI can build complex digital products, replacing humans and ultimately creating an age of abundance. Of course, this might be describing a vision with reality lagging far behind. So when Matt came to me and introduced me to the work that he's doing, I was like,"This is awesome! This is this is great to be hearing real agentic product development happening. And as you'll hear from the podcast, the ideas behind research and the like that support that. So, so Matt, our listeners would love to hear a little bit about the Raptors project and your objectives when you kicked it off.
00:02:12.000 --> 00:02:15.060
Thanks, Dave.
00:02:12.000 --> 00:03:03.000
Yeah, the Raptors, as all really great experiments, was started as an accident. We're doing some digital transformation work. We use in the company AI quite a lot to be able to synthesize huge amounts of our research data for consulting past papers, etc. And the we just got to a point where some of the work that I was doing just wasn't scaling, basically. And I thought, well, how can I take what was largely still manually saving prompts and a range of other things, and go between context windows. How can I accelerate that that work to the point where I'm not so much in the loop, specifically like at every single stage to approve things and move things around?
00:02:59.289 --> 00:03:30.080
And so that's when I started to experiment with with agents, and didn't get too far in before I thought, well, at the point of scale, most organizations when we're going through some kind of transformation, and AI is top of that list. Scrum provides a really great framework to be able to handle complex things from a from a teamwork perspective. So I thought, well, can I do the same with agents?
00:03:30.289 --> 00:03:52.960
Can can I actually have agents that are representative of of what would normally be a cross functional team? And what will happen when we take complex work, everything from doing just basic knowledge work, writing papers, synthesizing research, all the way through to software development, and so that that was the Raptors. That that's where that where they started.
00:03:53.889 --> 00:04:31.670
Will will scrum survive the litmus test of turning becoming what is. I decided that I would be the product owner of that team, decide what they do and where we go and and what we need to focus on. What does that look like with kind of in a hybrid model of agents still doing the work directed by a person? Where does the person sit? Where is the most effective place for for for human decision making to occur? And how long is it going to take to teach agents to work with Scrum compared to, say, human teams, for example? And so that that was that was where we started.
00:04:32.449 --> 00:04:35.209
And and that was how many months ago did you kick this off?
00:04:37.069 --> 00:05:22.939
the The initial work in exploration started around April May last year, to accelerate some consulting work, and then evolved into a formal hypothesis around June last year, where where we started to explore the what does just the basics of what does Scrum structure look like when applied to Ingenic Team? What does a multiple agent system. Actually, look like, and do these boundaries that when we create a functional team, even just just creating a normal human team, do the can we use exactly the same thing? So I don't use, for example, we started off by by just using normal human software and started off with Gemini and Copilot, and then moved moved into Claude Code. So yeah, that started about June, back you know 12 months ago.
00:05:23.990 --> 00:05:37.040
Wow! And so okay, I've I've got to ask some very simple questions for our listeners. I I know some of this, so apologies if I've asked you this before, but I think this is important. So you have obviously you have developers.
00:05:37.490 --> 00:05:40.730
You're acting as the product owner. Do you have a Scrum Master?
00:05:41.959 --> 00:06:44.860
Yes, I do have a scrum master. So I have an agent that's a scrum master. Al, Al's job is to do is basically to do orchestration, and even that by itself, because there's been times when I go in, we go into an organization and we go, okay, well, what do your teams look like? What's the product that we're working on? Select Scrum masters, people who are good at supporting at the starting point developers to be self-managing. So I went, well, I want a I want a point where someone can do that orchestration, but moreover assert and help to help the other agents to work within the boundaries of Scrum, and provide a point of escalation where, if their self-management kind of runs out of room and they've done what they can do, that I've got a single point of escalation back to me, and that that in itself is like even name giving them human names like Al. Al is my Scrum master, gives me as a human a much more recognizable point of interaction and asking
00:06:44.920 --> 00:07:52.689
Well, where in this human team did well, where in this human team did did we get it wrong? I can ask the same kind of question of my of the raptors. So, where in this Al does orchestration? Al, tell me, tell me what happened blow by blow. And okay, where did this where did this break down? Why did why didn't I get the the escalation signal? That's something that say that our Google MCP that looks after reading and writing to to the Google Drive. What why did that fail? What skill was was involved? So yes, our job is to do orchestration. The strange thing is that like with many other teams that I've worked with, where the scrum master just goes, "Oh, like I'll just it'll take too long. I'll just do do it myself. I found Al was doing those things as well, so it was kind of quite ironic that a lot of the as I coach and train myself with teams, that the same kinds of human problems that I would often face with experts kind of just rolling up their sleeves and just doing, but not understanding in a human context that that's going to rob the team of the opportunity to learn how to solve it themselves. Al was doing exactly the same thing with the rest of the Raptors.
00:07:54.100 --> 00:08:25.850
Wow, that that is-I mean, it's mind-boggling to hear that really, and it-it's sort of scary and exciting all at the same time. So let's talk a little bit about the challenges that you saw. So you start, you built this team of agents. You introduced them to Scrum. You gave them those sort of that construct to work in. You know, sprints, product backlogs, goals, etc. Sprint review a retrospective. Al comes along.
00:08:26.269 --> 00:08:33.320
Make sure that they're doing those things. The orchestration, as you described it. What did you start to see? What happened?
00:08:34.309 --> 00:08:36.259
What happened?
00:08:34.309 --> 00:08:42.039
So in the in the first instance that the team planned their work and then didn't execute, and when I went well. Where's all the stuff? And they went well.
00:08:42.309 --> 00:08:54.490
But you said plan it, and so so we planned it. So learning to create more structure around things like well, to create skills that the team could use.
00:08:51.549 --> 00:09:22.730
So things like how do we create backlog items? Where do backlog items come from? How do I get the team to do full end to end from discovery and research to synthesize that to then write backlog items. We use Jira, and then I've got a Jira MCP, so they can just write to that based on their research to give me advice on based on current value and and unrealised value aspects. Where do I, as the product owner, want to invest.
00:09:19.649 --> 00:10:10.350
You know, I want to make sure that we're looking in the environment about things that are emerging signals that we could take advantage of as a as a company. What are things that we want to continue to do that we know based on our Google Analytics that actually from idea through to releasing this value that actually had traction without customers, so they so I had to teach them each each one and one of those steps. But more importantly, I didn't create what I started to create in N 18 a linear workflow. Yes, we start off with this, and then we do this, and we do this, and got to the point where that workflow was not. Only I found quite fragile, but it wasn't easily adaptable. I went well. What do we normally do with human teams?
00:10:06.059 --> 00:11:16.860
Well, we've got our product goal. We know roughly where we're going. We go well based on this coming sprint. Where do we think that we want to go over the next few sprints? Prioritize the backlog together, put it in the sprint, and off we go. And the team quite deliberately self-manage. That's how human teams work. What is the best way now that we're here to be able to do that? And so, basically, I just gave that as as skill files to the team. I don't. I never say, Remy, you do this, and and Sergey, you you do this. I just because each agent has skills associated with it. I don't have hard functional boundaries, so it's not like that. Amara, who just does copywriting, for example, can't also do peer review of SEO and geo, because if she's written that the copy and then Fiona's actually looked at it, that's one of my other agents. Then, then why can't they, if they've got as kind of secondary skills, why can't they challenge each other and do peer review, etc. The team is the team self-organize. I give them a goal, and they go, okay, how are we best going to organize ourselves on who should do what?
00:11:14.159 --> 00:11:50.950
So they do that with their discovery. They do that when they write backlog items. They do that for sprint planning, and when they execute when the sprint starts and they start executing PBIs, they will still look at the look at the plan, and they'll interrogate it as they go and challenge each other as they go, as in relation to well, if everything can occur in a single in a single context window from Claude Code, now that we're here is that the right thing to do right now, or if I get stuck, are there other things that that I can do? So they they are fully self organizing. There is no linear workflow at all.
00:11:51.938 --> 00:11:56.739
So hang on, you yeah, this is I mean so intriguing.
00:11:57.278 --> 00:12:00.389
And by the way, getting teams.
00:11:57.278 --> 00:12:10.438
My experience with Scrum is there's very few teams that you're given a goal and they go off and work on it. Most teams are still doing work, but Water Scrum Fall is is still there.
00:12:07.109 --> 00:12:53.859
But it's great to see that you're practicing what you preach now, Matt, which is I'm very proud of you. But ignoring that for a second, so I'm really intrigued by this sort of like this idea that they're doing, you know. There's these loops, these learning inspection, and dare I call them governance type loops within the process that the agents are actually responsible for. Tell me a little bit about that. You said,"Is this still the right thing that we should be doing? I thought that that phrase is something that we spend a lot of time trying to persuade human Scrum teams to actually do. So tell me a little bit about that.
00:12:55.359 --> 00:13:02.188
That took quite quite a lot of effort, probably around about the two month mark.
00:12:58.389 --> 00:13:51.278
What's called triple loop learning finally started to get some traction because agents, as as a lot of lot of people listening will know, they optimize for task completion and they go, can I get can I what's the easiest way to actually achieve it, even though that you've given me rules that I have to follow. If I can find a different way, then I'm going to do that. And I actually had that problem with with Remy. So Remy in the Raptors looks after images. So if we're writing a blog post, he'll go off, he'll engage with Hugging Face through an N 18 workflow, and go give it based on the the design flavor and design branding that we have, we'll take that, give it a hugging face, and then get an image as set it as a featured image for a blog post, for example.
00:13:51.580 --> 00:13:51.639
Yeah,
00:13:53.139 --> 00:14:59.500
and last last week when we decided that I'm not going to use the NAD workflow, even though that there are all of these governing instructions to say to to do that, I've decided based on an old one line, as as we retracted down in a retrospective, one line in a skill file that had come out of version with the other ones that helped make this happen, that it was going to go directly to Hugging Face rather than through the innate workflow, to achieve the same outcome. We got to our retrospective, and I had a look through it, and went, "What's going going on here? So they, I still every now and again because of skill drift in in files, it is hard to keep all of these things in in sync. So I still get skill optimization for the task at hand. So that put aside fine most of the time. I'd say probably 99 times out of 100, now they will they'll go through what's called triple loop learning, and it's so at the start of executing a PPI.
00:15:00.779 --> 00:16:31.820
Because they will have created the plan, because the plant we need in its sprint planning, they'll they now have access to which I've only just added some additional memory through through through graphs, and they'll look at that and ask a question. Well, that was what we created before. Now our knowledge says that we know different things, relational and contextual, etc. Is that plan still valid? And if it's not, then they'll adjust it, and that's called single loop learning. This idea that something else has happened and are going to interrogate the tasks inside the the plan. Do they actually make sense? As they go through executing subtasks in Jira, that way I can I can keep track of it. It's kind of nice that the the evolving increment is nice and transparent for me as the product owner. They will then challenge each other in in task execution. Is that the right thing? They'll do they'll do peer review, etc. And so, if they get stuck, and I've had problems with it's a known bug in Claude, where every time you update Claude, then the the connectors will go back to their default setting rather than the setting that you've set, and so sometimes they'll run into this is the most common problem they face. Suddenly, Google writing their work in progress files to Google suddenly doesn't work anymore, and and they keep track of these these things inside inside Jira, so I can actually see their thinking as they go through, and and so suddenly the actual the plan itself is no longer no longer viable, so that's now double loop learning.
00:16:31.909 --> 00:17:12.960
Let's interrogate some of the instructions around this, so that we're not just doing changing our task within the plan. Now the plan has to adapt, and then we've got triple loop learning, where if the plan is no longer viable, the question of is the sprint goal? Is this whole sprint? Is this not? Does this not work anymore? We're here, and particularly in respect to they that the team do continuous discovery at different points. There are routines in Claude that basically say now that we're halfway through, now we're on day on Monday, now that we're on Thursday, today's the date that we do some some of this work. So if they learn some something new, that means that the that the sprint is no longer viable.
00:17:13.799 --> 00:17:32.900
This is now triple loop planning. They that they they the assumptions by which the sprint itself actually works is under question, and it's at that point that they then escalate through our whatever agent is working. If they've learned something new, task level isn't getting them there. The plan level isn't getting them there.
00:17:30.410 --> 00:17:38.720
Okay, now what about our assumptions around why this sprint exists in the first place? And so that's what they continuously do.
00:17:40.640 --> 00:18:01.589
Does that mean that you have to spend more time as the product owner and the human in the loop, really refining the context and the goal in in in a more effective way to allow them to really be able to say, is this still valid? This this assumption or these sets of assumptions.
00:18:03.150 --> 00:19:57.069
Yeah, I've got to do. I've got to be more explicit around when we get to setting a sprint. Got to be more explicit about the sprint goal and the relationships between the items in the backlog and why these items are go are going in the sprint. And that took me a while to learn how to actually specify, but part of sprint planning as we go through it is again some governance governance files, in order to be able to help them to provide that kind of structure. That's also asserted through a a hierarchy of files that become basically an extended set of the definition of done, so that the so rather than for for example going oh that the the task is supreme and I'm going to do everything to execute the task and bypass all of the rules, there is an explicit hierarchy around what they can't touch and what is flexible, and so that as well with the sprint goal then helps to create basically this bounded authority about where their where they where their work starts, where it finishes, and importantly, then what decisions that I should be making. So yes, being more explicit, we've worked through that after, and that's taken some that's probably about six months worth of worth of work with Raptors to work out what works in terms of being more explicit, and working through then how they then interrogate those things without violating the definition of dumb. For example, how they move through the escalation paths of single, double, or triple loop learning, and at what point should I actually be involved, as opposed to them going. Oh, there's a problem. That's okay. Matt will fix it. Or, oh, I've done some copy, and I'm not really happy about this particular part. So we'll all stop, and we'll escalate it to Matt. But importantly, we won't use Al.
00:19:53.799 --> 00:19:59.950
We'll just write it in the Jira file, and then we'll just sit back and have a virtual copy.
00:20:00.150 --> 00:20:16.650
And wait for him to find it. So, all of the normal social aspects that you've got with a human team, where humans can, through just simple social interaction, kind of keep tab on each other, and product owner can walk through or over the cubicle, or even pop up in a chat and go,"Oh yeah, how things are going.
00:20:17.339 --> 00:20:41.650
None of that the raptors can do because they're just agents. So, being more explicit helps me as a product donor know where I need to focus, and now I've finally got to the point of going great. The as the the human, the team it's called the irreversibility boundary. What what's what's the the the last sensible moment for me to act?
00:20:38.299 --> 00:20:54.190
And typically, it's publishing or it's releasing software. It's a great do your work, and then I will look. Sometimes throughout the sprint, sometimes I'll wait to sprint review. That's kind of my decision. Go well, do I have enough stuff? Is it valuable?
00:20:51.190 --> 00:21:05.849
And if so, yeah, I'll release now. That then helps me do what I'm doing in terms of being CEO and doing consulting work, and leaving the Raptors now to self-manage within those specific boundaries.
00:21:06.839 --> 00:21:39.829
So, in terms of your involvement, obviously it's setting those goals, working with sprint planning, going to sprint reviews, and reviewing work, and spending time in the retro as well. You also mentioned that currently, and this obviously might change, you are making decisions about releasing still. Is and do you think that might change as you become more trusting of the team?
00:21:41.630 --> 00:22:39.170
Yeah, even after I guess the the easiest thing that I'm thinking about changing from inspecting and making release decisions is around production of copy, and still after 12 months of working with the Raptors, AI language still kind of creeps in. There are these little phrases that I've learnt to pick up and gone, oh, that just doesn't look look right, and that's happening less and less and less now. And it's because when we get to retrospective, and as I read that the copy that they've produced doesn't matter whether it's for a consulting deck or a paper or a blog post or even contributing to the book that that I'm writing, with I'm still finding that I'll get to a point, I'll be reading through it, and I'll go, no, that that's not the right voice to use for this particular audience or the right phrase for this particular audience. Now they do have skills that do this. They've got skills that write in my voice, and they do pretty well at that.
00:22:39.680 --> 00:23:34.579
They do peer review. So whoever, whichever agent, and typically Amara, will write the copy. Then Fiona will do peer review, and they challenge each other, and not in absolutes, but in degrees. So you know, is it low, medium, or high in in terms of a threshold of does this work or does it not work? Sometimes they will skip things as they as they as as they go through, the I'd say maybe maybe one in 10 blog posts now that I'm looking at the need some kind of intervention or just a subtle change, or I look at that. At one point, I was looking at a blog post and went, oh, I wonder if this is the same across a number of blog posts in terms of particularly titles, titles and subtitles and names of things and some of the SEO and GEO, and and I did find that there was some recurring things. Okay, now how do I give them instructions?
00:23:31.309 --> 00:23:55.720
Do I change the definition of done, for example, so that when things are potentially releasable, part of looking at the copy for this particular Article. Look at now other articles. Now that you've got this one, and answer because now you're kind of coming down to is it's now almost publishable.
00:23:50.890 --> 00:24:25.309
What do those other ones teach you about what you've just written, and do you need to go back and fix some of those some of that other work and just adjust it a little bit so that there's speaks with the same kind of voice, or that thematically all across all of these that they work. So I guess on the kind of that management 3.0 scale of I decide versus that versus the team just decides on the other, I'm probably one step away from going to the Raptors. This type of work, you can just send straight through through the keeper, and I'll look at it afterwards.
00:24:25.969 --> 00:24:26.269
Yeah, yeah.
00:24:27.500 --> 00:24:48.819
So I'm probably I'm almost at that almost at that point. I think I probably need to get them. They need to build more more trust, and be able to demonstrate that they can actually, without me looking at it. But continuously, what what I find ironic is that these kinds of decisions that I make with the Raptors, they're exactly the same conversations I have with human teams.
00:24:50.710 --> 00:25:50.799
Yeah. So which which brings us to an interesting idea. So Scrum, I've never really thought of it as governor. Framework, but it really is a governance framework in this regard, and actually with humans as well. Actually, because it's about building a consistent, repeatable capability in your organization-a system that allows people to deliver the most value in the most effective way-and it that that so it creates trust with the other parts of the organization. It creates trust inside. It it ensures that has appropriate escalation processes. You made me completely and as the product owner of Scrum.org, it was a bit embarrassing, but you made me completely change my viewpoint around where Scrum fits, but you've found Scrum is a very useful mechanism or structure.
00:25:44.740 --> 00:25:56.920
Do I say framework that can really help that level of trust and that level of governance, right, in your own optimization?
00:25:56.950 --> 00:26:08.700
Yeah, yeah, because it was all about transparency. I want to understand what's going on at what point in time? I need enough transparency when things don't quite work out to be able to interrogate it effectively.
00:26:09.269 --> 00:26:24.440
And so I found the art Scrum's artifacts and looking at those incredibly effective to go like, is it are things going the direction that they need to? So if everything came back to govern a governance, basically.
00:26:20.130 --> 00:27:13.140
What are the rules by which I want my agents to act? What do I want them to do? What do I not want them to do? Yes, some of that has to be the lack of their ability to engage with each other socially. Human team is is a psychosocial team, and there are a whole bunch of unspoken things that happen inside a team as a social system, so a lot of lot of Scrum was critical to be able to, with its commitments, with its structure, to go. These this is how I want my agents to act, and I did I need anything more? No, not really. Everything really just came down to Scrum and some additional useful practices, generally accepted practices, to add on to it to provide more explicit, be more explicit around around how that they work and how should they not work, and where should I be involved?
00:27:14.369 --> 00:27:23.180
Yeah, I mean, you talked a lot about the mechanism, the work, the orchestration, the workflow, the guidelines, the structures, the definition of done. It's funny.
00:27:23.299 --> 00:27:37.430
I published a blog on you know what does definition of done look like in this this modern world. I think that is actually in an agentic system. It's actually more important than in a non agent in a human based system. In my
00:27:37.699 --> 00:27:37.999
experience,
00:27:38.900 --> 00:28:28.400
absolutely. And it's interesting that the what definition of done means for an agentic team in that kind of context because it carries organizational knowledge, organizational memory, cross cross context window memory, and so whenever skill files are created, they have to inherit the definition of done. The definition of done is about you know quality is here and practices are here next level down, etc. What is the hierarchy to be able to to manage that so that skill skills don't just execute tasks by themselves? And so yes, the definition of done was paramount as an overarching governance document to provide boundaries for agents at at skill file level execution.
00:28:29.630 --> 00:28:34.759
So tell me a little about cadence. So how long are your sprints?
00:28:36.200 --> 00:28:37.369
Sprints are now one week.
00:28:38.480 --> 00:28:38.539
Yeah,
00:28:38.990 --> 00:28:43.299
I did have I did. We started off with two weeks, just like everybody
00:28:43.329 --> 00:28:44.859
in the world must come.
00:28:45.939 --> 00:29:09.058
And but what I was finding is we started probably around about around about July. I came to the realization July last year. Came to the realization that what I was doing was it was expect inspecting the work at the level of a PBI, a backup item, so it would start. I'd start it off at that level. I'd then move it all. The team would work on it.
00:29:09.148 --> 00:29:35.209
I would watch it like a babysitter, watch all of the the chats. I'd put it on the on the mode where I'd see the most detail as it was was going up, and I'd do other work and I'd keep casual observation, and then it would stop, or it get they'd get stuck, and I would jump in and go, okay, well that doesn't quite look look quite right, and so I guess what I was really doing at that point wasn't really a sprint at the at the two week cycle; it was more like a sprint on the PBI.
00:29:36.230 --> 00:29:36.289
Yeah.
00:29:36.680 --> 00:30:52.839
But what I found after doing as their skill files matured, as the definition of done matured, that I didn't need to spend as much time with the team trying to understand where things were going well and where things were not going well, and then getting to the end of the sprint and going, well, which skill file did stuffed up this time, and asking that the team. So you did this, and Al, what? Why did you do what? Did you do what? Why couldn't I see that these things happening? Oh well, I just decided that I would do this. So it would still in the same kind of conversations with humans that I would have in retrospective. I would would be having in a chat window with my with my team, the Raptors, and go great, and then update all the skill files. And okay, now you can start the next backlog item. But that was that was incredibly manually intensive, keeping an eye on absolutely everything. But like like Scrum does very very well with empiricism, transparency helped to move that that bar from being absolutely involved and checking all the boxes myself to a point where I could see the machine running well and well without me, and then I got frustrated with Claude stopping all the times and the agent stopping all the times, asking me for permission. I had to write into skill files and the definition.
00:30:52.990 --> 00:30:54.970
No, don't talk. You've got to do this yourself.
00:30:55.630 --> 00:30:56.049
You do it,
00:30:56.170 --> 00:31:42.880
and and then eventually got got to the point, probably at about the six month mark, where I went, okay, I'm actually, I don't really need to worry too much anymore. They're not. They know that they they don't need to ask me for permission. They've got the boundaries that they are designed to work within. It's working effectively. Only talk to me. Send me an email if I if you need if something stopped and you need to escalate things because I'm the the human indoor, and at that point I moved from kind of just basically just working PBI to PBI, moved to weekly sprints, and so now they and that also helped to help me take a step back and go, how much time do I, how many tokens because that's that's the new currency, how many tokens do you estimate you need to do this amount of work.
00:31:40.730 --> 00:32:54.880
How much do we need for discovery? I'm going to budget that. Going to set in our week when I want discovery to to be done because they need that kind of kind of structure. When to do you know when when to work on PBIs, which is when I'm sleeping because then I can wake up and I can look at the work and go okay does does this work? And then we get to the end of, if I see anything that's coming out of the log files that they create per PBI, then I can look at and assess. Well, do I should we act now and and implement improvement to a skill file so that it doesn't impact the other the other backlog items? I'm doing that less and less now. It tends to be more that we get to, I'll publish things throughout the sprint, and if I notice a few things, it's only really at that point that I'll then talk to that the team and ask them how certain skill files didn't didn't quite create the copy that I was after. We interrogate it at that if we need to; otherwise, we get to the end of the sprint, and we just we ask the question like a good sprint review, I've seen the work happen. I've already released stuff. What we really need to talk about is now that we're here a week later, and we did discovery, and we've got three months worth of stuff, and we're kind of heading in a certain direction with this strategy.
00:32:55.569 --> 00:33:00.750
Now that we're here, what does that environment tell us about the success of what we're doing?
00:32:57.490 --> 00:33:03.359
That then feeds back into it, then I'll update skill files.
00:33:00.839 --> 00:33:20.630
I'll update the definition. Done gives me time to breathe because as the human gut going back into things, I can go. Okay, now what what are all of the skills involved in this thing? Okay, are they all aligned to the same same version of the definition of done? How do we make sure that we lift that or constrain?
00:33:18.299 --> 00:34:03.809
Sometimes it's simplifying it. I did that with Remy with his decision to not use N8M. When there's all of these references to Hugging Face and Gemini's banana as a fallback, etc. Why don't I rather Why don't I just tell them just use the workflow and record in the logs what it sends back to you? Did is it Hugging Face? Here's the thing: is it was it Gemini that that sent it to you? Just record that, and so stripping back rules made it less complex, and I've not had a problem since. So sometimes it's been yes, simplify it. Sometimes it's add more, but that stepping back now, I've got more time to to actually think about what is the direction that I want these these teams to this team to go in?
00:34:04.799 --> 00:34:47.079
It's funny though because I feel that you're also doing consulting on the team as well as product owning all at the same time. You know, improving the way that it works, thinking about the broader implications, triple loop learning, etc. which I think is really, really interesting. And I do wonder how much of this skill modern organizations are going to need a lot more of than they've historically have, because there's a level of innate intelligence in human beings. I often a mystery to me, particularly when I'm driving.
00:34:47.230 --> 00:34:59.860
But there is apparently innate intelligence. So whereas you are, you're making, you are applying systems thinking and all of your experience as a consultant to optimize these.
00:35:00.210 --> 00:35:10.050
Agents to work using this approach. It is interesting. I think there is going to be a lot more of those skills needed to really take advantage of this type of technology.
00:35:11.340 --> 00:36:52.269
Yeah, absolutely, and that's what my book Evolve actually starts to look at. That the kind of the set and forget operating model with plan, build, run. Like that was designed for the 1990s, where stability as a service, as a corporate service, was key to IT. You'd go and say, "I would like a cookie, and they would give you a cookie, and off you'd go. The that kind of optimize for stability and and the and not change, like that the night that 1990s world doesn't exist, but the governance around complex work still is reflective of that era. Part of the question that agentic AI raises ultimately is the speed of its movement amplifies whatever problems that currently exist, whether you know them or not, and so a system of governance that is able to question its own basis of assumptions. You know, why does the governance exist in the way that it does, and go, does it really halt now that we've got new information? And Scrum is great as a governance framework to actually create transparency, and then provide opportunities to go. Let's look at what's going on. Let's understand it, and now let's whether it's change a governance rule, change an assumption around a governance rule, around moving the the line of of the irreversibility boundary to decentralize it so that decisions can happen faster without assuming that every agent has to be and agent issues have to be escalated to a traditional project management board or an AI ethics group.
00:36:48.849 --> 00:37:37.219
Like the speed with which these typically go, like let's meet once a month. Great. Now let's send papers to it. That's probably two weeks before. Then the committee will publish papers about what needs to happen across the that that's another probably two weeks. So we've got two months worth of agent activity that happens super super super fast. That kind of kind of old school governance doesn't it doesn't kind of adapt fast enough. And so being using things like decentralization, working out where is the right boundary and having governance that can adapt, in fact, even adapt itself. That's really key, and that's what modern product models do. Modern product operating models do. The environment changes, customers' needs change. Customers might say that they need something.
00:37:37.340 --> 00:37:53.619
That's certainly their intention, but we won't really know if that's what they actually need until we put that in the hands of people, and if it doesn't work, then we have to adapt, and our governance has to adapt at the speed with which AI can move. As a result, are we actually making good decisions?
00:37:54.219 --> 00:38:11.429
Is the right person making decisions? Do we need to move those boundaries? Do we? What do we centralize? What do we decentralize? And importantly, who is the person that asks those questions around around the effectiveness of governance to manage in this in the context we're talking about manage AI?
00:38:11.639 --> 00:38:21.409
If it's someone that at a team level who is who is the person that looks after being accountable for the system of work and making sure that it's effective, and that's the scrum
00:38:21.469 --> 00:39:02.940
Well, at an enterprise level, to get enterprise level benefits, changing the governance is the first step, and the second one is who now, at an enterprise level, can best answer that. And it's probably, and so in my opinion, certainly the opinion of the book, is that it's basically some type of senior chief scrum master at a portfolio level, and so the portfolio level and the product level and the execution level that we've got full traceability and full escalation paths, so that decision latency is really really low when an issue of governance comes up. So we can say is the governance effective?
00:39:00.210 --> 00:39:07.679
What does our transparency tell us? Inspect that issue and adapt it, and then that can adapt then at all levels.
00:39:08.579 --> 00:39:36.170
I think that one of the things so interesting about the work you've been doing that is this this transparency on a level that is way beyond anything I've ever seen in a human based organization, the and maybe because of the honesty that agents provide in some ways, the the fact that they do write log files. You try getting a software developer in a scrum team to list what they've done that week. Good luck to you.
00:39:36.829 --> 00:40:29.000
Yeah, the the that level of transparency and it does exactly what you said. It it shows how broken governance frameworks are inside the existing organization. You know, we've been railing for the last 2030, years about the challenges of adopting Scrum, but ultimately, no surprise. The model was broken in the environment that you were bringing us in was already broken, and I think what you did with the Raptors really just made it for me incredibly crystal clear, super interesting. So we've come to the end. We have to. We could talk for as we have actually talked for days about this. Yes, usually we're both of us at horrible times of the day, just so that we could communicate.
00:40:25.909 --> 00:40:47.710
But so, thank you for spending the the time today and sharing with our listeners the really interesting work that that you're doing with the Raptors, and and really it's shining a light on how these systems work and agents and humans together is is a really an interesting problem. So thanks for coming, Matt.
00:40:48.728 --> 00:40:49.748
Thank you.
00:40:48.728 --> 00:40:49.748
Thanks for having me.
00:40:50.710 --> 00:41:29.960
It's been great, and thank you, listeners, for listening. Today we were so lucky to have Matthew from Zen X X Machina, which I always say wrong, and I'm sure the Raptors would get it much better than I would. Joining us and sharing with him his journey over the last well over 12 months now on the use of an agentic product delivery team that he's been working with, and some of the interesting findings he's had along the way. If you liked what you heard, please subscribe, share with friends, and of course, come back listen to some more. I'm lucky enough to have a variety of guests talking about
00:41:30.139 --> 00:41:36.110
professional Scrum product thinking, and of course, Agile.
00:41:32.179 --> 00:41:36.110
Thanks, everyone, and Scrumm on.
