speaker-0: Welcome to AI First Engineering, the podcast where engineering and technology leaders share how AI is transforming software delivery, quality, and the future of engineering. Let's get into it.
speaker-1: Welcome to AI First Engineering Podcast. I'm John Robinson, your host from TestGrid. And today we're going to talk about something that might make some people in the testing community a little uncomfortable. test automation is solved. So now what? so for years the industry's kind of poured a lot of money into automating tests, getting coverage update, running faster. But AI can do a lot of that now. It can generate tests, it can read requirements, it can review code, find things that a human is just never going to think about. But Maybe the question isn't how we automate testing anymore. So what it's, right? Are we automating the right things? Do we need to be thinking about something differently? if you've been anywhere near the testing community in the past decade, you're gonna know and recognize my guest, Richard Bradshaw. Great to have you, man.
speaker-0: Hey, how you doing? Nice to see you, John.
speaker-1: So before we get into it, for like the three people that are listening right now that don't know who you are, give us a quick version. Who are you? What are you doing right now?
speaker-0: Yeah, sure. So my name's Richard Bradshaw. I've been in testing software engineering for twenty well, eighteen years, I think, officially.
speaker-1: It's hard say that out loud, isn't it?
speaker-0: Yeah. But if I count the few years before I was eighteen, I like to say twenty, you know, because I was hacking from like fourteen years old. but yeah, most of my year most of my work's been in like testing, quality engineering, quality assurance space, whatever fancy words people want to use. And throughout that I got into public speaking. I used to run a lot of training courses, created the automation and testing namespace, used to run one of the major communities. And more recently, I'm kind of an an an an engineer now. Like which might come up in this topic.
speaker-1: It's wild how that happened.
speaker-0: so for the last eighteen months I've been writing production level AI agents, which has been really fascinating. and I'm also just about to launch my own company called Evolving Quality. to go back into the education and the consulting space. So I'm very excited about that.
speaker-1: So Yeah, that's awesome, man. Well I I want to talk about that more in a minute because that that is fascinating in and of itself. But here's why I wanted to have you on specifically for this topic. And and you and I have been in this space for a long time. We've kind of overlapped in some of our conversations that one of the areas that has been missing is is strategic, right? Like good test strategy, quality strategy. And you've been telling people that automation and testing instead of test automation is a better way to kind of define this. What does it mean if we say automation is solved? My guess is you probably have a very different definition of what was even being solved in the first place. So let's start there. Tell me, tell me what your thoughts are on that.
speaker-0: Yeah, I'll I'll go back to the automation and testing phrasing because for me, it's perfect for what's happening in the world at the moment. So we used to talk about test automation a lot being different or being a completely separate activity to testing. And it went down that way for a long time, right? Treated as two separate disciplines, separate people. Like one of the biggest oxymorons I hear is that we have an automation strategy and we have a testing strategy. Like that that is failing already, right?
speaker-1: Yep, but you missed the you missed the point there.
speaker-0: Yeah, you should have a testing strategy and then within that you should identify where you're going to utilize automation. And that's why I believe it's a solved problem. The idea of what traditional automation, which is a term I found myself using now to distinguish it from some of the more advanced automation that's now possible, that wasn't possible say eighteen months, two years ago. So traditional automation, automated tests, automated checks, we know how to do it. And we know how to do it well. The problem is, like why I say it's a solved problem is I don't I think we know how to do it, but I don't think the industry's accepted that we know how to do it yet. So we still have a lot of people who are trying to automate everything. Right? That problem has not gone away and now they're chucking AI at the same problem. So now they're just automating the wrong things quicker. Right? And consider probably of considerably less quality. So for me, the I The process of going, what should we automate? Why do we automate is a solved problem. And it splits in two. One is I want a tool to help me do this task quicker. This is a testing task. This is a human task, let's say exploratory testing. But I can build tools and I can use scripts, just like everyone is now using AI, the same mindset I've had for the last 12, 15 years around the terminal, command line scripts. building little Python scripts to do stuff for me. That mentality's always been there. I have a task to do. How can I do it quicker? So that's one side of automation that I still think is massively underutilized. And then we have this other side, which is this piece of functionality is so important that if its behavior ever changes, I need to know. That's the question we should be asking when we automate anything. And we know that. Because we people have tried to automate every single thing. They end up in test at hell, flaky tests. Like again, flaky test is another topic we can hit on. There's no such thing as a flaky test. It's just a bad test, right? so I think we know how to do it. And I think there's still a big education problem in that people don't think about the strategy as you said to this question. They don't think about what's worth automating, where the value lies, and what piece of information is actually so important that you should invest in automation, because it is an investment. It's not a one off activity. It's an ongoing investment to maintain that test. Or I don't even like to use the term test. Maintain that pieceable that feedback.
speaker-1: I mean it is it's code, right? Like it is it's a it's a it's a development project in and of itself to some degree.
speaker-0: Absolutely. And I think all the mistakes we've gone through, it they're all well documented. They're there's resources out there everywhere, which is why I feel it's a solved problem. To do automation well, we do know how to do it. But I think companies still think it's a shortcut and they're not investing enough time, as you said, in that strategy and how to supplement their existing testing with automation. The common approach is still the Big Bang. I've got I've got ten Johns doing manual testing. I want to automate them all. And it's like, no, you should. Make John more efficient. How can I take some of John's work away from him and slowly and slowly build it up so that John starts doing more valuable things and the scripts are doing some of that lower checking work that I like to call it?
speaker-1: A hundred percent. And I think you just kinda hit on one of the topics to me that I think goes often undiscussed because of the lack of a a a test strategy or a quality strategy or whatever term you you want to use. But there there's the focus to get to automation. But there's not a great understanding of why you're doing that automation. And in a lot of cases you're automating for automation's sake, right? Absolutely. It it It is just another tool, right? Just like you should be doing it. But I think having and understanding what those other things you should be doing are that you're not doing today, there's so many of people that especially in the QA space that don't realize there are a lot of other things that they should be doing. Right. And and if they're so tied into, I've got to automate this, I've got to hit this number for my automation and you know, how do I get there? That's missing the forest for the trees. Or better yet, you're you're focused on the leaf that's on the twig that's on the branch that's in the tree. Right? Like you're like really missing what's important. So
speaker-0: Yeah. Yeah, I feel a lot of that has gone back to like almost like we're seeing the same with conversations around AI, going back to first principles. So a lot of testing got in a certain way and then automation made it look like you could just automate it away. Whereas now, as you said, we we need to talk about why we test, which is becoming more apparent now because people think AI can just do it. So now we're we're being forced, which I think is one of the things I love about this AI movement at the moment. We're being forced to think about those early principles and the fundamentals of why we do stuff. Because people have a distrust of all of AI, but why not with automation, right? So it's the same thing. Like we've gone down that route now, and it's for me, it's why do we test in the first place? What in what information are we trying to gather? And that's what it's always been for me. It's it's information, it's knowledge work. How can we gather enough information to help our stakeholders make a decision? That is ultimately the bottom line of testing.
speaker-1: Absolutely. And that I think is a a a facet that often goes overlooked, which is w what is the ultimate reason we're testing to begin with?
speaker-0: Absolutely.
speaker-1: Right? It's not so that you have a job. It's so that the business can make the right decision at the right time and ha maintain their level of quality across the board.
speaker-0: Yeah, without a doubt. And I've always found like I love Jerry Weinberg, but the one of his famous quotes was, things are the way they are because they got that way. And that's another thing I saw with automation. They started to utilize automation in the way things were already being done, which was already wrong. So it just compounded and compounded.
speaker-1: This is one of my my like absolute pet peeves is that exact topic because the number of places that I've gone into either as an employee, full time or a consultant, and looked at what they're doing and go, So why are you doing this? Well, because that's how we've done it in the past and that's worked for us. Okay, but but but why? Like what what is your your thought processor? That's just what the guy before me did. So I've been told to keep it running. Okay, so we're gonna throw all that away because that's not telling us anything at this point. Right? Let's just start over from a good, fresh premise that actually tells us and we're gonna automate the things that actually do make sense, not just because. And it's you know, I don't know your thought on this, but I'd be curious to hear like one of the challenges, especially with this AI. movement to everybody using AI to build good tests and build strategies, et cetera, that assumes that AI is using good context and good representation of what good is. And I think we've I don't think we've ever really cracked that nut strategically in a way that has gotten wholesale adoption. So to say that we're using best practices, who's best practice?
speaker-0: Yeah. And it's one of the things I've found, like so for me there's no such thing as best practice, right? There's good practice in a given context. Yeah. So we can have a bunch of heuristics, methods, techniques, but you should go to that new place, analyze the context, and come up with the right approach.
speaker-1: No no two teams, no two companies are the same.
speaker-0: Absolutely not. No, no, definitely. And I think with tests again, Alan Richardson gave a fantastic talk like ten years ago now about the whole process of tests and how we went from a test to automating it and how people believe that it was the same thing. So, John, you run a test case for me and then I automate that test case. They believed we now had the same output, right? But when you ran that test case and you read the words, at that moment you mapped it all to the current context you were in. You remembered that planning meeting last week where you know, Dave was saying, I I can do that, that's dead easy and then Jessica was there going, No, no, you've got to be careful about that 'cause remember that bug we had last time? As you were running that test, you thought about that and you did a couple of extra tests while you were there, right? You did a you had you had a little look around, a little play.
speaker-1: Or maybe maybe something in the story changed that only got mentioned verbally and you had to account for it.
speaker-0: Yeah. And as you're doing that though, one of the biggest problems we've had as an industry, as you were doing that, you all as you were your output was though is you mark that test as passed. You and that you never told anyone you did more tests. And that was one of the biggest problems we had. People would pass test cases, but they didn't just do that test case. They did a hundred extra tests. They put it all into context, but no one ever spoke about that. We never spoke about that testing story. And now that shifted into automating it and that all got lost. So we gained rapid feedback and we gained Yeah. We gained the yeah, e quicker executions and repeatability, but we lost that serendipity and more importantly, now we're seeing that context that you're able to put in because you've been in the meetings, you've seen the code, you've read the PR, and now everyone's trying to give all that to AI to build up this context, but yet people aren't aware of how they're using it in the first place. So they're not aware they're doing it. And so if they're not aware, how can they give it to AI and how can they tell it what to do with it? And that's one of the f lovely things I found about building agents recently is asking people, what do you do? I I don't know. Why is that why is that a good email? 'cause 'cause it is why is that a bad email? Like people are really struggling to actually verbalise what good looks like and what bad looks like.
speaker-1: I I had this exercise, my last corporate gig I did, I was director of POA for HomeAdvisor, and we had three leads in our they were each responsible for doing releases on a regular schedule. And they all swore up and down. They followed the exact same process that was documented in our our wiki. And I said, Okay, then I want you to do an experiment. The next time each of you do a release, I want you to print that out and I want you to write Every single no matter how big or small it is, I want you to write every single thing that you do as you're going through. And when we're done, we're going to compare them all. And not one single person had a completely clean sheet. Some were doing things that the others weren't doing. This guy was doing more than these two were doing. This person was not doing some of the things that were on the list because they just knew them. So they just skipped them. And it was fascinating for them to realize, because they didn't know they were doing it. If you had asked them they would have sworn up and down that they were doing the same exact thing.
speaker-0: And that's one of the biggest challenges. So, you know, in the question of like how do we see like more, you know, potentially people using it to generate more tests, again, it comes a quick question of what's a good test look like for you in your context. And that fundamentally will always most likely be a human activity. It may be that a solution owner might start doing more of that. You know, we've had a lot of acceptance criteria movements, we've had bd, we've had people like Yeah, we've had that kind of trying to catch at the start, but then a lot of testers then became lazy and just just automated the acceptance criteria.
speaker-1: Or or the development teams leaned on them to write the acceptance criteria that they weren't qualified to write.
speaker-0: Exactly. So we've that problem has still not shifted, right? So right now AI is not going to help you solve that problem, neither will the automation tools. However, we well, sorry, it won't solve it for you, but now we can automate things that we couldn't before. So some of those heuristics can now be put into agents that might help you come up with acceptance criteria. I wouldn't trust it to give it it and say, you you off you go, go solve that. Don't come back to me. But if I'm trying to think of ideas It's a great tool to now help me that we couldn't have previously. But if I think about just traditional automation, I think we're gonna have a lot of a mess where, you know, tools are just going to start automating things because they can. And it again, it's not a new problem. It's what S Dets started doing. Because S Dets were like, Well I
speaker-1: Yeah, automating for the sake of automating.
speaker-0: Or we started measuring the wrong things. So we started measuring coverage. And we wanted more tests and more tests. So Estet started writing more tests and more tests. And then what does that do? Execution time goes up, maintenance goes up, flakiness or brittleness goes up, false positives goes up. So now all that time, we could have solved that by just going back to the basics of what is a good test, what do we care about, what is important to know, and what's the quickest way to get that. Is it to automate it? is it is it to improve our testability? Is it to give John a suite of super awesome tools that are gonna help him be ten times quicker when he tests?
speaker-1: Yes, but
speaker-0: That's that's back to the first problem. Yeah. It's you know, it's not a case of we need more tests. Often you don't, you need very few, but they've got to be the right ones in the right context, and that means having that human around who's aware of everything that's going on. And which why mate Vernon always talks about glue work. The best testers are the ones that are basically mapping everything to each other and they're aware of everything that's happening, which allows them to design tests really quickly because they've got all the context.
speaker-1: So, you know, we've we've kind of touched on this, but the first thing that orgs are doing right now with with AI, especially as it relates to tests and is generating more tests. we can generate tests really quick, right? Let's just point it at it and let's just generate quick. Is that actually improving quality or or is it just producing more tests?
speaker-0: I I think it depends how you what you do with those tests afterwards. So at the moment I'm giving a lot of thought to historically we've we've we've either had what some people refer to as kind of checking or script based testing, and then we've had this exploratory nature of testing. at the moment, I think we're in this space now where we've always said it's only humans that can ex do exploratory testing. because that is rapid information, rapid processing, using a result from one test to do the next. Whereas the scripted side, which is where a lot of automation lived, but as we just mentioned, secretly people were doing a lot of exploratory testing as they were just
speaker-1: They just didn't realize.
speaker-0: didn't call it that. And it's exactly the same with automating, which is another problem we'll see. A lot of people did a lot of testing as they were trying to automate something. Let's have a look at the DOM. Let me have a look at the console. Let me look at the API calls. All that is now disappearing because AI is writing it. So yes, we might end up with a test quicker, but we're losing some of that value that was built by generated by proxy because we're not following the same steps anymore. But I think the final what I was trying to get to was you kind of have this human scripted test. We have these automated checks that have been written by a human. Right. So I think we've got to start distinguishing the difference. That automated check written by me, you will have to assume that that's of value because I chose to automate it. Then we've got automated tests that an agent or an AI has decided is a good test. And it's going to put that into our code base for us. Someone needs to review that ideally. But we've then also got a lot of the tools now that will go off and explore your app for you. And they're going to produce a report. Now who's gonna review that report? So in a way that's gonna done a lot of tests, but it doesn't know what's good or bad. So we've almost got an extra possibility now to run more tests, but the information still needs digesting by somebody.
speaker-1: It just yeah.
speaker-0: That's the same skill set I've been mentioned about how you decide what's good information, what's not. So that step of what's good requires that context, requires that awareness of the testability, the product, everything else. It's a lot of skill that no one talks about and no one emphasizes. And we think we can just chuck AI, but I do think we are gonna have a new category of tests. And I think that's gonna cause a lot of confusion because like I said, you've got human tests, exploratory test, a human written automated check. an AI written automated check and then this AI agent just exploring. and I don't think there's too much wrong with either of these. It's just how you use them is
speaker-1: Yeah, it's just another tool.
speaker-0: the challenge. It's just another tool.
speaker-1: So let me shift gears a little bit. Is we've talked about, you know, AI and what it can do, but you know, it can read requirements, tickets, PRs, bugs, logs, reviews, all the things, right? Is the real opportunity for AI and testing actually not in the testing itself, but maybe outside of the test suite itself?
speaker-0: if you find some old podcasts of mine, right, like if I think back to eight years ago, I've always dreamed around having this kind of Jarvis for testing. Right. I've imagined being in this VR world where I put on my testing goggles and I have dashboards and logs and metrics and a voice assistant that I can talk to. That for me still feels like the holy grail of what we're trying to do with tools. Go and run that check for me. Go and run check twelve. Tel tell me if there's been any errors in the logs for the last five minutes. Go and get that latest document and summarise it for me. So I think right now, I feel that yes, it can go and do all that, and yes, it could theoretically go and automate tests. But ultimately, that there's a human responsible at the end. And that's what I've seen a lot of conversations on LinkedIn at the moment. We can have AI do all that and we can spend lots of money and time tweaking it to make it really good, right? Just like we have done with people. But if you take some of the people out of the loop, who has that responsibility when it doesn't work? And I don't like playing the blame game, but companies do. Who's getting the blame? What if some w what if someone dies? What if there's a financial disaster off the back of your software? I can promise you it's not the AI that's getting the blame, right? So it's either going to be the engineering manager, the tech lead, the CTO, whoever it's going to be, there still has to be someone. So I still fe Feel the same as I did with automation. It's about putting it in front of a human or a team of humans and being, what can we automate? Because it's still automation. It's still software. I said the same thing. Automation's just software. An LLM, an agent's just software. How can we utilize it to make ourselves more efficient? So, for example, I hate reading, right? I'm not a big reader. long documents bore me. So if I could get it to summarize a document for me in the way that I would summarize it. So, you know, I'm I remember with one of my former colleagues, Mark, we whenever we used to read a document when there was a problem, he would just flick for all the code examples, right? Whereas I would read the words. So we realized that he would just scan for code and read code and I would read the actual text that got written. But that I could now train an agent to do that for me. Well not trained, I could, you know, give it the right instructions that's gonna help me do that. And I think that's what we need to be looking at now. It's yes, give it the context you can, but it's not gonna do very good with all of this. Too much. If you think of what it takes to be a good engineer and everything you've got to digest and make sense of, and then at the same time you've got to remember that conversation you had with Jessica whilst making a cup of tea, right? And that's That comment from Dave four weeks ago when he was like, I hate that part of the system. It's so boring. Yeah. Is he gonna do a good job on that ticket? Are you gonna test more because you know Dave wrote that ticket? Of course you are. Is the AI gonna know that? Absolutely not. Right? Is the AI gonna know that there was you spent a week debugging that part of the system a year ago because it was notoriously flaky? We're not gonna have the models yet, I don't think, and the semantic structures To put all that because we don't know it ourselves.
speaker-1: And yeah, and I and we haven't cracked that nut of how do we how do we grab that human context still? Like the the things that that that doesn't exist in a data store somewhere or on a hard drive or in a transcript. Like there are things that literally only maybe sit in somebody's head where the hallway conversations. So
speaker-0: So to answer your question explicitly though, John, I think it definitely sits outside the test suite. This is not an automated test or a suite of tests, is still not the solution to testing quality anyway. Automation was never the solution. The the benefits of it were to free people up, get rapid information, allow it to be digestible. So I think it really sits outside of that of what how do we gather the information quickly, how do we make decisions quickly? How can we use AI to do that more efficiently?
speaker-1: So so let me ask you a question then. We we keep talking about automating our tests. Should we maybe instead be automating quality?
speaker-0: So again, I think this is where like, you know, not to be the one who brings up definitions, but like quality is in my when I talk about it at clients is value to some person. Okay, so who is that person? Or who are those people? And then we fall back to what do they care about? What do they like to know? That should then feed into your testing strategy, which should in turn feed into how you utilize tools. So on that same basis. I I think there's there's a hundred percent a lot more opportunity to automate stuff in the quality space that we've never had before. The amount of data that you can now crunch and pro get AI to process and spit out for you that would have all would ultimately have taken you hours to build those reports and to share that narrative can now be done with an AI. And I think having it do stuff like one of the best tools I built, which I talk about in AI, automation and testing. was I built a script that would go and pull down the latest reviews from the app store and look for ten twenty keywords and it would put them in Slack if it ever found one. Error, buggy, couldn't get my voucher, whatever it was, and they would get fired into Slack. So I could see them and go, that might be a test idea. I could now have AI do that for me. And I could have it go off, actually try and replicate the issue, find out what it thinks the problem is, and give me all that information in one big chunk. Which means I can digest it all.
speaker-1: Feedback loop is a lot more
speaker-0: That's it. And and I think we're seeing that with the way other people are using AI. How can we gather it quicker and digest it quicker?
speaker-1: I really like the answer. just like quality is so dependent on the individual or whoever's asking for the thing, right? And one of the things that I would like to see more of is organizations doing a better job of defining what quality means to them. Not at the individual level, but organization and I think this is one of the things that Apple's really good at. they've got their their fanboys and they've got their haters, certainly, but You do know an Apple product when you see it. You do know what the expectations are when you see it. Right. And that's I wish more companies did that because that would take a lot of the confusion out of this conversation, I think, because you you now have a goal that you can move towards that everybody knows and makes sense.
speaker-0: Without a doubt. And we're but going full circle on so many topics because Apple is a great example. It's a lot of people, they're high quality and you know, people have that. But also if you look at quality engineering, right, as a discipline, a lot of the time that was just rebranded tested. But nothing really changed. But quality engineering is whole team wide. Like a f a a true quality engineer should have the power or the ability to influence anything along the way. Yes. And they should be able to put quality checks and gates in at any point of that process. Not just this testing box. The way devs write code, the way stories are written, the way products, metrics. And it's never really been that for a lot of people. It's just been testing and automation wrapped up into a shiny new term to try and make it look more important, but it's it's not. And we're now we're seeing that highlight and the same so for me, you mentioned it then. QE is all about people. Quality engineering for me is purely people related. How can I educate people, train people, but then put things in place to help people do their jobs better? Catches
speaker-1: Yeah. A QE may or may not do testing. Right. Depending on the week, they may not they they may not. They're the voice equality. They're it's a totally different their their expectations are different, or at least they should be.
speaker-0: And we're seeing a shift where I'm seeing this shift now that a lot of engineering managers are taking that task on, which is great. But then we also go back full circle to QA. We used to get so much noise in our community about the term QA. But now QA is becoming really important. But QA to me is about process. What guardrails you know, that's a new fancy term people like, but what guardrails, what quality gates can we put in place to make sure the process is correct? And then you have testing. How can I Put the system, actually perform something with the system and see what it does. Those three things are very different, but they formulate the way that we work and the way that we gather this information and the way that it gets digested and used. And we're starting to see that clearer than ever. and AI can help with all those different bits. And that's, you know, I mentioned the new company. That's exactly what I'm trying to do in that space. Because I I think we know how to do it all. We've just not formulated it well enough yet.
speaker-1: Yeah, and and I think the lack of historical standards or best practices that we all agreed on because for for in some cases valid reasons, right? They just we didn't need to do that or or it worked this is what worked for our organization, we don't need to do it. But I think to some degree that's coming back to bite us a little bit now when all of that information is being used as training data for these systems and the bad practices are being used just as much as the good ones.
speaker-0: And if you're not aware, like we said earlier, don't notice. Do you want compound?
speaker-1: Yep. You you just you AI said to do it this way, so that's what I'm doing. And you wake up two years from now and your app is a mess. So let me ask you a hard question. we're getting low on time, but I I do want to, you know, ask something that that, you know, maybe a little bit controversial, maybe not, but you know, you've built your career on this automation and testing concept, right? Does that framing survive this age of AI or do you think it's gets replaced? What what is how does that work?
speaker-0: without sounding really arrogant, I think it was ahead of its it was ahead of its time. I think it stands I think that's why I've made the transition so comfortably because I've never viewed automation as this like go and just do this task for me and you're that's it, I'm never involved with it again. I've always gone the results of that automation are really useful to me. What why am I doing it? What am I doing it for? Why do those results matter? and I've written loads of things around, you know, the illusion of green, investigating failed tests. So I think now I've done that because I've been the one who says, I've tested this. I may not have written any tests, John. I may have not done anything. But if I look at that green radiator, let's say this green wall, if I look at this green wall as my build radio, and I go, nice.
speaker-1: Good with that.
speaker-0: That nod is me putting those results into context. And knowing that my tests are good, knowing what's changed, knowing what good looks like, and knowing that those automated checks was enough. If I look at it and it goes green and I'm like, hell no way. Like there is no way we've changed like with this fifty tickets in that build, there is no way not a single test failed. Yeah. Then I go and investigate. AI's not gonna do that. It's gonna go, it's green.
speaker-1: it's great. We pass. So good job.
speaker-0: So I think I think right now it's gonna force people to think about how we use tools. And like I said, the idea of automation in testing was I wanted to call it software in testing, but it made you know, it didn't it wouldn't have had the impact that we needed. But that's like we said, it's how do we utilise tools, software and you know, as someone yourself who's, you know, been around the tool space, there's so many tools and it's the same problem. How do you pick the right tool? How do you choose when to use an L L How do you choose when you should automate something? How do you choose to take the results that come off that? That is the real space at the moment. And that's what I was trying to teach in automation and testing. Knowing those value of that software, but knowing that you're the one doing the work.
speaker-1: I couldn't agree more. I I mean, I think that's where one of the biggest things that I'm personally worried about being lost as we move forward is is as the let's call it the industry as a whole is changing and you see a lot of the the senior people, the people that do have some of that knowledge moving out, some of the function moving from let's call them QA people, S Dets, whatever you wanna term you want to use this week, moving to these non tech Roles, there is a level of just good practices that you aren't going to know and understand. And somebody has to do that. There's always at least one person in an organization that is, whether they are officially it or not, are raising their hand and saying this isn't good enough. And if you don't have that person, you probably got a different problem entirely.
speaker-0: I don't know how much time we have, but I'm gonna start and you can choose to edit out if you don't
speaker-1: You know, we're let this this is the fun part.
speaker-0: But I think this is where it's a fascinating opportunity for a lot of testers and quality minded people right now. So I'm one of them. I've been trying to break down the walls between developers and testers for a long time, right? I've put myself in both camps and I've learned from everyone, but the tasks still kind of remain different, right? T D, this isn't a testing thing. This is about writing nice code. This isn't about quality. I'm I'm the tester, like devs can't test their own code, right? All this kind of conversations that we've had. but right now we've got this blending of roles. So if I use AI efficiently, I believe I can do more in a day. Right? I can now take on more than I have before. If I codify a lot of my knowledge into an agent or skills, they will. Allow me to take shortcuts, but still hold me accountable, which again will allow me to do more in the day. So if I can do more and other people can do more, this b separation of concerns kind of for me disappears a little bit. So for the last 18 months, I've been an engineer. I've never written I've written like production stuff for like testing communities and my own like little projects and stuff. Sure. But in the in terms of the companies, I've fixed a few bugs, but I've never shipped features, right? I've spent the last 18 months shipping the sole developer on a suite of agenic agents processing nearly 5,000 cases a week, right? And doing it incredibly well. And I've used agents to code most of that. But I know how to critique code. I know how to test. I know what good looks like. I know how to define what good looks like. So right now there's all this noise about, you know, developers using agents. And you know, are they not really checking the output? You know, they're just pushing it through because they don't know how. They've never needed to. I've been the person checking their output for them. Right. But now that you've got these STETs and quality, the shift into becoming an engineer is not that bad. And you are going to do that testing in quality. And if you're now sat alongside ten other developers, they're now your peers. You could argue they were your peers before, but they weren't. Like they they in some of the
speaker-1: Let's be real. We were we were at the bottom of the totem pole. Like
speaker-0: Yeah, but now you're again you're a w you're amongst them, yeah. That opportunity to spread that quality mindset and testing mindset is now necessary because that's what developers are now doing. They're now become a lot of them have become testers because they're testing the output. Yeah, they have to. There's a real opportunity there.
speaker-1: By necessity. Yeah. So so let me ask you a question on that. Without question, I think that's the direction we're moving, and if we're not, you should be. But it it brings to mind like this this kind of becomes a repeating theme, and I I've been thinking about this for a while. It feels like the next, let's call it wave, era, whatever of the testing experience where it used to be automation feels almost like governance is really the the the the next thing, right? Because a lot of what we've to talked about and you've described on here, that sounds like that's what that is.
speaker-0: Absolutely. But then I would break down the fact that governance almost feels like traditional QA again, right? These are this why what are you governing?
speaker-1: Sure. And and that having that definition, but governing the human AI interactions, right? Governing and putting those guardrails in place for these AI agents, right? The things that humans eventually grew out of having to have, let's call it a babysitter for lack of a better term, AI still does, right? Depending on who built the agent, where it came from, what it's doing, what the expectations are. Somebody needs to still be watching that.
speaker-0: Absolutely. And I feel that maybe, you know, as I said, I took a while to think about what I wanted to do when I decided that I might shift. And this concept of evolving quality is kind of where I'm at too. And maybe governance is the new word for the blend of QE, QA and testing. Right. Maybe that's where people are going. Obviously we still have external governance, right? There is standards and things to adhere to. But I think internally it comes back to a lot of It's never been easier to take shortcuts now, right? It's never been easier to do that. But if you look at some of the best engineers that we've had across the years, like that, you know, especially in the public speaking place, like someone like someone like Kelsey Hightower, and you listen to the way he would talk about the way he developed software, we know we have these tools. Like we have linting tools, we have quote quality tools, we have psycho complexity tools. We know what good tickets look like. we know what good automated tests look like. We know that we should look at blogs and production. So we know we know what we should do. And I think now it's a way of how you like you said, how you bring that into that software development lifecycle and make sure that they're being adhered to. So things like, you know, we talk about governance, but like the QE side of it, how do we train people not to take shortcuts, right? I said it's about people. How do we train them? How do we give them the right mindset? People love talking about mindset. Like what I've been told I have a quality mindset. What the hell is that? Even I can't define it, right? But apparently I have it.
speaker-1: I I'm waiting I'm waiting for that definition or score that we can use 'cause I'm waiting on
speaker-0: My people in my recent project talk about an AI mindset, right? Which again for me when they've spoken about that is very much like my automation mindset, you know, that we utilize it where it makes sense and we don't rely on it, we don't give it free reign, you know. A lot of these terms we have. So I think we have that people side, but then as you just said, like pipelines, right? Pipelines have been around for a long time. How many companies are just turning stuff off because it got in the way? I can never get that to pass. Let's lower the standard. Let's add an if statement here. Let's delete that clause there. Right? If that's in your code base now, AI will see that and go, is that what we do? Okay. Ignore.
speaker-1: I I don't know if you heard a couple episodes back we were talking about I was talking with Sunits, and he it basically said that exact scenario happened to him. He gave it a target to hit that night and overnight it hit that target by disabling the tests that got into that number.
speaker-0: So I think you're right in the sense that it is gonna be governance, but I think a lot of it is also just really good engineering practices that we again, we know. So we have we know you mentioned automation at the start. We know what good automation looks like. We know what good testing can look like. We know what good engineering and what clean code and everything looks like. Because we there's so much work has been done in that space. Yeah. And we can't just forget all that. Because that is the safety net for getting the most out of an AI agent. Without it, it will just solve your problem for you because that's what it's designed to do, like you just said. And it will do it. And it will do it in any way it sees fit. Which is one of the concerns I've had with the testing too. Like, sorry, what you reminded me of the well reminded myself, but like when I spoke about an automated check, it's do A, do B, do C, do D, because the goal is E. With an autonomous automated test, the goal is E. And it will get there any way it can.
speaker-1: Doesn't have to follow.
speaker-0: It might be 10 screens, it might be two, it might alter the DOM. To allow it to do it. But that test will still go green. And that's not the same green as the test that I wrote.
speaker-1: Right. Which is the goal is to hit that target. And that that I've always said this for a long time. Like don't automate to a metric or a goal in that sense, right? Define what your capability is supposed to be and what your success criteria is. That's your target. Doesn't matter what your number is.
speaker-0: It doesn't at all. But yeah, governance is going to be a hot topic. But I really want to stress to a lot of people in the listening to this, you know, whether you're an engineer, you're definitely going to have to be a lot more testing quality focused. And if you're a tester or quality engineer, you will know most of this stuff already. You might just not have realized it's important. And your job now will be to weave that in to the process to try and eliminate a lot of these issues. Same as we find bugs in the software, you're going to find bugs in the way people are now working. And they need to be addressed. And that is what true good QE would have always done. And now it might not be called QE. It might be called something else, right? We're gonna keep we're gonna have a whole plethora of job titles coming out.
speaker-1: You get a new title, you get a new title. Yeah.
speaker-0: Exactly. But the underlying activities aren't changing and QEs will have done most of them. They will have been bitten by a lot of the ones they don't know. But now you only need to know about it to get help from an L L to help you implement some of that and go down that route. So I I I I'm I'm very optimistic about the future. None of these jobs are going away. No, sorry, none of these tasks are going away. But I think they're gonna merge and I think they're gonna start becoming under different titles. And I think people need to get ready for that and embrace that. because you're still gonna get to do the work, but it won't look the way it does.
speaker-1: It it kinda sounds like you're you're saying that that QA, Q E, S D, whatever the the flavor this week is, needs to find their voice or or re find their voice and use
speaker-0: Yeah, we've always had to. They've been the shield for so long for bad practices, bad engineering. A lot of bugs have ended up being the fault of people. So now those bugs are just gonna get automated quicker. Yeah. Right? So and and unless the agents get coded with the right context, with the right good practices in that context, right? That's another thing. But that takes skill for someone to learn what good looks like here. That hasn't altered. And we are probably, in my opinion, best positioned to do some of these new roles. And that role might be agreed. You're half QE, QA, and but you're also a developer now. You don't need to know how to write React inside and out. You need to know how to define a good feature.
speaker-1: Isn't this the same conversation we had when Agile came around and everybody was like everybody is on the team and they're all the same level and they just have different functions they're responsible for? Like we had this conversation ten, twelve years ago. so Richard this is
speaker-0: And just to be clear, John, I don't think the specialisms disappear. I agree. I I I I think they stay, but I don't think you'll have separate roles for them. I think coaching will become more popular. I think you'll still need to stay on top of what the latest engineering, quality, testing, observability techniques are, but y the people doing some of that work may be blended. But also, very importantly for anyone listening, there's a there's always a tail. There they will be a tell a tale of traditional ways of working for the next ten, fifteen, twenty years at least.
speaker-1: Great.
speaker-0: so don't panic, don't fluster, don't be scared of any of this.
speaker-1: If anything, there will probably be more of a demand in five to ten years once people realize we still need these functions.
speaker-0: I I believe so. and I think the the change in my opinion is exciting 'cause a lot of it does stem down to quality or slash governance. And that is where we thrive.
speaker-1: Yeah. Man, this has been awesome. I geek out over talking about this kind of stuff. And if we don't stop because the clock says I gotta stop, I would I would keep going. but where can we follow you, find more info about you, learn more about what you're doing, check out the new website. where where where where are you at?
speaker-0: Yeah, so the best place is LinkedIn, friendly tester on LinkedIn still, but on websites, evolvingquality.com and Richard Bradshaw dot com and automationandesting.com should also get relaunched. all those things will hopefully be updated by the time this podcast comes out. but yeah, I'm very excited at the moment getting back into the content leadership space and really wanting to put some of this knowledge into practice.
speaker-1: So I mean you have your own podcast as well, right?
speaker-0: yeah, they have the podcast with my friend Vernon Richards called the Vernon Richards Show. you can find that on YouTube, Spotify, any of these channels. Yeah.
speaker-1: Very cool, man. Well, we will have to do this again. and and and revisit some of these these topics because I think just the speed at which things are changing six months from now this could look entirely different. It it it it's been wild. So I appreciate it. Everybody, thank you for joining us and we will talk to you guys again soon.
speaker-0: If you enjoyed this episode, follow TestGrid for more conversations with engineering and technology leaders building the future of AI-driven software delivery. Thanks for listening to AI First Engineering.
speaker-1: Welcome to AI First Engineering Podcast. I'm John Robinson, your host from TestGrid. And today we're going to talk about something that might make some people in the testing community a little uncomfortable. test automation is solved. So now what? so for years the industry's kind of poured a lot of money into automating tests, getting coverage update, running faster. But AI can do a lot of that now. It can generate tests, it can read requirements, it can review code, find things that a human is just never going to think about. But Maybe the question isn't how we automate testing anymore. So what it's, right? Are we automating the right things? Do we need to be thinking about something differently? if you've been anywhere near the testing community in the past decade, you're gonna know and recognize my guest, Richard Bradshaw. Great to have you, man.
speaker-0: Hey, how you doing? Nice to see you, John.
speaker-1: So before we get into it, for like the three people that are listening right now that don't know who you are, give us a quick version. Who are you? What are you doing right now?
speaker-0: Yeah, sure. So my name's Richard Bradshaw. I've been in testing software engineering for twenty well, eighteen years, I think, officially.
speaker-1: It's hard say that out loud, isn't it?
speaker-0: Yeah. But if I count the few years before I was eighteen, I like to say twenty, you know, because I was hacking from like fourteen years old. but yeah, most of my year most of my work's been in like testing, quality engineering, quality assurance space, whatever fancy words people want to use. And throughout that I got into public speaking. I used to run a lot of training courses, created the automation and testing namespace, used to run one of the major communities. And more recently, I'm kind of an an an an engineer now. Like which might come up in this topic.
speaker-1: It's wild how that happened.
speaker-0: so for the last eighteen months I've been writing production level AI agents, which has been really fascinating. and I'm also just about to launch my own company called Evolving Quality. to go back into the education and the consulting space. So I'm very excited about that.
speaker-1: So Yeah, that's awesome, man. Well I I want to talk about that more in a minute because that that is fascinating in and of itself. But here's why I wanted to have you on specifically for this topic. And and you and I have been in this space for a long time. We've kind of overlapped in some of our conversations that one of the areas that has been missing is is strategic, right? Like good test strategy, quality strategy. And you've been telling people that automation and testing instead of test automation is a better way to kind of define this. What does it mean if we say automation is solved? My guess is you probably have a very different definition of what was even being solved in the first place. So let's start there. Tell me, tell me what your thoughts are on that.
speaker-0: Yeah, I'll I'll go back to the automation and testing phrasing because for me, it's perfect for what's happening in the world at the moment. So we used to talk about test automation a lot being different or being a completely separate activity to testing. And it went down that way for a long time, right? Treated as two separate disciplines, separate people. Like one of the biggest oxymorons I hear is that we have an automation strategy and we have a testing strategy. Like that that is failing already, right?
speaker-1: Yep, but you missed the you missed the point there.
speaker-0: Yeah, you should have a testing strategy and then within that you should identify where you're going to utilize automation. And that's why I believe it's a solved problem. The idea of what traditional automation, which is a term I found myself using now to distinguish it from some of the more advanced automation that's now possible, that wasn't possible say eighteen months, two years ago. So traditional automation, automated tests, automated checks, we know how to do it. And we know how to do it well. The problem is, like why I say it's a solved problem is I don't I think we know how to do it, but I don't think the industry's accepted that we know how to do it yet. So we still have a lot of people who are trying to automate everything. Right? That problem has not gone away and now they're chucking AI at the same problem. So now they're just automating the wrong things quicker. Right? And consider probably of considerably less quality. So for me, the I The process of going, what should we automate? Why do we automate is a solved problem. And it splits in two. One is I want a tool to help me do this task quicker. This is a testing task. This is a human task, let's say exploratory testing. But I can build tools and I can use scripts, just like everyone is now using AI, the same mindset I've had for the last 12, 15 years around the terminal, command line scripts. building little Python scripts to do stuff for me. That mentality's always been there. I have a task to do. How can I do it quicker? So that's one side of automation that I still think is massively underutilized. And then we have this other side, which is this piece of functionality is so important that if its behavior ever changes, I need to know. That's the question we should be asking when we automate anything. And we know that. Because we people have tried to automate every single thing. They end up in test at hell, flaky tests. Like again, flaky test is another topic we can hit on. There's no such thing as a flaky test. It's just a bad test, right? so I think we know how to do it. And I think there's still a big education problem in that people don't think about the strategy as you said to this question. They don't think about what's worth automating, where the value lies, and what piece of information is actually so important that you should invest in automation, because it is an investment. It's not a one off activity. It's an ongoing investment to maintain that test. Or I don't even like to use the term test. Maintain that pieceable that feedback.
speaker-1: I mean it is it's code, right? Like it is it's a it's a it's a development project in and of itself to some degree.
speaker-0: Absolutely. And I think all the mistakes we've gone through, it they're all well documented. They're there's resources out there everywhere, which is why I feel it's a solved problem. To do automation well, we do know how to do it. But I think companies still think it's a shortcut and they're not investing enough time, as you said, in that strategy and how to supplement their existing testing with automation. The common approach is still the Big Bang. I've got I've got ten Johns doing manual testing. I want to automate them all. And it's like, no, you should. Make John more efficient. How can I take some of John's work away from him and slowly and slowly build it up so that John starts doing more valuable things and the scripts are doing some of that lower checking work that I like to call it?
speaker-1: A hundred percent. And I think you just kinda hit on one of the topics to me that I think goes often undiscussed because of the lack of a a a test strategy or a quality strategy or whatever term you you want to use. But there there's the focus to get to automation. But there's not a great understanding of why you're doing that automation. And in a lot of cases you're automating for automation's sake, right? Absolutely. It it It is just another tool, right? Just like you should be doing it. But I think having and understanding what those other things you should be doing are that you're not doing today, there's so many of people that especially in the QA space that don't realize there are a lot of other things that they should be doing. Right. And and if they're so tied into, I've got to automate this, I've got to hit this number for my automation and you know, how do I get there? That's missing the forest for the trees. Or better yet, you're you're focused on the leaf that's on the twig that's on the branch that's in the tree. Right? Like you're like really missing what's important. So
speaker-0: Yeah. Yeah, I feel a lot of that has gone back to like almost like we're seeing the same with conversations around AI, going back to first principles. So a lot of testing got in a certain way and then automation made it look like you could just automate it away. Whereas now, as you said, we we need to talk about why we test, which is becoming more apparent now because people think AI can just do it. So now we're we're being forced, which I think is one of the things I love about this AI movement at the moment. We're being forced to think about those early principles and the fundamentals of why we do stuff. Because people have a distrust of all of AI, but why not with automation, right? So it's the same thing. Like we've gone down that route now, and it's for me, it's why do we test in the first place? What in what information are we trying to gather? And that's what it's always been for me. It's it's information, it's knowledge work. How can we gather enough information to help our stakeholders make a decision? That is ultimately the bottom line of testing.
speaker-1: Absolutely. And that I think is a a a facet that often goes overlooked, which is w what is the ultimate reason we're testing to begin with?
speaker-0: Absolutely.
speaker-1: Right? It's not so that you have a job. It's so that the business can make the right decision at the right time and ha maintain their level of quality across the board.
speaker-0: Yeah, without a doubt. And I've always found like I love Jerry Weinberg, but the one of his famous quotes was, things are the way they are because they got that way. And that's another thing I saw with automation. They started to utilize automation in the way things were already being done, which was already wrong. So it just compounded and compounded.
speaker-1: This is one of my my like absolute pet peeves is that exact topic because the number of places that I've gone into either as an employee, full time or a consultant, and looked at what they're doing and go, So why are you doing this? Well, because that's how we've done it in the past and that's worked for us. Okay, but but but why? Like what what is your your thought processor? That's just what the guy before me did. So I've been told to keep it running. Okay, so we're gonna throw all that away because that's not telling us anything at this point. Right? Let's just start over from a good, fresh premise that actually tells us and we're gonna automate the things that actually do make sense, not just because. And it's you know, I don't know your thought on this, but I'd be curious to hear like one of the challenges, especially with this AI. movement to everybody using AI to build good tests and build strategies, et cetera, that assumes that AI is using good context and good representation of what good is. And I think we've I don't think we've ever really cracked that nut strategically in a way that has gotten wholesale adoption. So to say that we're using best practices, who's best practice?
speaker-0: Yeah. And it's one of the things I've found, like so for me there's no such thing as best practice, right? There's good practice in a given context. Yeah. So we can have a bunch of heuristics, methods, techniques, but you should go to that new place, analyze the context, and come up with the right approach.
speaker-1: No no two teams, no two companies are the same.
speaker-0: Absolutely not. No, no, definitely. And I think with tests again, Alan Richardson gave a fantastic talk like ten years ago now about the whole process of tests and how we went from a test to automating it and how people believe that it was the same thing. So, John, you run a test case for me and then I automate that test case. They believed we now had the same output, right? But when you ran that test case and you read the words, at that moment you mapped it all to the current context you were in. You remembered that planning meeting last week where you know, Dave was saying, I I can do that, that's dead easy and then Jessica was there going, No, no, you've got to be careful about that 'cause remember that bug we had last time? As you were running that test, you thought about that and you did a couple of extra tests while you were there, right? You did a you had you had a little look around, a little play.
speaker-1: Or maybe maybe something in the story changed that only got mentioned verbally and you had to account for it.
speaker-0: Yeah. And as you're doing that though, one of the biggest problems we've had as an industry, as you were doing that, you all as you were your output was though is you mark that test as passed. You and that you never told anyone you did more tests. And that was one of the biggest problems we had. People would pass test cases, but they didn't just do that test case. They did a hundred extra tests. They put it all into context, but no one ever spoke about that. We never spoke about that testing story. And now that shifted into automating it and that all got lost. So we gained rapid feedback and we gained Yeah. We gained the yeah, e quicker executions and repeatability, but we lost that serendipity and more importantly, now we're seeing that context that you're able to put in because you've been in the meetings, you've seen the code, you've read the PR, and now everyone's trying to give all that to AI to build up this context, but yet people aren't aware of how they're using it in the first place. So they're not aware they're doing it. And so if they're not aware, how can they give it to AI and how can they tell it what to do with it? And that's one of the f lovely things I found about building agents recently is asking people, what do you do? I I don't know. Why is that why is that a good email? 'cause 'cause it is why is that a bad email? Like people are really struggling to actually verbalise what good looks like and what bad looks like.
speaker-1: I I had this exercise, my last corporate gig I did, I was director of POA for HomeAdvisor, and we had three leads in our they were each responsible for doing releases on a regular schedule. And they all swore up and down. They followed the exact same process that was documented in our our wiki. And I said, Okay, then I want you to do an experiment. The next time each of you do a release, I want you to print that out and I want you to write Every single no matter how big or small it is, I want you to write every single thing that you do as you're going through. And when we're done, we're going to compare them all. And not one single person had a completely clean sheet. Some were doing things that the others weren't doing. This guy was doing more than these two were doing. This person was not doing some of the things that were on the list because they just knew them. So they just skipped them. And it was fascinating for them to realize, because they didn't know they were doing it. If you had asked them they would have sworn up and down that they were doing the same exact thing.
speaker-0: And that's one of the biggest challenges. So, you know, in the question of like how do we see like more, you know, potentially people using it to generate more tests, again, it comes a quick question of what's a good test look like for you in your context. And that fundamentally will always most likely be a human activity. It may be that a solution owner might start doing more of that. You know, we've had a lot of acceptance criteria movements, we've had bd, we've had people like Yeah, we've had that kind of trying to catch at the start, but then a lot of testers then became lazy and just just automated the acceptance criteria.
speaker-1: Or or the development teams leaned on them to write the acceptance criteria that they weren't qualified to write.
speaker-0: Exactly. So we've that problem has still not shifted, right? So right now AI is not going to help you solve that problem, neither will the automation tools. However, we well, sorry, it won't solve it for you, but now we can automate things that we couldn't before. So some of those heuristics can now be put into agents that might help you come up with acceptance criteria. I wouldn't trust it to give it it and say, you you off you go, go solve that. Don't come back to me. But if I'm trying to think of ideas It's a great tool to now help me that we couldn't have previously. But if I think about just traditional automation, I think we're gonna have a lot of a mess where, you know, tools are just going to start automating things because they can. And it again, it's not a new problem. It's what S Dets started doing. Because S Dets were like, Well I
speaker-1: Yeah, automating for the sake of automating.
speaker-0: Or we started measuring the wrong things. So we started measuring coverage. And we wanted more tests and more tests. So Estet started writing more tests and more tests. And then what does that do? Execution time goes up, maintenance goes up, flakiness or brittleness goes up, false positives goes up. So now all that time, we could have solved that by just going back to the basics of what is a good test, what do we care about, what is important to know, and what's the quickest way to get that. Is it to automate it? is it is it to improve our testability? Is it to give John a suite of super awesome tools that are gonna help him be ten times quicker when he tests?
speaker-1: Yes, but
speaker-0: That's that's back to the first problem. Yeah. It's you know, it's not a case of we need more tests. Often you don't, you need very few, but they've got to be the right ones in the right context, and that means having that human around who's aware of everything that's going on. And which why mate Vernon always talks about glue work. The best testers are the ones that are basically mapping everything to each other and they're aware of everything that's happening, which allows them to design tests really quickly because they've got all the context.
speaker-1: So, you know, we've we've kind of touched on this, but the first thing that orgs are doing right now with with AI, especially as it relates to tests and is generating more tests. we can generate tests really quick, right? Let's just point it at it and let's just generate quick. Is that actually improving quality or or is it just producing more tests?
speaker-0: I I think it depends how you what you do with those tests afterwards. So at the moment I'm giving a lot of thought to historically we've we've we've either had what some people refer to as kind of checking or script based testing, and then we've had this exploratory nature of testing. at the moment, I think we're in this space now where we've always said it's only humans that can ex do exploratory testing. because that is rapid information, rapid processing, using a result from one test to do the next. Whereas the scripted side, which is where a lot of automation lived, but as we just mentioned, secretly people were doing a lot of exploratory testing as they were just
speaker-1: They just didn't realize.
speaker-0: didn't call it that. And it's exactly the same with automating, which is another problem we'll see. A lot of people did a lot of testing as they were trying to automate something. Let's have a look at the DOM. Let me have a look at the console. Let me look at the API calls. All that is now disappearing because AI is writing it. So yes, we might end up with a test quicker, but we're losing some of that value that was built by generated by proxy because we're not following the same steps anymore. But I think the final what I was trying to get to was you kind of have this human scripted test. We have these automated checks that have been written by a human. Right. So I think we've got to start distinguishing the difference. That automated check written by me, you will have to assume that that's of value because I chose to automate it. Then we've got automated tests that an agent or an AI has decided is a good test. And it's going to put that into our code base for us. Someone needs to review that ideally. But we've then also got a lot of the tools now that will go off and explore your app for you. And they're going to produce a report. Now who's gonna review that report? So in a way that's gonna done a lot of tests, but it doesn't know what's good or bad. So we've almost got an extra possibility now to run more tests, but the information still needs digesting by somebody.
speaker-1: It just yeah.
speaker-0: That's the same skill set I've been mentioned about how you decide what's good information, what's not. So that step of what's good requires that context, requires that awareness of the testability, the product, everything else. It's a lot of skill that no one talks about and no one emphasizes. And we think we can just chuck AI, but I do think we are gonna have a new category of tests. And I think that's gonna cause a lot of confusion because like I said, you've got human tests, exploratory test, a human written automated check. an AI written automated check and then this AI agent just exploring. and I don't think there's too much wrong with either of these. It's just how you use them is
speaker-1: Yeah, it's just another tool.
speaker-0: the challenge. It's just another tool.
speaker-1: So let me shift gears a little bit. Is we've talked about, you know, AI and what it can do, but you know, it can read requirements, tickets, PRs, bugs, logs, reviews, all the things, right? Is the real opportunity for AI and testing actually not in the testing itself, but maybe outside of the test suite itself?
speaker-0: if you find some old podcasts of mine, right, like if I think back to eight years ago, I've always dreamed around having this kind of Jarvis for testing. Right. I've imagined being in this VR world where I put on my testing goggles and I have dashboards and logs and metrics and a voice assistant that I can talk to. That for me still feels like the holy grail of what we're trying to do with tools. Go and run that check for me. Go and run check twelve. Tel tell me if there's been any errors in the logs for the last five minutes. Go and get that latest document and summarise it for me. So I think right now, I feel that yes, it can go and do all that, and yes, it could theoretically go and automate tests. But ultimately, that there's a human responsible at the end. And that's what I've seen a lot of conversations on LinkedIn at the moment. We can have AI do all that and we can spend lots of money and time tweaking it to make it really good, right? Just like we have done with people. But if you take some of the people out of the loop, who has that responsibility when it doesn't work? And I don't like playing the blame game, but companies do. Who's getting the blame? What if some w what if someone dies? What if there's a financial disaster off the back of your software? I can promise you it's not the AI that's getting the blame, right? So it's either going to be the engineering manager, the tech lead, the CTO, whoever it's going to be, there still has to be someone. So I still fe Feel the same as I did with automation. It's about putting it in front of a human or a team of humans and being, what can we automate? Because it's still automation. It's still software. I said the same thing. Automation's just software. An LLM, an agent's just software. How can we utilize it to make ourselves more efficient? So, for example, I hate reading, right? I'm not a big reader. long documents bore me. So if I could get it to summarize a document for me in the way that I would summarize it. So, you know, I'm I remember with one of my former colleagues, Mark, we whenever we used to read a document when there was a problem, he would just flick for all the code examples, right? Whereas I would read the words. So we realized that he would just scan for code and read code and I would read the actual text that got written. But that I could now train an agent to do that for me. Well not trained, I could, you know, give it the right instructions that's gonna help me do that. And I think that's what we need to be looking at now. It's yes, give it the context you can, but it's not gonna do very good with all of this. Too much. If you think of what it takes to be a good engineer and everything you've got to digest and make sense of, and then at the same time you've got to remember that conversation you had with Jessica whilst making a cup of tea, right? And that's That comment from Dave four weeks ago when he was like, I hate that part of the system. It's so boring. Yeah. Is he gonna do a good job on that ticket? Are you gonna test more because you know Dave wrote that ticket? Of course you are. Is the AI gonna know that? Absolutely not. Right? Is the AI gonna know that there was you spent a week debugging that part of the system a year ago because it was notoriously flaky? We're not gonna have the models yet, I don't think, and the semantic structures To put all that because we don't know it ourselves.
speaker-1: And yeah, and I and we haven't cracked that nut of how do we how do we grab that human context still? Like the the things that that that doesn't exist in a data store somewhere or on a hard drive or in a transcript. Like there are things that literally only maybe sit in somebody's head where the hallway conversations. So
speaker-0: So to answer your question explicitly though, John, I think it definitely sits outside the test suite. This is not an automated test or a suite of tests, is still not the solution to testing quality anyway. Automation was never the solution. The the benefits of it were to free people up, get rapid information, allow it to be digestible. So I think it really sits outside of that of what how do we gather the information quickly, how do we make decisions quickly? How can we use AI to do that more efficiently?
speaker-1: So so let me ask you a question then. We we keep talking about automating our tests. Should we maybe instead be automating quality?
speaker-0: So again, I think this is where like, you know, not to be the one who brings up definitions, but like quality is in my when I talk about it at clients is value to some person. Okay, so who is that person? Or who are those people? And then we fall back to what do they care about? What do they like to know? That should then feed into your testing strategy, which should in turn feed into how you utilize tools. So on that same basis. I I think there's there's a hundred percent a lot more opportunity to automate stuff in the quality space that we've never had before. The amount of data that you can now crunch and pro get AI to process and spit out for you that would have all would ultimately have taken you hours to build those reports and to share that narrative can now be done with an AI. And I think having it do stuff like one of the best tools I built, which I talk about in AI, automation and testing. was I built a script that would go and pull down the latest reviews from the app store and look for ten twenty keywords and it would put them in Slack if it ever found one. Error, buggy, couldn't get my voucher, whatever it was, and they would get fired into Slack. So I could see them and go, that might be a test idea. I could now have AI do that for me. And I could have it go off, actually try and replicate the issue, find out what it thinks the problem is, and give me all that information in one big chunk. Which means I can digest it all.
speaker-1: Feedback loop is a lot more
speaker-0: That's it. And and I think we're seeing that with the way other people are using AI. How can we gather it quicker and digest it quicker?
speaker-1: I really like the answer. just like quality is so dependent on the individual or whoever's asking for the thing, right? And one of the things that I would like to see more of is organizations doing a better job of defining what quality means to them. Not at the individual level, but organization and I think this is one of the things that Apple's really good at. they've got their their fanboys and they've got their haters, certainly, but You do know an Apple product when you see it. You do know what the expectations are when you see it. Right. And that's I wish more companies did that because that would take a lot of the confusion out of this conversation, I think, because you you now have a goal that you can move towards that everybody knows and makes sense.
speaker-0: Without a doubt. And we're but going full circle on so many topics because Apple is a great example. It's a lot of people, they're high quality and you know, people have that. But also if you look at quality engineering, right, as a discipline, a lot of the time that was just rebranded tested. But nothing really changed. But quality engineering is whole team wide. Like a f a a true quality engineer should have the power or the ability to influence anything along the way. Yes. And they should be able to put quality checks and gates in at any point of that process. Not just this testing box. The way devs write code, the way stories are written, the way products, metrics. And it's never really been that for a lot of people. It's just been testing and automation wrapped up into a shiny new term to try and make it look more important, but it's it's not. And we're now we're seeing that highlight and the same so for me, you mentioned it then. QE is all about people. Quality engineering for me is purely people related. How can I educate people, train people, but then put things in place to help people do their jobs better? Catches
speaker-1: Yeah. A QE may or may not do testing. Right. Depending on the week, they may not they they may not. They're the voice equality. They're it's a totally different their their expectations are different, or at least they should be.
speaker-0: And we're seeing a shift where I'm seeing this shift now that a lot of engineering managers are taking that task on, which is great. But then we also go back full circle to QA. We used to get so much noise in our community about the term QA. But now QA is becoming really important. But QA to me is about process. What guardrails you know, that's a new fancy term people like, but what guardrails, what quality gates can we put in place to make sure the process is correct? And then you have testing. How can I Put the system, actually perform something with the system and see what it does. Those three things are very different, but they formulate the way that we work and the way that we gather this information and the way that it gets digested and used. And we're starting to see that clearer than ever. and AI can help with all those different bits. And that's, you know, I mentioned the new company. That's exactly what I'm trying to do in that space. Because I I think we know how to do it all. We've just not formulated it well enough yet.
speaker-1: Yeah, and and I think the lack of historical standards or best practices that we all agreed on because for for in some cases valid reasons, right? They just we didn't need to do that or or it worked this is what worked for our organization, we don't need to do it. But I think to some degree that's coming back to bite us a little bit now when all of that information is being used as training data for these systems and the bad practices are being used just as much as the good ones.
speaker-0: And if you're not aware, like we said earlier, don't notice. Do you want compound?
speaker-1: Yep. You you just you AI said to do it this way, so that's what I'm doing. And you wake up two years from now and your app is a mess. So let me ask you a hard question. we're getting low on time, but I I do want to, you know, ask something that that, you know, maybe a little bit controversial, maybe not, but you know, you've built your career on this automation and testing concept, right? Does that framing survive this age of AI or do you think it's gets replaced? What what is how does that work?
speaker-0: without sounding really arrogant, I think it was ahead of its it was ahead of its time. I think it stands I think that's why I've made the transition so comfortably because I've never viewed automation as this like go and just do this task for me and you're that's it, I'm never involved with it again. I've always gone the results of that automation are really useful to me. What why am I doing it? What am I doing it for? Why do those results matter? and I've written loads of things around, you know, the illusion of green, investigating failed tests. So I think now I've done that because I've been the one who says, I've tested this. I may not have written any tests, John. I may have not done anything. But if I look at that green radiator, let's say this green wall, if I look at this green wall as my build radio, and I go, nice.
speaker-1: Good with that.
speaker-0: That nod is me putting those results into context. And knowing that my tests are good, knowing what's changed, knowing what good looks like, and knowing that those automated checks was enough. If I look at it and it goes green and I'm like, hell no way. Like there is no way we've changed like with this fifty tickets in that build, there is no way not a single test failed. Yeah. Then I go and investigate. AI's not gonna do that. It's gonna go, it's green.
speaker-1: it's great. We pass. So good job.
speaker-0: So I think I think right now it's gonna force people to think about how we use tools. And like I said, the idea of automation in testing was I wanted to call it software in testing, but it made you know, it didn't it wouldn't have had the impact that we needed. But that's like we said, it's how do we utilise tools, software and you know, as someone yourself who's, you know, been around the tool space, there's so many tools and it's the same problem. How do you pick the right tool? How do you choose when to use an L L How do you choose when you should automate something? How do you choose to take the results that come off that? That is the real space at the moment. And that's what I was trying to teach in automation and testing. Knowing those value of that software, but knowing that you're the one doing the work.
speaker-1: I couldn't agree more. I I mean, I think that's where one of the biggest things that I'm personally worried about being lost as we move forward is is as the let's call it the industry as a whole is changing and you see a lot of the the senior people, the people that do have some of that knowledge moving out, some of the function moving from let's call them QA people, S Dets, whatever you wanna term you want to use this week, moving to these non tech Roles, there is a level of just good practices that you aren't going to know and understand. And somebody has to do that. There's always at least one person in an organization that is, whether they are officially it or not, are raising their hand and saying this isn't good enough. And if you don't have that person, you probably got a different problem entirely.
speaker-0: I don't know how much time we have, but I'm gonna start and you can choose to edit out if you don't
speaker-1: You know, we're let this this is the fun part.
speaker-0: But I think this is where it's a fascinating opportunity for a lot of testers and quality minded people right now. So I'm one of them. I've been trying to break down the walls between developers and testers for a long time, right? I've put myself in both camps and I've learned from everyone, but the tasks still kind of remain different, right? T D, this isn't a testing thing. This is about writing nice code. This isn't about quality. I'm I'm the tester, like devs can't test their own code, right? All this kind of conversations that we've had. but right now we've got this blending of roles. So if I use AI efficiently, I believe I can do more in a day. Right? I can now take on more than I have before. If I codify a lot of my knowledge into an agent or skills, they will. Allow me to take shortcuts, but still hold me accountable, which again will allow me to do more in the day. So if I can do more and other people can do more, this b separation of concerns kind of for me disappears a little bit. So for the last 18 months, I've been an engineer. I've never written I've written like production stuff for like testing communities and my own like little projects and stuff. Sure. But in the in terms of the companies, I've fixed a few bugs, but I've never shipped features, right? I've spent the last 18 months shipping the sole developer on a suite of agenic agents processing nearly 5,000 cases a week, right? And doing it incredibly well. And I've used agents to code most of that. But I know how to critique code. I know how to test. I know what good looks like. I know how to define what good looks like. So right now there's all this noise about, you know, developers using agents. And you know, are they not really checking the output? You know, they're just pushing it through because they don't know how. They've never needed to. I've been the person checking their output for them. Right. But now that you've got these STETs and quality, the shift into becoming an engineer is not that bad. And you are going to do that testing in quality. And if you're now sat alongside ten other developers, they're now your peers. You could argue they were your peers before, but they weren't. Like they they in some of the
speaker-1: Let's be real. We were we were at the bottom of the totem pole. Like
speaker-0: Yeah, but now you're again you're a w you're amongst them, yeah. That opportunity to spread that quality mindset and testing mindset is now necessary because that's what developers are now doing. They're now become a lot of them have become testers because they're testing the output. Yeah, they have to. There's a real opportunity there.
speaker-1: By necessity. Yeah. So so let me ask you a question on that. Without question, I think that's the direction we're moving, and if we're not, you should be. But it it brings to mind like this this kind of becomes a repeating theme, and I I've been thinking about this for a while. It feels like the next, let's call it wave, era, whatever of the testing experience where it used to be automation feels almost like governance is really the the the the next thing, right? Because a lot of what we've to talked about and you've described on here, that sounds like that's what that is.
speaker-0: Absolutely. But then I would break down the fact that governance almost feels like traditional QA again, right? These are this why what are you governing?
speaker-1: Sure. And and that having that definition, but governing the human AI interactions, right? Governing and putting those guardrails in place for these AI agents, right? The things that humans eventually grew out of having to have, let's call it a babysitter for lack of a better term, AI still does, right? Depending on who built the agent, where it came from, what it's doing, what the expectations are. Somebody needs to still be watching that.
speaker-0: Absolutely. And I feel that maybe, you know, as I said, I took a while to think about what I wanted to do when I decided that I might shift. And this concept of evolving quality is kind of where I'm at too. And maybe governance is the new word for the blend of QE, QA and testing. Right. Maybe that's where people are going. Obviously we still have external governance, right? There is standards and things to adhere to. But I think internally it comes back to a lot of It's never been easier to take shortcuts now, right? It's never been easier to do that. But if you look at some of the best engineers that we've had across the years, like that, you know, especially in the public speaking place, like someone like someone like Kelsey Hightower, and you listen to the way he would talk about the way he developed software, we know we have these tools. Like we have linting tools, we have quote quality tools, we have psycho complexity tools. We know what good tickets look like. we know what good automated tests look like. We know that we should look at blogs and production. So we know we know what we should do. And I think now it's a way of how you like you said, how you bring that into that software development lifecycle and make sure that they're being adhered to. So things like, you know, we talk about governance, but like the QE side of it, how do we train people not to take shortcuts, right? I said it's about people. How do we train them? How do we give them the right mindset? People love talking about mindset. Like what I've been told I have a quality mindset. What the hell is that? Even I can't define it, right? But apparently I have it.
speaker-1: I I'm waiting I'm waiting for that definition or score that we can use 'cause I'm waiting on
speaker-0: My people in my recent project talk about an AI mindset, right? Which again for me when they've spoken about that is very much like my automation mindset, you know, that we utilize it where it makes sense and we don't rely on it, we don't give it free reign, you know. A lot of these terms we have. So I think we have that people side, but then as you just said, like pipelines, right? Pipelines have been around for a long time. How many companies are just turning stuff off because it got in the way? I can never get that to pass. Let's lower the standard. Let's add an if statement here. Let's delete that clause there. Right? If that's in your code base now, AI will see that and go, is that what we do? Okay. Ignore.
speaker-1: I I don't know if you heard a couple episodes back we were talking about I was talking with Sunits, and he it basically said that exact scenario happened to him. He gave it a target to hit that night and overnight it hit that target by disabling the tests that got into that number.
speaker-0: So I think you're right in the sense that it is gonna be governance, but I think a lot of it is also just really good engineering practices that we again, we know. So we have we know you mentioned automation at the start. We know what good automation looks like. We know what good testing can look like. We know what good engineering and what clean code and everything looks like. Because we there's so much work has been done in that space. Yeah. And we can't just forget all that. Because that is the safety net for getting the most out of an AI agent. Without it, it will just solve your problem for you because that's what it's designed to do, like you just said. And it will do it. And it will do it in any way it sees fit. Which is one of the concerns I've had with the testing too. Like, sorry, what you reminded me of the well reminded myself, but like when I spoke about an automated check, it's do A, do B, do C, do D, because the goal is E. With an autonomous automated test, the goal is E. And it will get there any way it can.
speaker-1: Doesn't have to follow.
speaker-0: It might be 10 screens, it might be two, it might alter the DOM. To allow it to do it. But that test will still go green. And that's not the same green as the test that I wrote.
speaker-1: Right. Which is the goal is to hit that target. And that that I've always said this for a long time. Like don't automate to a metric or a goal in that sense, right? Define what your capability is supposed to be and what your success criteria is. That's your target. Doesn't matter what your number is.
speaker-0: It doesn't at all. But yeah, governance is going to be a hot topic. But I really want to stress to a lot of people in the listening to this, you know, whether you're an engineer, you're definitely going to have to be a lot more testing quality focused. And if you're a tester or quality engineer, you will know most of this stuff already. You might just not have realized it's important. And your job now will be to weave that in to the process to try and eliminate a lot of these issues. Same as we find bugs in the software, you're going to find bugs in the way people are now working. And they need to be addressed. And that is what true good QE would have always done. And now it might not be called QE. It might be called something else, right? We're gonna keep we're gonna have a whole plethora of job titles coming out.
speaker-1: You get a new title, you get a new title. Yeah.
speaker-0: Exactly. But the underlying activities aren't changing and QEs will have done most of them. They will have been bitten by a lot of the ones they don't know. But now you only need to know about it to get help from an L L to help you implement some of that and go down that route. So I I I I'm I'm very optimistic about the future. None of these jobs are going away. No, sorry, none of these tasks are going away. But I think they're gonna merge and I think they're gonna start becoming under different titles. And I think people need to get ready for that and embrace that. because you're still gonna get to do the work, but it won't look the way it does.
speaker-1: It it kinda sounds like you're you're saying that that QA, Q E, S D, whatever the the flavor this week is, needs to find their voice or or re find their voice and use
speaker-0: Yeah, we've always had to. They've been the shield for so long for bad practices, bad engineering. A lot of bugs have ended up being the fault of people. So now those bugs are just gonna get automated quicker. Yeah. Right? So and and unless the agents get coded with the right context, with the right good practices in that context, right? That's another thing. But that takes skill for someone to learn what good looks like here. That hasn't altered. And we are probably, in my opinion, best positioned to do some of these new roles. And that role might be agreed. You're half QE, QA, and but you're also a developer now. You don't need to know how to write React inside and out. You need to know how to define a good feature.
speaker-1: Isn't this the same conversation we had when Agile came around and everybody was like everybody is on the team and they're all the same level and they just have different functions they're responsible for? Like we had this conversation ten, twelve years ago. so Richard this is
speaker-0: And just to be clear, John, I don't think the specialisms disappear. I agree. I I I I think they stay, but I don't think you'll have separate roles for them. I think coaching will become more popular. I think you'll still need to stay on top of what the latest engineering, quality, testing, observability techniques are, but y the people doing some of that work may be blended. But also, very importantly for anyone listening, there's a there's always a tail. There they will be a tell a tale of traditional ways of working for the next ten, fifteen, twenty years at least.
speaker-1: Great.
speaker-0: so don't panic, don't fluster, don't be scared of any of this.
speaker-1: If anything, there will probably be more of a demand in five to ten years once people realize we still need these functions.
speaker-0: I I believe so. and I think the the change in my opinion is exciting 'cause a lot of it does stem down to quality or slash governance. And that is where we thrive.
speaker-1: Yeah. Man, this has been awesome. I geek out over talking about this kind of stuff. And if we don't stop because the clock says I gotta stop, I would I would keep going. but where can we follow you, find more info about you, learn more about what you're doing, check out the new website. where where where where are you at?
speaker-0: Yeah, so the best place is LinkedIn, friendly tester on LinkedIn still, but on websites, evolvingquality.com and Richard Bradshaw dot com and automationandesting.com should also get relaunched. all those things will hopefully be updated by the time this podcast comes out. but yeah, I'm very excited at the moment getting back into the content leadership space and really wanting to put some of this knowledge into practice.
speaker-1: So I mean you have your own podcast as well, right?
speaker-0: yeah, they have the podcast with my friend Vernon Richards called the Vernon Richards Show. you can find that on YouTube, Spotify, any of these channels. Yeah.
speaker-1: Very cool, man. Well, we will have to do this again. and and and revisit some of these these topics because I think just the speed at which things are changing six months from now this could look entirely different. It it it it's been wild. So I appreciate it. Everybody, thank you for joining us and we will talk to you guys again soon.
speaker-0: If you enjoyed this episode, follow TestGrid for more conversations with engineering and technology leaders building the future of AI-driven software delivery. Thanks for listening to AI First Engineering.
