अनुलिपि 🔗
speaker-0: TDD is far more important now with AI because I can get it to go like write me a whole bunch of tests. I can then review those so that like it shows the intent that I'm after, and I can go, okay, that's I'm ready for this. I'll write the code, but do not touch the tests. And suddenly it's like, ⁓ okay. It it like for me I find AI tends to be a lot more creative in how it's thinking about how it needs to do it rather than going 'cause I'm sure everyone listening, you, me, we've all had it where it's like go fix this test. It's like this test can't be fixed, so I've deleted it.
speaker-1: Yes, I'm running an experiment right now on my phone.
speaker-0: Welcome to Browser Stack Talks, the podcast where we turn testing pain points into playbooks. I'm your host, David, and today we're stepping straight into the engine room of modern software testing. We're breaking down old silos between writing code and testing it, moving past superficial metrics to discover what truly makes build quality ⁓ into software from day one. ⁓ joining me today is Gil. ⁓ Gil, please introduce yourself to Our audience.
speaker-1: Hi, ⁓ First, thanks for having me on the podcast. Thank you, David. I'm Gil, Zilberfeld from Testing Gil. I'm a trainer and a consultant on everything that's related to good software and also bad software sometimes. ⁓ And today we're going to talk about things that I deal with really daily from unit testing to API testing and AI. ⁓ talk about that as well probably and that's me.
speaker-0: Awesome. ⁓ in doing my research, I saw that in ⁓ spare time you like to play video games and you like to do a lot of that. I think kind of most people love kind of video games nowadays, especially people our age. but you also like to kind of slay other types of monsters that I think are quite prominent. In the world of development and testing, and that's kind of flaky tests and bad architecture and testing hype. ⁓ so with that, I want to start kind of at a foundational level. ⁓ I've been a big fan of like test-driven development. I know you are a big fan of it too. ⁓ and in your book you You have this great point that kind of testing examples are normally too simplistic or kind of in the real world legacy code actively resists unit testing. How do you normally go about coaching people into kind of test-driven development, adding tests into these kind of areas where like they actively because ⁓ kind of I think it's a a frustration point, they don't understand it enough to be able to write the code. How do you tend to get people thinking that way in an established code base?
speaker-1: Well, the main, are two real problems, okay? The first one is a very small problem, but people see it as a big one. And they say like, how can I write the code? How can I write a test if I don't have a code in there? Because they used to have code that they're going to test. But that's really a small bump in the road. The real big thing is that the system is so convoluted and complex and they understand part of it and they know that the parts that they don't understand, if they changed it a bit, something else will break and they will lose sleep for three days. ⁓ So there's the fear of what will happen and can I actually use it in a system that's big, complex, been there for 10 years and ⁓ that's the main thing. How do I... coach people into this mindset because TDD is very simple really. ⁓ It's very easy to explain that the implementation is hard. And it really is going back to like simple stuff. Like what do you want to do next? What are the dependencies? How will you know that this actually worked? And going through these stages, you actually build a test. the test may be ugly. It may be not exactly what you want to do, but it does convey, I want this to work in the end. And when you do that, and it actually works for our API tests as well, ⁓ or web tests or whatever, you start with the intention, the description of what actually you want to work at the end, and you put it in ⁓ code, actually, and then you start adding the code in order for it to work. Then you do it again and again and again. It's like the implementation becomes very simple. The stumbling block blocks is okay. order. The problem is that example is you can't, can't, you that you probably have a test and code working within a few minutes. Yeah. Like every round. ⁓ Sometimes it won't work on big systems because you have to move things around in order for this to work. So, the expectation is that it does work like in an ideal mode, but in real systems that are not refactored, well, ⁓ people don't understand them well or fear them. It's going to take a while, but once you, you convey the intent in like, This is what I want to work. People get it.
speaker-0: Yeah, in my experience, ⁓ and it sounds like it's in your experience too, is that like people worry about what it's going to look like at the beginning, right? Like no one's going to review this and go, Yes, that's fine. It's like, yeah, but you're not ready for the review at that point. You're kind of you're kind of you're putting a a little knife into it and then kind of squeaking it open slightly and then going, ⁓ okay, I can see inside now, let me see if I can do a bit more but at the same time you don't want to damage it. Learning and doing that, and I think that's where it's important that part of the TDD phrasing is always kind of red-green refactor, and it's that refactor that people like tend I find tend to forget that they need to do right. And it doesn't mean that you factor your code, it's like refactor your test, refactor your thinking, ⁓ right? Because like you've learned something along the way, ⁓ and it's it's it is a cycle because you're like you're constantly learning. why do you find people skip that kind of third step and like how do you get them to think of it more from an architectural point of view that like when you're doing these things that like actually the refactor
Load full transcript
part is in incredibly important.
speaker-1: Yes, it is. And it even becomes a lot more important these days and people throw it away more these days. ⁓ Refactoring is like solving a puzzle. You're studying a video game. You start at point A and you want to be at point B. Why do you want to do this first? So people just don't have an understanding of Why would I want to do this? Like things are working right now. And the answer is yes, but we're going to do more things after this test. We're going to add, we're going to generalize things. It's easier to do this if the code will be ready for that. And the opposite of that, by the way, is if I don't do this, for the next test, I will change the code for the next test and do the refactoring at the same point. So ⁓ I think that's the brilliance of DDD, the refactoring part, is that once you have something working with evidence, you can play with it until you get to what you actually want, and you want for it for not just fun, but for future things, future like two minutes from now, or future like when I look at this code six months from now, I want to understand it. It's okay to play with it because I know I'm not breaking anything.
speaker-0: Yeah.
speaker-1: and people use to do all these things together. And TDD tells them, break them apart. Now today, it's even worse because AI allows you, or code agents, allows you to like press a button or whatever you're pressing, and it will create loads of code and loads of tests maybe. And you can ask it, well, doesn't look like this. Change it for me. And it will change it for you. ⁓ Whether what you wind up to is what you actually wanted, it's something else. ⁓ And it may, and the problem is that it's too much for us humans to review correctly. Yeah. If you, if you went through TDD, like a human, and you at least write some code, some test, you're reviewing at least what you're doing. And we kind of, sort of outsourced both writing and reviewing to AI. So we seem to think like if everything works, okay, I don't need to touch code, the code agent will do this for me, but things will go wrong. And we will need to get into there and understand what's happening and look at the mess that you did. So I think one of the other things that it's not part of TDD, but it's part from what I'm teaching is do this in very, very, very, very, very small, small, small, small steps. Think, break the problem to what you're doing in very small steps. And if you think like, can this be done smaller? Do that. Because apart from making the cycle go faster, which it will probably do. You're still in control. You understand what you're creating. You're viewing that. You confirm that what you've done actually answers the right problem that you're trying to solve. And there's not extra code, which is a funny story about that. That's created and coding agents do that a lot. And TDD tells you not to create the code that you didn't ask for, was not covered by tests. So it's a very small
set of rules and a lot of intent behind it. And people usually read the rules that don't understand the intent and they try it on a very adult looking system and it bumps into reality there.
speaker-0: Yeah. ⁓ we will get into an AI section later, but like one of the things you said there that I thought was ⁓ for me I'm finding really important when writing code ⁓ with AI agents and things like that. Is I'm finding T D is far more important now with AI because I can get it to go like write me s a whole bunch of tests. ⁓ I can then review those so that it like it shows the intent that I'm after and I can go, okay, that's I'm ready for this. Now write the code, but do not touch the tests. Right? And suddenly it's like, ⁓ okay. It it ⁓ for me I find AI tends to be a lot more creative in how it's thinking about how it needs to do it, rather than going, because I'm sure everyone listening, you, me, we've all had it, where it's like, go fix this test. It's like this test can't be fixed. So I've deleted it.
speaker-1: Yeah, it's more like I can't pass this test, therefore I can delete it. ⁓ Yes, I'm running an experiment right now on myself. ⁓ I'm building an app for myself. So it's not like production app. What I'm trying to do is teach my agent to work with TDD the way I would do it. And it's very hard. Like what you mentioned, like extra code.
speaker-0: Mm-hmm.
speaker-1: or even skipping steps like the red step, like the writing of right to failings test. ⁓ It needs a lot of support system and planning up ahead and confirmation and verification. It actually did that. Yeah. Because I can't. It's not that I don't trust it. don't trust it. But the reason I don't trust it is that it's not only creative and making things that you didn't ask for. ⁓ If you give it something in TDD, these are the rules. You need to follow these rules. ⁓ It may not follow them and it may not tell you that it didn't follow them. Yeah. So it also becomes sort of a Detective episode of, where are my tests now? And it, it apologized many times. And sometimes it becomes, if you are interested in enforcement, this is like a tip for people who do vibe coding and think prompts will actually do what they, what you ask them to. So for me, a prompt is a suggestion. If it's not important, like high risk thing. Sure. But imagine like I had like that. It didn't do the, the red step. didn't write the test before going to write the code. For me, this was a, an up breaking thing. Like this is my application. This is, I want to build it. If you can't build it that way, I can't build it that way. Okay. So I need to enforce it. How do I enforce it? I can't prompt harder. I tried. It didn't work. It's still a prompt. So the solution for that was like, example, for the coverage thing, like the extra lines of code, I needed to check the coverage of what it created for and to say, and this is like an external step and it has to be deterministic, right? It has to run every time. It's not up to the
agent to decide I'm not running it Or for the red a red part, ⁓ run the test to make sure it fails. ⁓ Again, I needed to rip it out of the prompt, put it in a script and not allow the prompt to decide not to run it. And for that to happen, I needed another mechanism, like what I chose was already complex, but like a Git file, a Git file that gets committed every time, every... sub step in order for it to say, ⁓ I'm not at the wrong state. I need to rewind. So all things that if you just wanted to want to follow rules for me, but if you want to enforce something that is generated for you by AI and it's important for you, the enforcement itself becomes very complex, costly, sometimes buggy.
speaker-0: Yeah. Which I think this leads nicely into kind of how how should engineering organizations measure that true effectiveness of developer testing then, right? Because you've got all these things all over the place. And you got like ⁓ I hate it personally, but you've got these companies or got ⁓ engineering teams will go, ⁓ we need to hit eighty percent of coverage because that's healthy And you go, Not really, but whatever. ⁓ Or like like should they do that? Or should they kind of think about like mutation testing or regression rates or kind of things like that? How do how kind of when they're going through all of this and they're doing all their refactoring, they're doing their thing, they think it's all good, how do they know that they've got good test maturity in that test suite?
speaker-1: The problem with the answer, it's like it's very complex answer. The reason the organization go to coverage, it's like ⁓ finding something under the lamp. So it's very easy to do. It's very easy to measure. It's very easy to say, you're 78 % covered. We're almost okay. It's not because not all lines are ⁓ as important as any other. So some of the lines are important and this changes over time, right? Things that were important for me a year ago are not anymore because nobody uses this feature anymore. And we're not thinking like that. We're not thinking, okay, what is important? What is important now? And this is like risk-based testing, right? It's like going back to what is important, what should work, what should never happen. And organizations who try to do this are more human-centric. They get a lot of more information from the humans rather than from the machines. AI even makes that worse, right? If you do rely on... If your application does something with AI, like you have AI-based features... What is the quality today? You run it again, you get a different answer. You need to look at trends. AI amplifies everything. We still need to do the same thing. We need to understand what is working, what is not working, what is the risk of this if it doesn't work? So the question is still there. And mutations testing is ⁓ sort of, it's becoming like, It's a very nice tool, but let me ask you. If you have an application that you write or part of writing, how much of the code that your team, not just you, writing is the thing that's running in the application? How much percent do you think you contribute to the application?
speaker-0: ⁓ it it'll be single digits percent.
speaker-1: Probably right. So you can run mutation tests on that Right, but does it tell you exactly or not exactly even in a broad sense of all the things that can go wrong? so it's And in TDT is And test first is almost like that as well, right you think about well, I'm going to DD at least is about making sure or promising myself the confidence that what I write now works as I expect. And it's an assumption, right? What works on my computer now will probably run in production later. But this is only a bit of code that will run from the entire thing of all the servers and services that I didn't write and the dependencies that you took because you don't write everything on scratch. we need to think holistically about... what we're doing and we don't have enough time to do that. So we want to at least the time that we put in and not just write or generate code to make sure it works and spend more time on the things that are important rather than things that are less. So like I said, the answer is not like, okay, you buy a tool and then you get a report and it will tell you. If you want the answers, have to ask questions usually are not like, Yes or no, one to zero, one to 10. So sorry. That's reality.
speaker-0: Yeah, no, I I I like I l I love that answer. I think it's it's really important that kind of ⁓ 'cause I I think a lot of people aim for perfect, ⁓ where kind of good is actually where you need to be because perfect has too much maintenance. And I think kind of that leads to nicely into the next set of questions I wanted to ask you, which is kind of like ⁓ like on your YouTube channel. ⁓ you kind of you've been doing a lot of like ⁓ advanced API testing and how that works. ⁓ highlighting that teams kind of skip tests when kind of things look hard, ⁓ so like authentication, ⁓ expiry or kind of very hard to do things. ⁓ and that ⁓ can kind of lead to blind spots. ⁓ what is the biggest blind spot that kind of when you're doing these things and you kind of you you own you know that you only contribute to the small bit, but kind of the small bit could be the most important bit. ⁓ how do you kind of help them kind of with their testing and then if that API is important to kind of the front end, how do you kind of guide them through being able to do both at the same time?
speaker-1: Well, first of all, have to treat APIs with respect. And I don't think we give them the respect that we should. ⁓ Because first of all, they're very complex. We don't think of them as complex, we're kind of thinking, yes, in a request, you get a response in a status code. That's it, right? That's what an API is supposed to do. But I'll take REST API as the kind of an example, People know that mostly, but this also applies to ⁓ other things. ⁓ It has headers. It
can use tokens. It has context. For example, let's say you have an API that adds a book to a library or store. ⁓ Same API can work differently ⁓ for an admin or for regular user. Okay. In regulator applications, for example, you can delete stuff, right? So you may have an API that deletes a record, but it doesn't really delete it. Only if an admin does this, it may be able to delete the record. But if your regular user deletes the record, it's not marked as undeleted or something like that. So it looks like an actually take ⁓ API testing as like a ⁓ unit testing. ⁓ part of what, ⁓ higher level, does, ⁓ it does something small as compared to the entire application. And you want to know that this is done correctly, but correctly means a lot of things. This is as opposed to the unit test because you know, this is a bit of code basically does what it needs to do. Yes, there are like asterisks, like, ⁓ you run it on different threads. Maybe something else will happen. But it's very compartmentalized. You think about this needs to do this. APIs can do a lot more and are impacted from a lot more. So this is where the respect comes from. And this is also for the blind spot because when somebody, the developers or the product people tell the testers, test these APIs, there's a lot of hidden context that's hidden in the application. what's happening in the implementation of that. ⁓ And this is like when it's coming to, okay, so I'll check that if I add a book, it will return it to 101. Everything's okay. Did it actually add the book? Can I get this book on another API? Can I delete this book? Can you delete my book that I added? ⁓ So there's a lot more.
speaker-0: Yeah.
speaker-1: apart from implementation stuff like that, usually you don't run an API on its own. It's not an isolated case. You run a lot of things in CRUD. It's easier, but it doesn't have to be. ⁓ You can get a book that you edit. You can't get a book that you deleted. It makes sense. Everyone has some sort of experience in that. But again, we will lower this to the, this is how a CRUD system works, but a more complex system puts things in queues. Things don't happen immediately. So we need to know a lot more of how things are built, how they will run. ⁓ If you do something like add a book and you're adding it to the database and the code inside it, okay, is there, says that if you can't connect to the database, you will try, okay? You don't see it outside the API. because they tell you the API will do this for you. But if you knew that there was a retry, then you say, okay, I won't test this. Is this actually working? Okay, so you need to know how the code is built. So the black box thing, I wasn't still against it. People have this thing that it does a thing, but no, it does multiple things in different ways. So we need to ask more questions and then with the time that we have. test that things actually work the way we expect.
speaker-0: I th I think that's really important. kind of ⁓ it's a really good insight and is one that I try explain to people as well, is that like ⁓ systems are complex, right? I worked on browsers for a very long time. Browsers nowadays are the equivalent of an operating system under this under the hood, right? ⁓ and then they'll go, ⁓ my Selenium test, my playwright test, my Cypress test is flaky. You're like, okay, there's a lot of moving parts. ⁓ and the analogy on
speaker-1: beyond the browser. ⁓ yeah. ⁓
speaker-0: ⁓ and the analogy that I've I've kind of come down to is ⁓ you're like you're in a a a a skyscraper and you call for an elevator. Right. Sometimes it will take thirty seconds because it's just on the floor below you. Sometimes it will take three minutes because it's you're on the top and it needs to get all the way from the bottom to the top. And along the way there might be other people pressing buttons and the systems like And but you all you're seeing is the button and waiting, right? Like that's that's it for you. ⁓ and it becomes s when you kind of look at it as a whole living organism, suddenly systems become a lot more interesting to test because kind of you you know, suddenly you can start thinking about how regressions might come into it because like you go, ⁓ there's this thing over here that I never knew about. But until I hit this flaky test, right? And then suddenly I learnt about this one thing because ⁓ and systems they don't care about your feelings, right? Like they'll just like they do what they're supposed to do. And like if it's a cue, they'll wait until the queue is flushed and then they'll do stuff, right? Like they don't care about it. ⁓ and I think kind of it becomes really important in ⁓ you describe it as the discipline of clean testing, right? Like is that you can't really think about these things and you think about every layer along the way and every bit of infrastructure. So from kind of if you can't scale your test locally, you might use a service like Browser Stack. ⁓ but then there's the internet in the way, right? Like they can help you scale and that's but that internet becomes an interesting thing ⁓ in the way. ⁓ how do you kind of When you're doing all of this and you s people are seeing all these problems like flakiness, false positives, false negatives, how do you kind of because you do a lot of coaching, how do you coach them into like, right, let's break this down into something manageable and then figure it out that manageable thing and solve that manageable thing at a time?
speaker-1: personal or organizational. And the reason is not just because systems are complex and none of it is like the same everywhere. But it's the level of knowledge and experience people have with the system and how it's built and the architecture and what works there for a while. And we haven't seen bugs from it. And what surprises us And we get like a barrage of ⁓ customer complaints when we don't even know where to start. So, ⁓ which is a good point to start actually.
But so you have to take into account all these things and the documentation that you have and the automation suite status that you have, what you're testing, what you're not testing and create a whole kind of map of what we know, what we'd like to learn, what surprised us curious, we're curious about, or we don't want to get calls at night ⁓ and say, okay, in order for this to work, what would we like to know? And this is like ⁓ a distinction about tests. Tell us really what we want to know. And if we're treated like that, say, okay, it may be it's not enough to let's say, let's say we don't have any unit at all. Developers like, are very confident in what we do. So we have a full automation suite though. Okay. It works for us. It gives us the answer that we want, but when something goes wrong, we don't know where it is. So it's like a whole debugging session every time. Okay, so what would we like to know? We would like to know what's happening inside. What goes on first? What kind of errors do we see in the logs? What trend of things are we seeing overruns? ask these answers and then say, okay, these are important for us. Let's answer them. How can I answer them? We can answer them with automation tests. can with manual testing, like every once in a while, we can answer them with. ⁓ scripts that run and collect and, ⁓ pass logs. We can put in things that dump things into logs because they're not there. Things are not visible. How will we know that? So again, it's getting, it's getting boring, but it's getting back to the, to the, we want to be control of, to know, ⁓ to lower the risk. And once we specify this and taking into account all the things that I mentioned before, what we know about the system, how it behaves, how it behaves in different environments and separate customers. Then we say, this is what we're focusing on, this is what we'd like to test, this is how we're going to do it. And then we test it, it may not work. Sometimes even the great ideas that don't work immediately.
speaker-0: Yeah. ⁓ and I think one of the things you've you've highlighted is that kind of you want your tests to kind of always guide you and teach you, right? Or ⁓ I think you've described it before as executable documentation. and I think kind of that's ⁓ building that out tends to then give you more insights along the way. the other one, ⁓ and I
speaker-1: you
speaker-0: I had a a podcast a a while back with ⁓ Marie Cruz and one of the things we were talking about that might help people here is the observability, building observability into your system, right? Because then you can like here's this like problem that's happened and you can trace it all the way through. and you can even if you got automated tests, you can add kind of trace markers. So you can go from your test through the system and then suddenly it becomes a lot more interesting. ⁓ and that gives you a bit more insight ⁓ so that you don't have ⁓ as many blind spots in the system that I think a
lot of people tend to have ⁓ or kind of they just rely on just logs or just ⁓ I've got this 500 error, I don't know where it's coming from. Like being able to have a a meaningful system ⁓ around observability, open telemetry, whatever. tends to give people a bit more insight and give them the confidence to be able to recreate those things and then once you can recreate those things you can go as you say, go learn what what did I want to learn at that point and solve that.
speaker-1: I think it's a bit more than that. What you said is completely correct. It's also comes back to we run tests basically to make decisions. Can we release that to be fixed about? And there's a lot of kind of assumptions that we make about the test reports that we see at the end. And today, It becomes a lot more... easier. You must ask more. What do these results actually mean? If AI generated the tests for you, the API work, okay? If it checks just the status code, what does it mean? What, what, what you get, can create a whole lot of tests that pass. Yeah. But can you use these results to make the decision later? So it's not just about learning. It's about ⁓ first spotting a light on the system and tell us this is how we see today. And what we're looking for and try to enhance is what does this mean for us? Can we use this data as is? Can we trust it? Do we need to add more? Do we have blind spots on it? Do we think we have blind spots that we don't see and we need to investigate more? know, when with complex systems, you don't see what you don't know that you don't know. So more questions. And this is not a one time thing. It's not like you do it. You're going to automate the test suite. You run it. Everything will be fine. And we'll tell you the same thing over and over again. Things change all the time. So we kind of need like meta tests that look over trends or times. Testing is becoming a lot more interesting and complex and we still need to do it with the same workforce.
speaker-0: Yeah. ⁓ and I think ⁓ this is a a good segue into kind of people using AI, right? Like they're trying to get the same amount well, they're trying to get more done in the same amount of time. and they've got these ideas around self healing. I know you've been talking about it recently, and kind of ⁓ what it might look for kind of having these coding assistants, ⁓ coding agents, coding bots to help. Let me just start on kind of the one that I think tends to get a peop a few people's backs up ⁓ in the testing space is around this idea of self healing agents. what does self healing agent mean to you? ⁓ because I think that's really important and sets the context.
speaker-1: So I'm running ⁓ a tutorial on agile testing days in November about build your own self-healing agent. And self-healing sounds like magic and some people sell it like magic, but it's really very simple. ⁓ implementation may be a bit complex, but if you think about it, go back to write a test. You have an intention of running the test and things change. Why did they change in the page or something? It's very simplistic, like saying the thing, the page and some of them are, ⁓ change all the time and you want to ignore them. ⁓ so what you have really is a test that you trusted yesterday and now it fails and you ask, well, why is it failing? And it can fail for different reasons. But if you identified the problem as like, the locators or selectors are not in the place where they put them or where the developers put them yesterday, that is not worth my time. I want it to, I want the test still to work. And that's self-healing really self-healing makes the test work again. Sometimes for the wrong reason though. Yeah. Okay. This is like the magic we're talking about. Sometimes it's black magic. So, ⁓ the, the, the tutorial I'm running is like, like removing the curtain and then like, in the getting into bits of understanding, look, okay, how do we identify this magic, this problem? What has changed? Can we classify this as something that we can work around? The work is very simple, right? Find it on the page where it's new location and then rewrite the test. This is the whole magic and it will run. The problem with that is ⁓ if we're dealing it. tool problem or a tool solution. Yeah. We're missing out on all the cases that the tests failed for the right reason. Yeah. Okay. Because it's very easy to heal something that creates a problem because it didn't bring the right information from the backend. And now something's changed on the page and now it will look at another thing and the another thing here there is there. So the test passes again. So that's the real problem with this black magic. You need to understand that you're fixing the right thing, which can be applied to a lot of things in test.
speaker-0: Definitely. like I've seen it ⁓ when I've asked kind of I'm getting it to do something ⁓ on vibe coding, ⁓ as the kids would say. and like I've seen it hallucinate weird and wonderful edge cases. And I was like, I don't think that's ever realistic. or kind of ⁓ I've also seen it introduce biases into a system that should be fairly neutral. And I appreciate humans might introduce their own biases, right? Because like they build out tests from their own knowledge base, right? And that's what AI is doing in the same place. ⁓ how do you kind of help teams spot those types of problems, ⁓ or be at least aware of it? Because if you're aware of it, then you can solve it.
speaker-1: It's a bigger, it's a bigger problem. Okay. Because like I said, if you're a coach, you would say, did you think about this? And they're starting thinking about this and they're going to attack it in some way. Yeah. So I would say, okay, bring a list, right? Bring a list checklist and they will start thinking about it. The problem is that they want, ⁓ the problem is that organizations kind of. treat testers today as workers, let's say, they can and we can and place where testers have to test
with the same numbers, a lot more code, okay, because AI generates a lot of code. And we needed to, at the same time, ⁓ deal with things like we didn't have before, like for example, ⁓ You know that we had SQL injection before and tonight, today we have prompt injection and history poisoning, all kinds of things that AI has brought us. And if yesterday what you had for security, for example, is you probably didn't have a security person or tester in your organization, what you do is like outsource this. There's some kind of test like everything reminds the red team will come and check how you're doing and so on. But this is no longer true. Now you and you need to understand security and you need to understand performance and you need to understand tokens because it costs you money. Yep. ⁓ so we need to understand a lot more and, to, to, to become like a good tester or very good test team. need to understand a lot more than the system that we're testing, which is already like big. And on top of that, the organization needs to understand we are dealing with a lot.
speaker-0: Yeah. I still think
speaker-1: So the organization, yeah, because otherwise it's still a test of problem. they, maybe we'll throw another tester there. It will be okay. No, the problems are a lot bigger than the risks are a lot more than we used to contend with. So maybe the expertise is not usually the expertise is not in the organization and you need to learn and we need to say, okay, how to build. the new things into our methodology tokens. How do we ⁓ organize our CI pipelines so they don't cost us money all the time? Okay. So this is no longer a CFO problem. This is like an engineering problem.
speaker-0: Yeah. And I th I think it's it's really important like because I see like I've seen a lot of companies go, ⁓ we don't need QA, we don't need as many engineers, ⁓ we we can just use tokens. ⁓ and Yeah, I I ⁓ I find the like when CFOs say things like that or companies say it, ⁓ you just like and then you look at all the like AI companies and they're not
speaker-1: Good luck with it.
speaker-0: hiring necessarily data engineers. They're hiring engineers to solve other bits of problems, ⁓ from security, performance, things like that. ⁓ they're still hiring people ⁓ in the age of AI, ⁓ while other companies are going, ⁓ we don't need so many and it's like, well, the problems are just a lot more scary now. Like security problems I think are phenomenally scary in the age of AI. ⁓ and especially 'cause we've democratized everyone can write code now through AI agents. You're like, okay, cool, but like, you know, don't just n pull in dependencies that you've never heard of or things like that. Or, you know, you need to set up trust boundaries or things like that. It becomes fascinating to see. But now we've said, like, hey, marketing team, you can write some code too.
speaker-1: The problem is that for the last two and a half years, I think it has become a lot more apparent in the
last six months, especially with Fable and... Suddenly a government decides if you can run your business, right? Somebody else's government. But it was so much focused on productivity. So ⁓ what everything will become faster with the hidden assumption that everything will work the same. We're now dealing with people starting to understand that it's not the same. It's a lot more, a lot more to review, a lot more risks, and we need to understand a lot more. We need to deal, give you an example. I had a discussion with a friend, recording of the podcast on Sunday. And he said, he's a developer and a tester and well, the harnesses are getting a lot better. And it's true, especially if you're like, vibe coding or doing a lot of, with lot of. operational stuff with prompts. Which is true. Harnesses are getting more cheaper for you, better context. Maybe part of it is a bit reading mind, but they don't understand the whole business problem that you're trying to solve. No matter how
speaker-0: Yeah.
speaker-1: how well you try to explain it to them. Even if you have like a junior person on your team, even you, do you assume that if you explain it, things once everything will go correctly? No. Why would you assume that with robots that don't understand the whole context? And this is what we're trying to do with the reckoning that we're beginning to see, I think that it costs a lot, that it doesn't give us the same results that we expected. So we need to compensate in some way.
speaker-0: Yeah, definitely. Awesome. ⁓ Gil, we've ⁓ reached our kind of quick fire round, ⁓ where I'm going to ask you some very quick questions. ⁓ and the idea is that we try give the a short, succinct answer, ⁓ and we go from there. are you ready? Cool. ⁓ what's your favorite testing or development tool right now?
speaker-1: It's Claude, but I need to make my requests very nice, very specific.
speaker-0: Awesome. ⁓ name one book every software engineer or QA lead should be reading.
speaker-1: it may sound old, but extreme programming explained back and back. It's going to be correct forever.
speaker-0: Yeah. No, it it's a very good book. I highly recommend people read that one too. ⁓ what is the most underhyped QA metric ⁓ teams consisting consistently ignoring
speaker-1: I think it's still external bugs. And it's because how organizations built and they respond to the loudest customers. So ⁓ we inside see what's the loudest customer said, but we don't see all the bugs coming in. It's not followed, it's not tracked and something we were missing. And a lot of information is there.
speaker-0: Mm-hmm. Yeah, that I agree with. ⁓ if you weren't working in kind of the tech space, what would you be doing now?
speaker-1: really like to build stuff, but probably teaching something else. I really like the teaching.
speaker-0: ⁓ I I I I I find that a lot with people who do coaching. It's like a lot of them go, ⁓ you know, it's either farmer, chef or teaching. Like that that tends to be the good ones.
speaker-1: I'm not going to do... maybe cooking, yeah. I like cooking.
speaker-0: Awesome. ⁓ thank you so much for joining me today. I really appreciate your time and your insight that you were willing to share with our audience. ⁓ to our listeners, if they wanted to follow you, see what you're doing, where is the best place to do that?
speaker-1: So ⁓ on my site, testingguild.com, there's also the blog there. And people want to be better developers. I'm speaking about testing right now as not as testers, but becoming more quality architects. And a lot of aspects of that that I'm working and trying are in there in the blog. So go there.
speaker-0: To everyone, ⁓ please make sure that you kind of hit the subscribe or follow button where on whatever platform you are so that you can ⁓ see future episodes that are coming out. Gil, thank you again for joining us and to everyone, see you soon.
speaker-1: Thank you. Bye bye. you