TRANSKRYPCJA 🔗
speaker-0: Focus your tests on the riskier areas, and then you can start having those quality gates. And then suddenly it's not just a thous like you're running a thousand tests, nine hundred and ninety-eight are passing, you've got those two that are flaking, you've not done anything near them, right? But if you had that one test that was passing, and I'm not saying you should have only one UI test, right? It should be kind of one key test that tests that area. ⁓ and then if you need to add another one, that's fine. But every step is a if you forget something about you, that's your fault.
speaker-1: AI is obviously the the big thing and I think it's really important to learn about what it is, what it can do, opens up the opportunity to to be the the quality advocate in your company, right? At this point everyone tried Chat GPT. They understand that you can ask questions, it feels very natural, right? Maybe you have even tried like cursor, clot code, codex, all these coding tools.
speaker-0: Welcome back to Browser Stack Talks, the podcast that turns testing t pain points into playbooks. I'm your host, David, and today we're diving into the philosophy that's reshaping the the way we think about quality. The idea that testers should develop like engineers and developers should test like they're in the hot seat. ⁓ my guest is Philip, who's kind of done everything, but I will let him introduce himself.
speaker-1: Thank you, David. Thank you very much for for having me on your podcast. I was looking forward to being here. ⁓ yeah, if I if I should like add anything to introduction, I think I think you summarized the the important part, the the testing and development world. I always felt like I'm somewhere in between. I advocated for a long time that we should be talking together much more and not building walls between our departments. ⁓ so that's my life's mission. I've worked as a tester, QA lead, test on automation engineer. I worked as a front end developer for a while. and right now I'm working as a developer relations engineers at a company called Kodo where we try to ⁓ solve the problem of AI code quality and AI code ⁓ governance. ⁓ so we're building a platform for that. I'm sure I'll mention it somewhere, but that's The that's where I'm currently, that's that's who I am.
speaker-0: Awesome. ⁓ and so like one of the things that's ⁓ I've always found quite interesting in the work that you've done is that you've you've pushed quite heavily. ⁓ and I've I've always tried to push the same thing is that ⁓ if you're doing test automation, you're some form of developer, right? And that goes without saying, despite a lot of people going, No, I'm a tester, it's like, no, if you're writing code, any form of code, you're a coder. W how deep that goes, that's a different type of question. ⁓ But one of the things that you've always argued, no matter what who you are, is that you need to understand what the building blocks of any application is, right? ⁓ why
Load full transcript
is that why is that distinction and being able to understand that so important for modern quality engineering?
speaker-1: I I have a theory on this, and I'm sure ⁓ so I have already had our arguments with people about this. ⁓ so some people even might feel that if you step way too much into the developer space, you might lose something from your testers' perspective, which I don't think is the case. I I I would say it's it's quite the opposite. I would say that stepping into developer space like ev makes you even better ⁓ in your testing field. That's what I always felt. And the reason why I give like countless of arguments and examples on that in my ⁓ in my past talks. ⁓ the the very simple argument is that all right imagine you're you're testing a login screen, right? Your developers have done some work on it and you're ⁓ you're trying to test that and it's And it's a robust thing. Like even those two fields, name and password are like pretty robust because you have to look for ⁓ happy paths, the etch cases, the the the typos and all kinds of different cases, log in with Google, etc. And imagine jumping into that with like no knowledge of how that system works, and you find a bug, you cannot log in. So you go ahead to your developer and say, like, hey, it's it's it's broken, you need to fix it. And they go ahead and fix it, not providing you with the information of what was wrong. Yeah. You might have like 50 test cases to just test that login. But you know, if the the the problem was that they accidentally wrote the the redirect URL and made a typo in that, how much of work do you really need to do? And having that extra information, having that that understanding of how the system works. helps you make your testing decisions better. That that's my main argument. And the second argument I tend to give is that we have test ⁓ sorry, not test automation, but like quality insurance testers, quality assessment, ⁓ happening in all kinds of fields, not only de ⁓ not only developers. Like we got this in aviation, we got this in Formula One, my favorite sport, right? And those people are Are like experts, right? The F1 quality engineer is an engineer who understands aerodynamics, understands fuel, understands all kinds of has like all kinds of different expertise. And I feel like we don't always have that requirement for the QA testing area. I know like some people might draw distinctions between those. I I ⁓ I look at the whole QA testing as as one bigger chunk of thing of of course you have some specializations, but I do think like we no don't always have that sort of expertise requirement. Right. If you can u it if you can test it as a user, that's good enough. And I would argue it's not good enough. You need to get a little further.
speaker-0: Definitely. ⁓ and one of the things you said there, and I'm going to kind of stretch your analogy slightly longer, ⁓ is like ⁓ when it comes to Formula One, right, everything is tested to literally an inch of
its life, right? Because if if it fails, someone could die. Literally, right? ⁓ and one of the interesting things that ⁓ early on in my career I got really into I like I started my career as like process reengineering and then kind of fell into testing by accident because I was doing automation in a different way. Yeah. And then ⁓ but one of the things that I learned along the way was this ⁓ kind of it's the Toyota method where you're kind of you're constantly making small improvements and everyone can own those small improvements as you go forward. ⁓ and the idea there is also that you could go, I'm going to Build an engine, but now I'm going to test it, right? And I'm only going to focus on those tests. ⁓ which I think ⁓ led me to kind of thinking, you know, why is everyone, especially because like you came from kind of doing a lot of front end and like Cypress and all of that, is like everyone focuses on these end to end tests, like you said, like the login. And sometimes that's not the thing that needed to be tested in the first place. ⁓ how do you kind of When you speak to people in your past, ⁓ conferences, things like that, how do you get them to go like it's kind of shifting left, but without using the term shift left, because I think that has been overused and I don't think people use it properly anymore. But to kind of think about that, like taking that analogy further of like, ⁓ you'll f you'll you've done this and I now need to understand it before that goes into the the car before we carry on testing.
speaker-1: So the question is like how do I like open up that conversation or how do I show people the way ⁓ how to yeah shift more to to the development side of things? Yeah. ⁓ well my as I said, like my first argument is that it's good for you, right? It's going to like make you better in your own profession. And I think, I hope everyone wants that, right? The the second thing where I would maybe point ⁓ people to where I think it's ⁓ it's a good place to start. For me personally it was API testing. Right? Because once you get an understanding, let's say you have some some basic REST API on your ⁓ on the service you are testing, if you have that documented or even if you don't and you just open the network panel in your ⁓ in your ⁓ console, right? ⁓ then you get a slightly different model of what's going on in your application, like slightly different like mental model of what's going on in your application. And I think like that's ⁓ for me that was one of the big aha moments where I realized like all right there's a now I understand what the difference between front end and back end is and and that there's a way between there there's like a highway between between those two. Right. And ⁓ and I didn't know that before, right? So I think like that's that's a good ⁓ good start. And another like even better, and I think like these days it's even easier to start, is like build your own app, like try to build a simplified version of the thing you are testing. And even if the simplified version means creating a to do list, then I think that's a that's a good start. For me. that opened up so many different horizons as a tester, like I I suddenly understood that there is some sort of state machine, right, on my on my front end application, which not only like gets that API data, but actually stores it inside the memory of the browser. Yeah. And that that that was a whole new thing for me. And then like what is the mechanism of how a button gets disabled? what are the input fields and stuff like that. Like you get to the more granular things, which some people might argue you should start with. I'm not sure if I'm like on board with that argument. I see the point. But also like learning stuff ⁓ not just in case, but just in time. I know if if you heard that ⁓ distinction, but I love that. Yeah. Like when you learn stuff just in time, just in time when you need it and when you need to understand them. I feel like that that feels like a most natural progression. So that was the case for me. I would probably like go out go start from this place, but everyone of course needs to find their their own path. ⁓ so I guess that that could be my number one advice.
speaker-0: Yeah, definitely. I also find and we'll get more into it later, but like kind of the ⁓ the big push into AI at the moment allows people to at least collaborate with AI to build out and build out that knowledge, right? I think that's the real killer feature when it comes to AI is that like it can dissect stuff and ⁓ there are times where I regularly put in a prompt where like I ask her to explain something I still don't understand. And it's like, could you explain that to me like I'm five? Yeah. And then it goes, okay. There's even been times where it's like, could you draw a picture for me? And then it does like cartoons because I'm five. Right. I find like at least for me, that's the one of the things that I like doing. ⁓ but it the key thing here, I think, is also just being able to speak to someone just who can expand that just in time knowledge. ⁓ so collaboration becomes super important along this way. Yeah. Like, do you agree with that statement?
speaker-1: I I do. I know that there's like a hot debate around AI on whether it is making us dumb and lazy. I honestly don't feel that effect on me, maybe because I'm getting dumber, I don't know. But but I feel like I'm learning so much more. ⁓ because I'm l using it as a learning tool. I'm also bit of a like a conversation thinker. Like I I oftentimes when I when I need to think something through, it really helps me to have a conversation with someone. Yeah. So having AI and having a conversation with it really helps me think ⁓ through some some things, right? I know that many, many testers use AI to create test cases, right? Which arguably maybe not the best idea, but if I were using to use it that way. I think it would help me like kick off some ideas like all right, it
gave me five, I'm going to create another five on my own because it gave me inspiration and maybe delete those two which were ⁓ irrelevant. So I I would definitely agree, I would definitely say that that AI has been a a great learning tool and specifically for that adjusting time knowledge.
speaker-0: Yeah, definitely. ⁓ yeah, I'll g I could easily go on to more of that, but ⁓ there's a a first question that I need to ask is like ⁓ because I think this becomes important for kind of the entry gate, ⁓ so to say, into getting into kind of the technological space, like development testing, things like that. ⁓ so for someone who's listening who comes from that non-technical background or less technical background. how how do you kind of see like understanding the building blocks, maybe a little bit of AI, to becoming like a technical quality consultant? Because that's the kind of the goal that I think all of us should be going for, even if you're inside a team, but like you should be the the person that like for my team, I am the consultant on quality ⁓ I might not s I'm not gonna solve everything, but you can ask me questions.
speaker-1: Yeah. that's a that that that's a loaded question, right? I I ⁓ so just a minute ago I gave an gave an advice on how to sort of reflecting on my own journey of how how I moved a little bit more into the dev space to to get better at testing, right? ⁓ and I think the same principle applies nowadays. Only now I think There might be a slight difference in what we are about to learn, right? AI is obviously the big thing. And I think it's really important to learn about what it is, what it can do. ⁓ and then I think that opens up ⁓ open up opens up the opportunity to to be the the quality advocate in your company, right? So what I mean by that, ⁓ For example, ⁓ I think at this point everyone tried ChatGPT. They understand that you can ask questions, it feels very natural, right? Maybe you have even tried like cursor, cloud code, codex, all these coding tools. ⁓ and maybe you get like sometimes you get good results, sometimes you get bad results. Understanding the basic stuff around how these things work and how they can give you good results is really crucial. So for example, if I say context window and you don't understand what that is and how to manage it, then this is a great area to look into, right? So so I just so I don't leave it like that, context window is basically almost like a RAM of your of your LLM. It's not really a RAM because it doesn't really store information, but it's the working memory, working capacity of your LLM. And you might heard that ⁓ Cloud Opus four point whatever has a one million token window, which is obviously like how much information you can put into it, how long your conversation can be. The problem is that when you look into benchmarks, most of the LLMs, doesn't matter if they come from Entropic, OpenAI, or some other provider, have a huge drop off in performance around two hundred thousand tokens. So that means you need to keep your conversations short. You maybe need to spawn different different agents to work on partial problems and not try to ⁓ like have LLM solve a huge ⁓ huge problem like find all the bugs in my code base or something like that. And and the reason why how this ties into into quality is once you understand how how these things work you start to think in terms of agents right so how how can we orchestrate agents and loops and all kinds of agentic systems once you have that understanding you start to realize that there are some reasons there is some resemblance of how we work with ai to how we worked with humans and assembling developer teams right so you have the generation layer, the developer layer, and then you have verification layer, if you will, the tester layer, right? Yeah. And once you create that division of ⁓ labor or like separation of concerns, right, you have your cloud code, your agents that generate the code, and you have your agents that verify the code, you start to understand like where how to like make the delivery of quality software in this AI agentic era, if if you will. So I think like getting knowledge in these kind of areas is going to be very vital for the tester 2.0, if you will.
speaker-0: Yeah, definitely. And so when I had Merritt on the podcast a while back, and she said something similar to you, and ⁓ one of the things that she kind of put across to everyone was that like it's just a tool that allows you to do these things, but the most important part is that it gives you freedom in time, literally time, to be more imaginative, right? Because th we're not doing these rote tasks, like writing a test case, for example, right? Like you said earlier. I could write ten, but those ten you could get AI to write ten, but those ten are okay, right? But now you can go, ⁓ what is it missing? I can actually look and see like that because I've got time to focus on it. ⁓ and it bec it becomes quite fascinating when you look at systems that way. ⁓ and you can build things out. ⁓ and it gives you opportunity to build new and fun and exciting things, which then unfortunately leads to weird and wonderful bugs that start coming in. one of the things that I wanted to talk on, ⁓ especially around this, ⁓ and I think it's really important, you and I are roughly on the same page here, is kind of the ⁓
speaker-1: Yeah. ⁓
speaker-0: fighting flakiness, right? ⁓ when it comes to tests. And I think kind of AI, when you start running it, is probably gonna catch some of those flakiness because it doesn't understand the underlying system, right? Which I think is you've said super important that we need to understand it because suddenly the front end is speaking across a highway to the back end, right? and I like so I've said in the past, the reason why you got flaky tests is because you don't understand that highway, but you also don't understand how the browser's working in the background, right?
Because it's running things asynchronously, your brain tends to think synchronously is a mismatch in in kind of that point. ⁓ I know that you've also got a few other ideas when it comes to flakiness. Could you tell people what you think are the other causes of flakiness?
speaker-1: Yeah, I ⁓ I had ⁓ so in my previous work at Replay.io, we talked a lot about test flakiness. We had a whole product on ⁓ on that. and at one point we reframed flakiness as a gap knowledge. Yes. ⁓ it's just ⁓ it's just a knowledge ⁓ knowledge gap. There is something working that you're not aware of, right? And that can be That can be your your ⁓ your React framework doing some operation which you haven't accounted for in your test. Because in a way, why should you? Right? You want your test to be clean, just like go ahead, press this button, fill this form, send it, give verify ⁓ it was it was sent. But there's like so many different operations. happening on your front end, so many different operations happening on your back end and they have different timings, right? And obviously some of those things will never ⁓ manifest themselves when you review them manually. But if you have a tests automation that is going super fast, just like ⁓ flying through your application, it might reveal some of those cases ⁓ that that just caused some random bug or your input being deleted as you try to type into it. Or you click too fast on something because there was an asynchronous operation reloading that button you wanted to click on. Right? And yeah, you know, there's ⁓ we had a debate online about this because we have different testing frameworks now and some testing frameworks try to try to mitigate it, try to find out whether an item is actionable. And these checks are great, but of course, and I and this is the part I think we agree on, ⁓ just just because you are using a a different framework doesn't have doesn't mean that that gap of ⁓ knowledge is not still missing, right? Yeah. And when you whether you do your click through JavaScript or you do it through web driver call or you do it in any other way, you might have that same problem on the same app. The difference might be, you know, how quickly they go, what kind of verification checks they have in place. But even though if one framework ⁓ you know ⁓ performs better than the other that performance difference might be sort of accidental because you still have the same app with the still ⁓ s with still the same issues in you know the asynchronous operations management and and all that kind of stuff. ⁓ so I think like for me it was really helpful to look at ⁓ flakiness as a knowledge gap because your application is doing something in the background if you try to click a button and you wouldn't be able to do that. ⁓ the question is just what it is.
speaker-0: I I quite like the term knowledge gap. ⁓ I a hundred percent agree with it because it's ⁓ it it it creates ⁓ a different way of looking at the problem. The thing I've always found fascinating, ⁓ kind of having worked in this space for a long time, is that like you always get the people going, ⁓ you don't need a wait library, for example. Gleb said it famously for Cypress. Pavel said it when he was working on Puppeteer, and then he reset it when he started working on Playwright. It's like we don't need weight libraries. And then six to eight months later, weight libraries suddenly appear, right? Because that knowledge gap, ⁓ people keep hitting it and then going, ⁓ it must be the framework. And it's it's it's not. And ⁓ I think it then creates interesting discussions because then ⁓ it all almost borders on flame wars ⁓ in the the old days like I'm old enough to remember when we were like this IDE is the best thing it's like IDEs are IDEs right like they've got yeah some are better than others in one way but then they have weaknesses in others so it's you've got to like find the balance that is right for you. and I think kind of ⁓ it then creates Like if you see it as a a knowledge gap, a knowledge gap can be closed, right? You can go learn about that thing and go, ⁓ wow, okay rather than go, Okay, I'm gonna spend the next six months rewriting my tests in this other framework that's claims to be better because it doesn't have a knowledge gap, but it does because I don't realize it yet.
speaker-1: Yeah, the the thing is I I used to give ⁓ give an example, maybe a visual aid would help here, but imagine having a button element and inside that button you have a span element. Right? And within your framework, whatever you use, you might have a framework that checks for a button to be actionable, right? But if you instead of targeting the button, target the span element. That span element might not have any any attributes that could be checked on for actionability. So you might still have the same dead click happen if you're targeting the wrong element. So if you know which thing to target, if you know that you should target the button element instead of the span element, then your framework, if it has those ⁓ those checks, might help you. But if you don't have that knowledge, you can have any framework in the world, it's not going to help you ⁓ write more stable tests. So like the problem is the problem of flakiness is elsewhere. Yeah.
speaker-0: ⁓ when you were a ⁓ a Cyprus ambassador and like I was working on Nightwatch, it came out roughly at the same time. It was fascinating to see that you like there was this move from kind of because of flakiness, move away from heavy end to end test to kind of more component testing. Like I'm just gonna test this React component or whatever. Doesn't need to be React, but I'm just using it as an example. ⁓ I know I started seeing a lot of people get better understanding of how the system worked because now suddenly you you are literally just testing one part of your application. ⁓ and you can mock out certain bits, like if it's going to the database, whatever, you can mock it
out and suddenly it's like testing became fun again. is that something you saw too? ⁓ and like d did it improve the way that people were testing?
speaker-1: Well there's there's two two things, right? The the first thing, me using Cypress was basically a matter of luck. I was ⁓ I was just in ⁓ just starting with test automation. Honestly, I couldn't get through the the Selenium installation. For some reason, it was really too complicated for me. And then ⁓ Cypress actually had a had a one of the first like showcases ⁓ on a conference in Bratislava in Slovakia. And my colleagues came in, hey you sh maybe you should check it out. And the rest is history. You know, I I asked for an for an invite. It was like version zero point seventeen or something like that. It was early in beta. No one even told me that I shouldn't like rely our whole company's testing strategy on a beta product. So ⁓ I did that and it was fun to use. It was easy to understand. So that's that's how that happened. ⁓ there was wasn't like super professional evaluation of of things. But one thing that Cypress had going for it is that it was in JavaScript. It was written in the language that my fellow developers been using. If I started doing in Java, I know Selenium has support for for JavaScript as well, but the community around it is not as big. ⁓ And ⁓ yeah, if if if that were ⁓ different we we I might have a whole different ⁓ history behind me. ⁓ the other thing about component testing, I think that that component testing was really like a smart idea on Cypress's ⁓ end. ⁓ though I think it didn't really play out the way they planned for it. Because ⁓ component testing ⁓ was something that developers would be heavily using before. They were using tools like Jest and maybe some other I'm spacing out on the name, but but Jest is like probably the most popular. The problem with Jest was that it would be running on this virtual browser. ⁓ I I forgot name VDOM or JSDOM or something. Yes, VDOM, right? Basically rendering ⁓ rendering your component inside a terminal, right? Yeah. So you would load your terminal, you inject all the state inside it, and then check for different things. And ⁓ it's been widely popular, but also kind of annoying to use. And what Cypress did was like, Hey, how about we test this component inside a real browser? And then not only you can test it, but you can actually see it. And you can use dev tools to interact with it once it's like loaded in your in your browser. ⁓ And ⁓ soon after that idea was ⁓ was implemented also by vtest. ⁓ right, so now you can you know load your components in vtest in a real ⁓ playwright browser. I believe you can even use web driver IO ⁓ for that kind of thing ⁓ now. And I think it is really smart because Sometimes y testing your different states of components is all you need, right? We're getting back to that testing pyramid where you like start really low level with your basic building blocks and then you have one or two tests that go end to end and and you're fine. ⁓ the thing nowadays is that because end to end tests are so pretty, we build so many of them and they like really give you this feeling of confidence that yeah we have these user stories so that so our user stories will never break. The user that will go through this story will never experience a bug because we have this path tested. Yeah. Well the thing is if you if you have, you know, eighteen hundred tests like that, it's it's not going to give you the benefit you yeah you thought it is going to give you. In fact you now have a maintenance problem. ⁓ but I went on a tangent with with the component test. I do think there's a value to to to sum it up.
speaker-0: Yeah. No, definitely. ⁓ and like 'cause you've been kind of speaking as an ambassador, you work at Kodo and other places, like ⁓ if you have you come across teams that are struggling with unstable ⁓ tests and what is the so I'm assuming that's a yes, but like what is the first metric or signal that they should look ⁓ when trying to break it down. Like is it ⁓ an environmental problem? Is it the code itself? ⁓ is it the test? Right? Like because it's that knowledge gap that you were mentioning. What is that first metric that you go? This is where you start from and then you go from there.
speaker-1: I I think for me it wouldn't even be metric. It would be sort of a qualitative thing. Okay. Where where you see teams merging their PR even if they have failed tests. So if they start to do that, the moment they start to do that, what are those tests really? Right? Another thing, ⁓ th this was this was a feedback from a from a team back when I was working at Replay.io ⁓ because we were trying to solve the problem of of flaky tests for people. Thing is, doesn't seem so many people care about flaky tests. And number one argument is like if this flaky test that is constantly failing, if what if it was really pointing to a real failure, we would know already. Like this is a this is a crucial customer path. That if this was broken, we would know already. And imagine making as a developer making a change that's nowhere near the happy path that that you're breaking, but your test is failing anyway, of course you're going to merge it, right? Because you can just open up a page and see everything is fine. And ⁓ and and that's a problem. And I know this is this is not a metric, this is a qualitative problem. Yeah. ⁓ but I think it speaks volumes on whether your test automation is useful or not. ⁓ Yeah.
speaker-0: Yeah, I I I think that's a that's a really good answer. one of the things I've written about, at least this year, ⁓ I've written about it many times is kind of the idea that ⁓ focus your tests on the riskier areas and then you can start having those quality gates and then suddenly it's not just a thous like you're running a thousand tests, nine hundred and ninety eight are passing, you've got those two that are flaking, you've not done anything near the
Right. But if you had that one test that was passing all the time and suddenly it becomes flaky, that in itself is a signal that you should go, Yeah, ⁓ I should look at that thing, right? And I'm not saying you should have only one UI test, right? It should be kind of one key test that tests that area. ⁓ and then if you need to add another one, that's fine. But every step is a deliberate thing rather than just building out tests for the sake of building out tests.
speaker-1: There's an interesting thing about like having one test, right? Because there is a ⁓ I've played at some point I played with ⁓ code coverage. ⁓ I I wanted to explore this topic and I wanted to like instrument my app and and do all kind and do all that ⁓ fun stuff. And I added one single test will which just opened the app. And that would already create 40% code coverage, right? Which tells you a little bit ⁓ and if you ever built ⁓ an ⁓ front end app, you understand why that is. Because usually everything boils down to the main app JSX or TSX component, right? So if you load that and it loads all of its dependencies, you already have like a huge chunk of your component tree loaded. And if it loads, then of course you hit that hit that forty percent. ⁓ And I'm not advocating for for like code coverage being like a good metric, but it hints at what an and what kind of end to end ⁓ what kind of power an end to end test has. But also if you want to crawl from those forty percent to eighty percent, you're going to end up writing like two hundred tests. I mean, of course, depending on how big your application is. But it's ⁓ it's a slow way up. ⁓ which might not be worth it because with those 200 tests you'll have the maintenance problem that I mentioned. And also a great argument ⁓ I've heard recently and I'm spacing out on the name of the speaker, but he shared a thing that like sort of blew my mind. The he was doing an analysis on like what kind of bugs eventually make it to production. And there was this overwhelming number I think it was like 80% or maybe even more. But basically all of the bugs that were created by developers were bugs by omission. So that means something that should be there was not, and that created the bug. So imagine having a hundred percent code coverage. You still have bugs because some things are not accounted for. You have a switch statement that is missing the default bar or missing some case. ⁓ you still have a still have a buggy application. So that again like sort of challenges like what should you even do with your tests? How many you should you should write? And should you even be concerned with things like ⁓ code coverage or ⁓ having hundreds of end-to-end tests?
speaker-0: Yeah, I it's I people who know me and people who've listened to this podcast will tell you ⁓ very quickly that I despise code coverage. I see it as a vanity metric at best. Yeah. ⁓ because you could have a hundred percent, like you said, ⁓ full of bugs. ⁓ it could be omissions, but it ⁓ it's also kind of the edge cases, things like that, right? Yeah and it it has no bearing on ⁓ quality in the sense. ⁓ it can give you ideas of what you're missing in your tests. Yeah, exactly. But nothing more than that.
speaker-1: Yeah, it's like ⁓ it's like if you're ⁓ if you ever played like strategy games like Age of Empires, Command and Conqueror, you have that fog of war, those uncharted areas. Yeah. ⁓ and I view code coverage as that. Like there are some areas which I haven't touched on, so maybe I should cover them, but not for chasing the number, but for like having a good map view of my of my application.
speaker-0: Definitely. ⁓ so I think it'll also be like we kind of lightly touched on AI. I wanna just kind of do a little bit more ⁓ before we head into our quick fire round. ⁓ One of the things I've seen is around kind of this AI native testing versus kind of deterministic traditional testing. So deter like in this sense I mean like you do a thing, you should know the results you're getting. AI suddenly throws all of that out because it's all probabilistic. ⁓ How do you how have you seen kind of that changing the way people start doing testing now?
speaker-1: Yeah, I think we ⁓ so there are two main areas in my view where things are changing, right? First we have apps that are largely written with use of AI, right? And then you have AI enabled apps where AI is integral part of the application. So your typical chatbot, right? Yeah. On your on your homepage. And I think like these two categories ⁓ require different approaches when it comes to testing. Yeah. ⁓ and there there are overlaps, but what what I see is the main problem with ⁓ with the first category, applications created with the help of AI, is velocity. Right? How do you How do you govern the amount of velocity that ⁓ that you now have, right? Because you might have 10 engineers on your team. Now suddenly they have three, four, five times the output. ⁓ that creates not only like the velocity of the code, but also like the cognitive load of your tester, of your senior engineer who is reviewing all that code. ⁓ and then of course there's this haziness of like. All right, what even are the implementation details of that? How can we be sure that there are no bugs, your security issues, some you know, other problems where you change one thing and it breaks a thing into ⁓ on ⁓ whole another different place? Yeah ⁓ and I've heard on testing conferences that this is going to be like a tester golden era because we're going to start hiring more testers to to test all that velocity and I have bad news like that's that's not going to happen. I don't see I don't see this happening at all. ⁓ in fact we kind of see the opposite thing, right? I feel like the the velocity problem calls for for a smarter approach on verification, right? Yeah. And me obviously working for a for a company ⁓ that's doing work doing work on this verification layer, I I kind of see like hints on
what might be the future, right? ⁓ we have AI code generation, so now we'll add AI code verification to that. And on on first beat, that idea seems like it's a bad idea, like all right, we're going to have AI review AI. ⁓ but if you if you have a good separation of concerns If you think about cloud code and how it is optimized to give you a good code, then like you you know that there's some system prompts and some harnesses around it to be optimized to write that code. Right. So now you have to create a whole another machine which has harnesses and system prompts and all kinds of setup to be good at verifying the code, to be good at indexing your code, being able to take all that context in and call it in the right moment. and and be concerned with your code correctness, with your ⁓ not breaking any other like connected repos or services with your security and other things. And so having that verification layer does not mean we're going to rely completely on AI of doing the verifying work. But it gives us it kind of solves these velocity problems where you cannot up possibly read twenty thousand lines of code a day. No. But what you can do is have a basic check of all of the correctness, security, and other problems you might be looking for anyway, if you were if you were looking into that code, but it frees up your cognitive capacity to think about like the bigger problems, like the architectural problems. Like, do we even want this in our code base? And all kind all that kind of human ⁓ process decisions. Yeah. ⁓ so that's like the velocity problem. The the applications that have AI as integral part of ⁓ of the system, that's a whole another area. And I think like many testers nowadays are trying to look into that. And I think it's an interesting area. Like we have this eval testing, right? Where observing how the application is being used. We're very concerned with security of those systems. Like how can we not have ⁓ prompt injection hacks when we open up a gateway to an a ⁓ to to an LLM from outside? And this is like largely unsolved problem. I really recommend listening to some of the talks that Sander Schulhoff has made. He's been ⁓ a real pioneer in this in this area. ⁓ And ⁓ and it he's like really interesting to to listen to because he says we now have a spring of companies, you know, that are selling your like security harness. The problem is it doesn't work. Like you you're getting sort of a fail ⁓ fake, you know what's the word I'm looking for? Like reassurance that your system is safe. But as soon as you open up that chatbot to the world, it's always open for any types of ⁓ outside hacks. ⁓
speaker-0: My my favorite one at the moment of that, I don't know if you saw this, was high ⁓ high worth Instagram accounts were hacked ⁓ using the meta bot who is like, I forgot my password. Here's can you change it? And they like they managed to kind of talk the bot into going, ⁓ okay, yeah, you're you're the person. Right. ⁓
speaker-1: Wild, yeah. And the thing is, most of these you might have some security you know ⁓ what's it called? Like some ⁓ some security systems in place. The thing is the moment you even like change language, they're going to get like progressively weaker. So if I try to hack a system in Slovak language, I might have a higher chance that if I try to do that in English. Yep. Which is wild if you if you think about it. And we're going to have more and more of these. ⁓ And I think anyone who who will try to like open up their chatbot, AI chatbot to the world, ⁓ might ⁓ might reverse that ⁓ that decision. ⁓ I know there's a big push, you know, for having like your support tickets ⁓ solved, not by human, but instead by by a chat bot. And I and I think like my prediction is some of the companies might like backpedal this decision and rather have some ⁓ responsible human being on other side of that chat. But we'll see.
speaker-0: Definitely. Right. ⁓ we've reached the quick fire round. ⁓ so in this point I'm gonna ask you questions. ⁓ You and I, we're both wafflers. ⁓ so you're not allowed to waffle now. ⁓ it needs to be a short, sharp answer. Okay. Are you ready?
speaker-1: I'm not, but let's do it. ⁓
speaker-0: ⁓ favorite testing tool that you've used recently?
speaker-1: I'm gonna say playwright.
speaker-0: Okay. A book every QA lead should read.
speaker-1: Never split the difference by Chris Voss, who's an FBI negotiator.
speaker-0: Yeah. I you're not the first person to recommend that book. Yeah. ⁓ the most underhyped QA metric.
speaker-1: ⁓ really? Okay. think I've said it already. How many force merged PRs you have?
speaker-0: Awesome. ⁓ and the final one. If you weren't in testing or kind of tech, what do you think you'd be doing now?
speaker-1: ⁓ music. Definitely music. I'd be a songwriter.
speaker-0: Awesome. Philip, this has been amazing. Thank you for reminding us what quality means to ⁓ you and to the rest of the world. ⁓ and kind of just explaining culture and of technical al excellence and user empathy. ⁓ I've really enjoyed this. ⁓ thank you for your time today. If people wanted to see what you're doing or where to follow you, w where can they find that out?
speaker-1: Best way would to be just to connect with me on social networks. So I'm at LinkedIn, Twitter, and Blue Sky a little bit. ⁓ so you can find me there. And I have my own page, but ⁓ I I don't blog that often anymore, but maybe I'll pick it up once again. ⁓ thank you so much for having me. ⁓ it's been a very enjoyable conversation. I've been looking forward to this and I'm glad that I finally got to be part of this.
speaker-0: Awesome. Thank you for being here. ⁓ and to everyone listening or watching, please make sure that you have subscribed so you can be notified when our next episode comes out. ⁓ and we will see you soon. Thank you everyone.
speaker-1: Bye, everyone. ⁓