ТРАНСКРИПЦИЯ 🔗
speaker-0: think you've mentioned one already, so latency, and that's one of the most popular ones. It basically refers to your small percentage of requests that basically take significantly longer to process for a small amount of users. And if you look at the average, ⁓ it might show that your application is healthy. Performance testing to... you know, real production signals rather than like second guessing. But performance is not my job, it's the QA's job. Not only get involved when there's a production issue, so try to talk. We focus a lot on doing load testing, testing the protocol level, but that means we're ignoring the user experience. So browser testing as part of your overall performance testing and that means rather than having a lot of... thousands of browser virtual users, you can actually do tests being exposed to thousands of virtual users and then see how that impacts one another.
speaker-1: Welcome to Browser Stack Talks, the podcast where we turn testing pain points into playbooks. I'm your host, David, and today we are deep diving into the world of performance, reliability, and how you actually see what our systems are doing under pressure. Our guest today is Powerhouse in the testing community. She's a senior developer advocate at Grafana Labs and a frequent speaker. ⁓ but I can't do you any justice. Marie, please introduce yourself.
speaker-0: Nice, thank you David. ⁓ So as David mentioned, my name is Marie Cruz. ⁓ I've been doing software testing now for more than 10 years, but I actually transitioned to become a developer advocate. So I've been with Grafana Labs now for, geez, almost four years ⁓ this August. It's one of those things where I wasn't really sure what developer advocacy is, but... I didn't realize that I was doing it already, like trying to speak at meetups, writing blog posts, sharing learnings about a specific tool or a practice. And so it's great that now I'm actually getting paid to do this. I'm also the co-author. So I co-wrote a book about contract testing with Louis Prescott. So the book is called Contract Testing in Action. So if you want to know about contract testing, ⁓ how is it different from, I guess, integration testing, then yeah, I also co-wrote a book about that. ⁓ While I'm not working, so on the side, ⁓ I'm a mom of two kids. ⁓ I also have two big dogs, so household life is pretty busy. But apart from that, I am currently into doing crochets. I like to do photography and I dabble a bit with karate with my daughter. So that's how I keep myself busy these days.
speaker-1: Awesome. ⁓ one of the things that you just said there, ⁓ I'm sure you've met ⁓ Jenna Charlton. Jenna's on my team. and I'm sure you and her must be ⁓ kind of exchanging the good, the bad, the ugly of crocheting 'cause she I know she's so good at it.
speaker-0: Yes I'm actually a newbie when it comes to crocheting. I know that she does a lot of crochet but when I met her in person I still wasn't doing it. It was just this year
Load full transcript
that I started to do it. ⁓ When we had our community conference in Grafana, one of their swag is actually a crochet kit. I'm still yet to finish it. If I made a mistake, always just start again because my mind is just, you know, I have to like perfect this. So it's taking a bit of time, but ⁓ I've done like, you know, simple things like a scarf. I've done a hat, but then it was too small for my daughter. So I had to give it to my youngest son, but he doesn't like to wear it. So yeah, I'm still a new, like I know a newbie, but yeah, just keep on practicing.
speaker-1: That's awesome. And and hopefully at the end we'll be able to kind of ask you a bit more about that. ⁓ when we get to the quick fire round, I'll ask I'll I'll make sure that we have a question around that. ⁓ but I think kind of the one of the things we we should definitely talk about and hasn't really been talked about ⁓ on this podcast yet, is ⁓ the modern performance engineering mindset. I kind of feel I I saw it in a lot of places. So ⁓ I just did a recording ⁓ with ⁓ Baz ⁓ Dykstra. I don't know if you've met him before. and ⁓ he was talking about like you know the maturity model of every company, even from the biggest companies, they tend to get this things wrong and without realizing they're getting it wrong. ⁓ and so you've been a advocate for moving away from the old test at the very end model. ⁓ and so you've kind of been pushing for more performance, things like that. How do you like so so that we can clear things up, like how do you t define the difference between traditional load testing and kind of what is considered modern performance engineering?
speaker-0: That's a really ⁓ excellent first question. And to make it simpler, for me, the difference is when and how performance is addressed. if we think about this, like traditionally, know, load testing or performance testing of any kind, you only do it right at the very end. ⁓ It was often treated as this ⁓ final checkpoint of asking, can our system actually handle the expected number of users before? You know, we go live and then teams would run a very large test. They're going to review the results. You're going to find, you know, issues and then do a fix. And then that's it. They're going to only do it like during at the very end. And I think that approach work ⁓ better when releases where we're ⁓ very infrequent and I think systems back then, they were much more like predictable. But nowadays, you've mentioned systems are now mature. So I think even our testing practices is also getting mature. There's a lot of talks about ⁓ quality engineering, shift left, shift right, continuous, all those buzzwords. And it's pretty much the same with performance testing. it should, like modern performance engineering, it ⁓ aligns with this practice of of shift left and people ⁓ already know this, but shift left is really about finding and fixing issues as early as possible instead of waiting until the end of development. in the context of performance, that means we're not waiting for the final load test to discover that you have a service that ⁓ can't scale or you have a database query that's too slow. So I find that modern teams now, are trying to integrate performance thinking into every stage of the development ⁓ life cycle. you know, teams now, are starting to consider performance during the design, the development, obviously testing stage, but even in the deployment and in the production stage. So prior to joining ⁓ Grafana Labs, so I was working ⁓ as part of this team and you this is where I introduced K6 as well ⁓ and we were trying to really break it down into much more smaller tests because we know that if we leave it right at the end it's gonna be much more difficult you know to test so I think nowadays it's really more about you know starting performance early because we're doing that with all the other types of testing so why not performance as well
speaker-1: Yeah, I I I agree. I think kind of ⁓ people should be doing that. ⁓ the the thing that I also see from like speaking to customers and things like that is that teams struggle with the when to start. Right. They go, I know I need to start it sooner. and everyone's like you said, all the buzzwords, shift left, shift right, right? Like all of these things. And you're like, I've got to do all these things. ⁓ how can
speaker-0: stop.
speaker-1: How have you seen people practically do that with the shifting left of performance testing without kind of having a negative impact on kind of their CI C D, their pipelines, all of this? How how have you seen that?
speaker-0: And I think this is a ⁓ statement because many teams, they do struggle with ⁓ shifting performance testing to the left because they think that it means running ⁓ large production scale like Lotus earlier. ⁓ But in fact, that's actually not the goal. ⁓ And when we think about shifting performance testing left, it should be really about starting with a small, know, FAS automated performance check because that will naturally fit into your CI-CD pipeline. And a good approach that, you know, I've tried personally and something that, you know, I try to share with the community is to use this layered ⁓ testing strategy approach to performance. So that could mean, you know, you run a... ⁓ We have this concept of smoke testing. It's the same with performance testing. You just have to run a very small ⁓ performance check as your smoke test. ⁓ You can do that as part of your pull request pipeline. ⁓ You can keep that to just one or two critical scenarios. If you have ⁓ an important key back end ⁓ point, focus on that. ⁓ Keep it to a... ⁓ very low concurrency as well. again, the goal ⁓ is to ⁓ not find a lot of obvious regression issues. It's not to really simulate peak traffic. ⁓ So keep it to a short duration as well, because again, you don't want to block the developer's workflow, because if they see that the performance test is running for such a long
time, then they're going to be you know, ⁓ annoyed and like they're gonna think why is this part of the full request family is gonna slow them down. ⁓ So I think you should initially just try to separate what the fast performance test is versus the deep ones. So the deep ones, this is where I talk about like your large load test, your soak testing, your stress testing. So those still matter, but you can run them let's say on a scheduled run or you can run them ⁓ on demand or before a major releases. But then having these couple of minutes performance checks as part of your ⁓ full requests pipeline that can catch obvious regression issues. So I think teams should start looking at ⁓ what are the critical endpoints that I need to write a test and then focus on. doing that first rather than a full-blown ⁓ load test.
speaker-1: ⁓ something you said there just kind of triggered a memory is that like you said how you you know doing something even if it's ⁓ kind of a shallow test to do that is still important and it's to me it's a lot similar to kind of how people go ⁓ I can't do test driven development because it's going to slow me down, right? Like and so to me it's like I can't do a performance test because it's gonna slow me down. It's like, well yeah but at the same time If it means you get releases quicker and of a higher quality ⁓ from the start, rather than going, How quickly can I do it? It's about how quickly can it go out there and stay out there without having to be reverted, right? Like it that is far more valuable.
speaker-0: Yeah, it's really important to sell those as well because I think, you know, there is a lot of misconception with performance testing. And I think that's quite the same with other cross-functional requirements because, you know, they are often harder to measure. Whereas if we just focus on doing functional testing, like we know that, like, we can see it. Like if something is broken, then we can actually see it. But performance testing, I think it requires more ⁓ convincing. And I think this is like, for example, you're right, like with the test-driven development. So it's the same with performance. Like it's spending a little bit of time earlier on, but then it's actually going to speed up your entire process because you're actually going to find issues earlier on. And this is why I mentioned a while ago to focus on critical endpoints because let's say you have an endpoint for your login, for your search, or for your checkout. Just write a very simple test for that. Again, it doesn't have to be a lot of concurrency. It could just be a very small amount of virtual users. And if you notice immediately that there's a lot of error rates, that the latency is slowly creeping up, that your logs are returning a lot of, let's say, errors, and it's pointing to this one service, then you know that you've ⁓ caught an obvious regression. really a big like this high value in writing smaller performance testing.
speaker-1: Yeah, definitely. ⁓ and then kind of I I don't know if you've had this as well, ⁓ because I've kind of I've dabbled in all forms of testing. They ⁓ i is that like people tend to then focus on when they think performance, ⁓ they think of latency and throughput and error rates. what are the Hidden metrics is a bet I can't think of a better term, but the hidden metrics or behaviors that teams should look for ⁓ before it's too late.
speaker-0: Yeah, I think you've mentioned one already, so latency, and that's one of the most popular ones. But one thing that ⁓ I guess a lot of people maybe don't know is there is this idea of tail latency, is ⁓ it basically refers to your small percentage of requests that basically take significantly longer to process for a small amount of users. I'm not saying that latency is not a good metric to track. If you look at the average, ⁓ it might show that your application is healthy. But then, ⁓ the latency, so this is where you look at what your P95 is or your P99. So your 95th percentile, your 99th percentile, and even 99.9 percentile. Because let's say if you have ⁓ a latency of, I don't know, your average is let's say 200 milliseconds and then whatever testing tool you're using in terms of performance ⁓ it reports that you know p95 is equivalent to ⁓ let's say 400 milliseconds so that means that 95 percent of your users are experiencing response times ⁓ less than like ⁓ up to or less than 400 milliseconds but then there's five percent that are experiencing a much longer request. And then you need to basically see the difference between your P95 and your P99 because if it's quite high, then that means that for a small amount of percentage, but that percentage can still be a lot of users, it can mean that their experience is not going well. don't just focus on the average latency, also look at what the different percentiles are because that is a good indicator. ⁓ if you have some users who are also experiencing some issues. ⁓ And then the other one that I can think of, so ⁓ there's this acronym called USE. So we have U for utilization, S for saturation, and then E for error rate. And I think ⁓ saturation is often the forgotten USE metric. because it's not very visible, it's not very hard to measure. Because if we think about utilization and error rate, let's say ⁓ you can immediately see if your, for example, if your CPU is at high utilization, for example, 90%, you can see if your memory is high, you can see that this errors are happening, that you've got a lot of requests failing in your error logs. But ⁓ I've noticed that saturation is not always exposed as a single metric, it's not always visible, but it's important because it gives a much earlier indicator if your utilization and your error rates are gonna go up high. So when I say saturation, I refer to the amount of work that your resource can't handle at the moment. So if the saturation is slowly creeping up, your latency and your error rates can still be fine. But
then if you can see that the saturation is starting to go high, then that's a good early indication that, okay, this is something that we need to focus on because this is gonna impact utilization and error rates later on. And again, it's quite hard because this could be like different signals. It could be like your database. ⁓ connection wait time, could be like if you're looking at your request backlog or it could even be like what the disinput output weight. So it's not just one metric. So teams often have to look at different signals altogether. ⁓ But yeah, I think if you can try and ⁓ measure or observe saturation earlier on, then... you're already in good tracks to also monitor the rest and to see if the rest are also gonna start going high.
speaker-1: Yeah. and I think it's the ⁓ something you said there was very interesting. Like if you say four hundred milliseconds is your ninety-fifth percentile. I I remember when I started doing a lot of ⁓ front end performance. ⁓ kind of I did that very early in my career and there was a big push. ⁓ it's gonna show my age really, but the ⁓ there there was a lot of work by Yahoo at the time and they had a nice tool called Y Slow. ⁓ that would like kind of tell you where everything was. And the the kind of the golden number was always like you need to be below two hundred milliseconds. Everything after that ⁓ becomes perceptible to users. And if the if a sl site is slow or ⁓ at 200 plus milliseconds, ⁓ people would just turn off, right? And I s I think the attention economy now is even worse than that like sixteen, seventeen years ago when that was the kind of the research that was coming out. not that I want to put numbers onto it. ⁓ is that number still kind of realistic from what you've seen? is it kind of even faster now?
speaker-0: Yeah. ⁓ I think so if we talk, I'm gonna just ⁓ carry on with the performance with the front end performance because for example now the golden signal that Google recommends is like Google Web Vitals and I think they also mentioned that number I'm not exactly sure what but like if I go to Google Web Vitals you can exactly see what the actual recommendations are and if you've exceeded you know past this number then you you're gonna be losing customers already, it's gonna impact your SEO ranking as well. So I think even though ⁓ it's still, I mean, our attention span is obviously much more shorter, they're still valid. ⁓ It's like the recommendations from years ago are still valid, because I think overall the important point is like, if you see that your numbers I guess are much more higher or longer compared to a competitor then yep, it's a safe assumption that you are losing some of them because if you have some pages that are slow and doesn't really follow a lot of the recommendations and Google Web Vitals is a really great way as an example because we can use a lot of tools out there, we can either use ⁓ like lab data via Google Lighthouse or No, there's much more. I've
forgotten because this was like years ago, but there is a tool that basically it allows you to measure what the performance is for actual users. So Lighthouse is great, but it's only for like lab data. So it's much more controlled. But then there's actually a tool that Google recommends for also looking at the field data. So yeah, I think the recommendations are still very much valid.
speaker-1: Okay. ⁓ and then kind of moving on to keeping the mindset is that like ⁓ just like quality is a team sport, ⁓ I do believe r reliability is also one of these things. ⁓ how how have you seen teams kind of share out that responsibility between developers, QAs, SREs, how anyone else really?
speaker-0: Yeah, ⁓ I think this sort of sets ⁓ teams apart. if you have, like for example, if you're really practicing ⁓ modern performance engineering, you know, I'm just gonna focus on the lead disciplines. So developers, QAs and SREs, so they all have different perspective ⁓ from how they see the system. And so if... all those disciplines if they come together and they don't ⁓ compete because I think there's a lot of overlap and they share the same goal when it comes to reliability. let's say for example developers, know, they are the closest to the code so they can really play a key role in preventing performance issues ⁓ as early as possible. So they should think about ⁓ performance when they design features, when they write pull requests, ⁓ when they pull in a dependency, they need to check, okay, what's the performance risk if I add another dependency to my package? Or if they're reviewing someone else's changes, ⁓ just try to think about performance angles ⁓ at a very early start. ⁓ And then on the other hand, know, great professionals, like we bring in the user, we bring in the workflow perspective. So the way they can help teams is they can help define realistic test scenarios, you know, rather than second guessing, okay, I think we need to use this number for virtual user. We need to use this for duration, like make that realistic and you can use production data to basically ⁓ inform your testing. And this is where QAs and SREs can really work together because SREs, bring the production perspective, they bring a lot of reliability aspect, they understand what the SLOs are, and they've seen a lot of incident patterns. ⁓ And so I think if SREs also work together, they can help connect ⁓ performance testing to real production signals rather than second guessing ⁓ the QA team or the developers second guessing what performance test they're right. ⁓ Developers as well, shouldn't, because I mentioned a while ago, they shouldn't really compete. They should work together. So developers shouldn't say, but performance is not my job, it's the QA's job. you know, QAs, shouldn't really be left to find performance issues like at the very end. And then ⁓ SREs, shouldn't ⁓ not only get involved when there's a production issue. So try to talk to, you know, area as well much earlier on. And I think when the responsibility is shared like that, then testing becomes less about blame as we all know and it's more about having that fast feedback loop. So it's just all benefits after this. Like you can catch problems earlier, you can understand risks better. So I get it's hard to start, but I mentioned a while ago, just start small, keep talking to different disciplines and yeah, it's all gonna give you a lot of benefits.
speaker-1: I I think I think it's important that we we do describe it as a team sport, right? Like ⁓ every application is not just one engineer who's written all the features. Maybe for a small startup, maybe, but like w when you get to kind of something a bit more than ten people, it's kind it's definitely a team sport at that point from kind of the salespeople who are involved, the performance people, the SREs, the product managers, all of that. And I think it's it's important because I think it leads to ⁓ a better quality product at the end of the day because they should all have a a a way of informing it. And also kind of a ⁓ the thing I liked you said there was that ⁓ people can reinforce like go, this is what we consider the minimum performance test that we need to have. one of the things when you were talking there, that kind of made me think is that like you know, developers should know more about the performance. one of the things I've noticed and I and I say it from kind of my own experience of you using it, is that like using a gentic development ⁓ performance doesn't necessarily gel until you start getting other bits of signal coming back. ⁓ and I don't think a lot of people think about that performance point early enough anymore. I think it's becoming worse. ⁓ is that something you've noticed too, or is it just me?
speaker-0: I think, yeah, if we're relying too much on agents but then we're not looking at the telemetry signals, then that can cause a lot of misinformation. And so ⁓ I think in today's age of AI-driven development, using a lot of agents, it's great. They can help us define what scenarios to use, but then it should always be back up. with your production telemetry signals because ⁓ your performance test, they can obviously tell you that, all right, something is wrong because my request duration is high, we've got a lot of request errors, but then it's just the symptoms. So it doesn't really tell you ⁓ why that happened. So this is where you need to rely on your telemetry signals. ⁓ You can... ask the agent for some help, like if you need clarification on what those signals are, like how it can be improved. But there's still a lot of value in, for example, you actually looking into the telemetry signals and correlating them with your test run because they can really give you a clear picture of what is happening when and where it happened. And in some cases, it can also tell you how you can fix the issue if you've got ⁓ like for example, like continuous profiling, ⁓ you know, so you
can look at your CPU and memory profiles while your test is running. So there's still a lot of value in doing that rather than just relying on your AI agents.
speaker-1: Definitely. ⁓ and I think that hopefully creates a really nice segue into ⁓ doing using more observability in kind of everything you do because ⁓ observability started rising up like becoming a lot more prominent, say twenty nineteen, twenty twenty. at least from what I saw, right? So it was still in its infancy, but it was rising very quickly. ⁓ and you've been ⁓ kind of involved in that with kind of K Six and the Grafana ecosystem. could you ⁓ before we kind of deep dive into that, could you give ⁓ people a like a quick overview of what K6 is and Grafana for those who don't know?
speaker-0: Yeah, so K6 is ⁓ basically a reliability testing tool. So it started off as a load testing tool, but now we are talking about it from a reliability aspect because it also covers a lot of tests apart from ⁓ load testing. So it's there. It's a tool that can help teams ⁓ perform reliability testing ⁓ at a much earlier stage. ⁓ It's written in Go, but you write test in JavaScript. This is one of the things that caught me when I first started using it because I was like, ⁓ I can write it in JavaScript. But then like the actual code base is actually written in Go. I think ⁓ it's probably one of the ways as well how we can get a lot of developers involved if it's like a programming language that... People are used to and I think JavaScript is one of the most popular. So ⁓ you write the test in JavaScript, it fits naturally into your CI CD pipeline. It's open source, which is awesome. if it has a lot of downloads, so it just shows that people are ⁓ using it ⁓ on their day to day performance activity. And apart from the core K6, there's also other offerings. So for example, we have K6 Browser, which looks more into browser performance testing, more functional testing, and it's inspired by Playwright. So ⁓ the transition shouldn't be as difficult. So if you have a background, know, writing test in Playwright, and you want to start using K6 Browser to also... ⁓ measure performance testing ⁓ on the front end and to look at the user experience, then ⁓ that will fit right into your workflow. ⁓ Grafana, so a lot of people, we know Grafana as this visualization and monitoring tool. So before I joined Grafana, so they actually acquired K6. And I think this is a good sort of... marriage because it just shows that testing and observability, go hand in hand because in order for us to do better testing, we need observability and then your testing can also feedback into your observability stack. So in Grafana, we have ⁓ the core ⁓ LGTM stack as we like to call it. So these are all open source and it's basically our ⁓ you know, ⁓ offering to, you know, get people started with observability. So if you want to measure your telemetry signals, you can use any of our ⁓ LGTM stack to start ⁓ basically getting those data. So if you want to get logs, we've got Loki. So that's L for Loki. And then the G is the Grafana, which visualizes all your ⁓ metrics. If you've got traces that you want to be collected, can use ⁓ tempo. ⁓ And then for metrics, which I guess it's one of the most popular, ⁓ let's say if we think about this pillar of observability, metrics log traces, like metric is the most popular. And in that sense, we have ⁓ so ⁓ for Mimir. ⁓ It's all open source. Mimir is quite inspired as well by Prometheus. So if you're familiar with using Prometheus for your metrics, then ⁓ Grafana can also ⁓ help with that. that's just a quick sort of background. There's also a lot of other open source projects ⁓ as part of ⁓ what we offer, but that is sort of like the big ones, the LGTM stack. And then of course, there's also K6.
speaker-1: Okay, cool. ⁓ and so when people are using these, ⁓ kind of what are the critical items that people need to be looking for ⁓ when using these from blogs, metrics, traces, like in their systems? Like because like it's cool to set these things up and then they go, Cool, what next?
speaker-0: Yeah, so I think let's maybe do this in the context of let's say if you're running a K6 performance test. ⁓ I mentioned a while ago that K6, it obviously tells you that there is an issue, but it doesn't really tell you why it happened. So ⁓ to get started, you've got to instrument your ⁓ application and you can send a lot of the telemetry data to you know you don't have to use the LGTM stack there's a lot of like data source ⁓ out there so for example you can use Prometheus if you want to store metrics about your system and metrics is great because ⁓ it can tell you which component is know strained at the exact moment that your k6s is ⁓ running so For example, you can look at ⁓ HTTP requests, you know, duration or the total of HTTP requests, and then you can correlate that with your performance test so you can see ⁓ what is, you know, changing when your K6 test is running. Because you can, if it's all in, I guess, one stack and one dashboard, it's much easier to view, you know, the results accordingly. ⁓ you can obviously ⁓ look for logs. So if you have like a lot of application logs and you can obviously send that to Grafana as well. And then when your case excess is running again, you can like logs because they tell you ⁓ what is actually happening. So they give you what the actual stack traces, like what the actual error is. So at least you can. use that if it's correlated with your K6S. ⁓ The one other ⁓ thing that this I'm still quite fairly new so I didn't really use this when I was ⁓ you know when I haven't joined when I was still ⁓ in my previous roles ⁓ but you can also capture traces ⁓ and in Grafana you can use tempo which Again, it's open source, but basically Traces can tell you ⁓ where the issue is happening. So Traces allow you to follow what the user request is doing, because it can track from one component to another component,
and it can tell you how long that took and if there are any ⁓ errors. once you're if while your k6 test is running you can also look at your applications traces because it can tell you for example which specific ⁓ database query was taking you know a much longer time rather than second guessing and you know doing your troubleshooting much more longer and then the last one this one i because this i haven't really done any project related to it, but I just know ⁓ some background. So we also have this pillar in observability called profiles. And this is the fourth pillar, which is quite new. originally we only had metrics, logs and traces, but now we also have profiles. So this is where ⁓ it can tell you why your code is actually slow. So. ⁓ Nowadays, there's tools out there that can help you automate continuous profiling so that ⁓ while your test is running, ⁓ it can also capture your CPU and your memory profiles and it can tell you why your code is slow. So without really using observability and telemetry, can sort of guess that a load test, ⁓ it can tell you that something's broken, but then... Once it's broken, you don't know why, so you're going to spend a lot of time trying to debug the issues. Whereas if you've got all these signals as well, and it's all correlated as part of your test run, then it can really give you this clear picture, and you can help your team debug these issues much more quicker as well.
speaker-1: I I've got a a a few talks later this year where I take observability in into account because I think it's such a ⁓ massive thing that I don't think a lot of people do. And like earlier you were saying about how SREs bring real world into their tests. And it's like, well, what if you then take the same data that SREs are looking at and pull it in yourself? ⁓ and I think that becomes Like I think there's a very interesting space there at the moment. I think it's still kind of growing and it's kind of like I said earlier, the ⁓ observability, it's been around for a while, but it still kind of feels like it's in its infancy. A lot of people are using it now, but it's still kind of people are trying to find that maturity like of w different levels of maturity. ⁓ and I do think ⁓ observability driven development is kind of the next big thing. ⁓ because it's it's interesting from ⁓ Selenium playwright, whatever, you can kind of you can set up your trace f in a in your test and then say, This is the marker for my trace, send it all the way through observability. And suddenly there's like, ⁓ wow, I can actually see the coverage right before because that was always one of the interesting ones where like, ⁓ I I hate code coverage as a metric generally. Because I think it it's a vanity metric at best. And it tells you what it tells you what you're not testing rather than what you are testing. and ⁓ but now you can go, ⁓ I can see it all the way through. ⁓ how like have you seen other people doing that? ⁓ and what is the kind of the takeaways that you've seen from those people ⁓ when they are doing that?
speaker-0: Yeah, I think like you said, it's still not much widely ⁓ practiced. Similar with when test-driven development really came into maturity, there's people who are doing it and then there's people who aren't doing it well and then there's people who are like, don't do test-driven development. I think it's quite the same with observability-driven development and I'm obviously just speaking from my personal experience, but if I, for example, attend testing conferences and if I ask different people, do you do observability? What observability tooling do you use? I think they understand the term, but in terms of actually practicing it, it's still quite early days. And it's great that you mentioned that you're gonna start to bring in observability, because I think we need a lot of talks. We need a lot more guidance ⁓ on how observability can help with better testing, which can lead to then better quality. And ⁓ I actually posted ⁓ a LinkedIn post today just to give an example on how testing and observability goes. you hand in hand. ⁓ And one of the ways that you can do it is if you use production data to really ⁓ design your performance test. Because I think in the past, like, and I've been quite guilty of this, whenever we do, okay, if we're gonna write the performance test, what should be our virtual user? And then we'll just give a random number. So it's not really a realistic scenario because it might pass with whatever you provide, but then if you deploy in production, it can still fail because it doesn't actually mimic what the production signals are. now, open telemetry as well as, it's now a standard nowadays, it's easier for companies and teams to get telemetry signals. I think that will be a game changer because this will mean that more and more teams will start. to use production signals to help them design their ⁓ performance testing. And it's not just limited to performance testing. So, you know, if you're using ⁓ like front-end performance signals, then you can also, you know, write like better synthetic checks that run, I don't know, ⁓ regularly so that at least your synthetic checks ⁓ also cover what are the most important user journey that your production users are. are doing. I think yeah, I have a good feeling that it's gonna start to ⁓ be more talked about. I think yeah, the more we spread word about the sustainability-driven development, its benefits, and I guess just to show as well how it can be done rather than talk about the theory, because maybe people are like, okay, I get this, but what tools are available here for me? I think that would be ⁓ awesome if, if that can be done at different conferences or ⁓ online talks, things like that. yeah, word of mouth.
speaker-1: Yeah, no, I'm definitely doing it. ⁓ I've been playing with observability for a while and I was surprised by again, it was the maturity of it. It's that kind of ⁓ I read
extensively, trying to keep up with everything. and like I find it interesting to see what people are doing all the time and then it's like, Huh, why is no one doing talking about this? And it's like, Well, you know, 'cause to me it feel felt natural a natural next step. ⁓ But it's ⁓ it's kind of sometimes that natural next step. People just need to go. I had that thought too, but I didn't like you said, I don't know how to get started. So hopefully, ⁓ hopefully if people see me, I've got two talks of on it later this year. ⁓ and I'm happy to do more discussions with other people. Also ⁓ so we've reached the point of the kind of podcast where we do a quick fire round and the answers need to be short and succinct. mostly I I've got it because I like to waffle and so I'm trying to make sure that other people don't waffle. ⁓ right. Are you ready?
speaker-0: Yeah. I'm all set, yeah. Okay.
speaker-1: ⁓ what's your favorite K6 feature that people
speaker-0: Definitely K6 browser because not a lot of people know that K6 can also measure front-end performance and something that I like to talk to a lot of people is this concept of hybrid approach to performance testing because we focus a lot on doing load testing, testing the protocol level, but that means we're ignoring the user experience. So what K6 browser allows you to do and because this is part of the core K6 OSS, ⁓ ⁓ you know, packages, it brings ⁓ browser testing as part of your overall performance testing. And that means rather than having a lot of thousands of browser virtual users, you can actually do this hybrid approach where you have, let's say, one or two browsers while your load test is being exposed to a thousands of virtual users and then see how that impacts one another. And it's all in the same tool. That's why it's my favorite because if there's a one tool that can... do one thing yeah then I'll go for that.
speaker-1: Awesome. ⁓ one book every QA lead should read in twenty twenty six.
speaker-0: I'm gonna be really biased but please read Contract Testing in action. It's the first book dedicated to contract testing and Luis and I spent a lot of time trying to make sure that it also covers a lot of different systems whether you're doing web, mobile, messaging service. So yeah, have a look at it.
speaker-1: Awesome. ⁓ if you weren't working in tech, what would you be doing now?
speaker-0: I would love to be a photographer so I really enjoy photography. did my brother's wedding a couple of years and years ago. ⁓ So ⁓ if I'm not working, I would love to do that. yeah, doesn't really... I have a lot of other priorities but yeah, I would love to do that as a full-time.
speaker-1: Awesome. ⁓ what is going to be your next crochet project?
speaker-0: So I'm trying to finish this. So in Grafano, we have this mascot called Grot and it looks like this dinosaur. So I'm trying to finish that and I've done like the body and I can see that it's, you know, forming. ⁓ But I still have tons to do. So I still have to do the hands, the legs and the actual, you know, dinosaur like tail. So it's still gonna take me some time and with kids indoors and I have a really young. toddler. ⁓ Every time I try to reach the crochet, he's like, nope, we're not gonna do it, mom. So I'm hoping to finish that soon.
speaker-1: Awesome. ⁓ Marie, thank you so much for your time today. This has been s awesome. I've learnt a lot. Hopefully everyone else has learnt a lot. your way of making kind of the ideas that are actually really complex, not so complex, has been really good. And thank you for sharing your expertise today. Where where can people ⁓ see more of what you're doing, learn from you? Where can they find that?
speaker-0: Yeah, I am on LinkedIn, so just connect with me, ⁓ Marie Cruz. ⁓ If you want to read stuff that I've done, see events that I've spoken at, I've added it to my personal blog, testingwithmarie.com. ⁓ And it tells you as well what events I'm doing next. But LinkedIn is probably the best, yeah.
speaker-1: Amazing. ⁓ so please do follow Mary. you will definitely be learning things. I've learned some things today. and if you've enjoyed this episode, please make sure to subscribe or follow on your favorite ⁓ platform ⁓ so you can get the next episodes. and I hope to see you soon. Thank you all for joining us today.
speaker-0: Thank you so much. Thank you.