NOEL: Hello and welcome to Episode 72 of the Tech Done Right Podcast, Table XI's podcast about building better software, careers, companies, and communities. I'm Noel Rappin.
I'm going to keep the intro short and sweet this time. I think I set up the situation in the show itself. Thanks for listening and if you listen to the end, there'll be a brief announcement about the show.
So, here's the setup we don't normally do "previously on" on this program. But previously on, we had Dave and Andy on Episode 68 and we spent a long time talking about how Dave Thomas had stopped testing for a while to determine what would happen. And what actually happened was, as far as he could tell, nothing. It didn't really change the code that he was writing at all, he felt. And what he took from that was that having spent 30 years thinking about testing, he had internalized the design requirements that come from TDD. In the episode after that, which is Episode 69, for those of you playing along at home, I had a long conversation with Sandi Metz. We talked about testing to some extent. We talked about Dave's comments, we talked about teaching testing and we both kind of agreed and sort of talked around the idea that those of us who teach testing were not necessarily doing a great job of explaining why testing was a good thing or why people should test. And that as a partial result, Rails developers and a lot of Ruby on Rails code is sort of poorly tested or/and possibly poorly designed. I think Sandi would argue that there's a higher order case of teaching design, which is a problem. So, with the secondary prompt that I am actually, as we record this teaching, a testing and design workshop for a half day for a bunch of relatively junior developers next week, and I'm wondering what do you think I should tell them? Because in response to that, I literally basically sent up the bat signal and got three multiple-time guests on this program who are now all going to introduce themselves. And Betsy, you can start.
BETSY: [Laughs] Hi, I'm Betsy. I am the Co-founder of Cohere, a software education and management training agency. You can check us out at wecohere.com. I also currently work for the LTSE, which stands for Long-Term Stock Exchange and it is exactly what it says on the tin.
NOEL: And Avdi?
AVDI: Hi, I am Avdi Grimm and I run Ruby Tapas, which is a Ruby education screencast service, and do various other things.
PENELOPE: Hi.
NOEL: And...yup. Go ahead.
PENELOPE: Hi. I'm Penelope Phippen, I'm a lead maintainer of the RSpec testing framework and I'm also currently working on a Ruby auto formatter called Ruby Fmt.
NOEL: Okay. Now, what do I tell these poor people who agreed to come listen to me for a couple of hours, talk about testing and design and stuff?
BETSY: One of the things that I immediately thought of when you were talking about Dave and Andy's ideas is something Katrina Owen said when she was promoting 99 Bottles of OOP, which is that she and Sandi were talking about Martin Fowler's Refactoring book. And Katrina kept on insisting that particular techniques and processes and et cetera were things that were in the Refactoring book because she had learned from it and they were things they did and they were things that she did. However, they weren't actually directly in the Refactoring book at all. They were saying that Katrina had kind of folk wisdom added to processes that she had originally derived from that book and that she just kept on iterating on and iterating on in ways where she ascribed them to Fowler. But they were actually entirely her original work by that point. I think about that a lot when I think about how we think about TDD and what it gives us. A lot of us, testing is something where we learned a lot of lessons about design from it. And so we axiomatically think that those design lessons have to have come from testing, but I'm not entirely sure that that's true. I think that might be attribution bias.
PENELOPE: I think that's great. I'm inclined to agree with everything that Betsy just said. One of the things I've sort of found in my own practice is that like I can sort of do what you might describe as the academic TDD. And then that leads to some of the same things that Katrina shows falling out in the book where you have everything dependency injected and all the objects are really tiny and like they are almost functions like that not really even doing what we might think as OOP-y stuff. They end up more looking like pure functions. But on a mostly daily software engineering basis, I don't do that. And the reason I found is that a lot of people will cargo-cult what I'm doing but not actually do the thing as prescribed. Noel, you and I have gone back and forth on this a little bit about there are a whole set of practices that are almost TDD but aren't TDD and that those leave you in a much worse state than if you had just written code on your own without any support from tests, whatsoever. And I have found that without very, very careful guidance, everyone falls somewhere into that space.
NOEL: Yeah. To elaborate on that, yeah, there's a lot of things that I see in people's code or in code that I wind up doing where you have either like a very long test or a very long series of end to end tests that is written sort of around the code. And while it's sort of TDD, it doesn't have the baby step, small tests, small piece of code piece that I've always found to be the most valuable part of the TDD process. Instead, it's a large test and a large chunk of code to make it pass. And that can give you a setup where you have tests and you have the burden of maintaining a test suite. But test suites are not giving you confidence in making changes. So in some sense, you're carrying more code but you're not gaining any confidence about whether you can make a change safely. So that's one of the things that I see. And the prevalence of people falling into that kind of process is the sort of thing that makes me think, and this is another thing that we were kind of going back and forth on, was if I try to teach one person TDD and I fail, that could be on them. If I try to teach a room full of people TDD and they all fail, that's possibly on me. If we all try to teach roomfuls of people TDD and we all fail, that might be on the idea. It might just not fit with what we're trying to do with it or it might, I don't know, all kinds of stuff. I don't really think I have a great answer to that. Avdi, what are you thinking?
AVDI: One thing I've noticed is that we've kind of been using the word testing and then the term TDD interchangeably.
NOEL: Right.
AVDI: So I think this is a thing that has happened in the past, maybe 10 years as a result of the growth in popularity of, I guess, basically the ubiquity of like the extreme programming practice which test driven development was a core part of that. And when we talk about testing, a lot of times what we're really talking about is automated developer testing. And then TDD, I would say, is even in a subset of automated developer testing where you write the test before you actually write the code that you're going to be testing. And that matters to me because I'm not just trying to like get into semantics. But the other day, I was facilitating a group doing a mob programming session and we kind of figured out what we wanted to implement. And so, they dove into the next step to implement that code. And I was like, "Wait a second, hang on. There's nothing broken right now. And that's a problem. There is nothing that is clearly breaking and that will change as we write this code." And the answer to that turned out -- at various points in development process -- the answer to that turned out to be a particular command line, a rake command line invocation that was failing, and that as we wrote more code, failed differently and finally stopped failing as we pressed up arrow and enter over and over again. And in another case, it was a page on their site that we hit the reload button on and it slowly showed more and more stuff. And then eventually was doing the thing that we wanted it to do. And that is testing - fooling around at a REPL is also a testing. And it's not automated testing, but it is testing. And to me, a big part of why TDD was important was just instilling the testing mindset. And that to me as a testing mindset is just the idea that we don't go too long without some kind of feedback from the code, some way for the code to talk to us and say, "Here's what I'm doing now." And that can be a lot of different things. It can be a repeated command, it can be better logging. But something to give us quick feedback. And it's not always TDD, it's not always writing automated developer unit tests beforehand.
NOEL: Right. And on the other side of that, there are all sorts of things we call testing that are not really developer testing, that are regression testing or the kind of things like a QA. I work with a fantastic QA engineer and he is continually pushing me to write more tests because I've written tests to make me comfortable from a developer perspective, but he sees holes from a QA perspective and he's almost always right. There are more expensive tests than I would consider writing from a strictly TDD perspective, but from his perspective, they're almost always something that covers a case that's not covered. To me, that confusion of what is a test and what is an automated testing, and what is TDD is certainly confusion that I see in conversations about this.
BETSY: And what's interesting to me about that is that on the same token, when I'm teaching TDD to newer developers, the thing that I always have to cozy them into is yes, it is okay to throw tests away, it is okay to write a test that gives you the immediate impact on something you want to get to the next step. But that test can have been useful for that moment and not be useful for literally any moment after. And just because you've written the test doesn't mean you need to keep it.
PENELOPE: Yeah, I couldn't agree more. In my teaching, I've never been able to convince anyone, although I'd be interested to hear from the rest of you, is that it's totally okay to delete tests. And sometimes tests have negative value. You can have a unit test or an integration test that is actually hurting you way more than it's helping you and you can delete it. But I find that most teams and particularly Rails teams don't have anywhere near the degree of confidence in their applications to be able to throw any of that away because they don't know what does and doesn't have value.
BETSY: I think a lot of that is that you can't have that idea of what does and doesn't have value until you have good instincts around coupling, and what kinds of couplings are useful or acceptable, and what kinds of coupling are just going to hurt you in the long run.
PENELOPE: Yup.
NOEL: Yeah. And I think that gets to the higher order point of where the object oriented and the test driven stuff sort of intersect, which is how and whether we teach design and that sort of design instinct, and whether there's any way to teach it other than writing a lot of programs and seeing how they fail over time. Another thing that I see people push back on in teaching TDD is the concept of writing what Gary Bernhardt called "sliming" the test, putting just the constant value and to make the immediate test passed even though it's obviously not the final value. I have been in cases where I have been teaching TDD and I say like, "Yeah and we'll just make this test pass by having the method return three," or whatever. And people just start laughing, like they are very resistant to the idea that you would put that temporary code in even if nobody would see it. There's a certain sense, I think especially among entry-level developers, that if it's not perfectly polished all the way through, they don't want to even type it.
BETSY: For me that gets to another thing that happens all the time with entry level and sometimes even late entry level and early mid developers, which is they have a profound discomfort with methods that just return a constant value no matter what. And in my opinion, if you're decomposing code properly and using polymorphism appropriately, you're going to wind up in a case where you have a method that just returns a static value so that you can maintain an interface, especially if you're writing Ruby.
PENELOPE: I send people to Avdi's "prefer methods instead of constants" talk all the time because I like to do it that way.
BETSY: I think at some point, some of it is that they have this definition of a program in their head and a program executing output, and that means you need to write an algorithm which is something that is much more complicated than just returning a static value.
NOEL: Another thing that Sandi asked me, which I actually surprisingly did not have a great answer for, was how and why did I learn TDD? And I guess I would throw that question out to you. How did you learn TDD? Why did you learn it and was it immediately apparent to you that it was a valuable technique?
BETSY: I think I learned TDD for a way that is actually, according to my friend Natalie, incredibly traditional, which was I had an anxiety disorder, the computer freaked me out and it gave me a way of being less freaked out by the computer. There is something about the routine of getting fast feedback is mentally useful for me. I think that that's true of a lot of people. I think that that is a legitimate purpose for certain forms of test driven development. And I think the tension lies in the fact that since it's not okay for us to talk about that as legitimate purpose, we make up all kinds of technical reasons why it has to be better to justify something that is better for us in terms of our work process.
NOEL: Did you find it hard when you first learned it? Did you find it valuable when you first learned it?
BETSY: I found it complicated when I first learned it in large part because I was implementing it as a very confident mid-level developer who had gotten senior put into her title for reasons that surpasseth understanding. The reasons were that I was the only developer on the project and they needed a way to keep me, and dealing with this horrible legacy code base and trying to get any kind of control over it whatsoever. And so, I had an extremely unpleasant experience as one often does trying to impose testing on a code base that has grown up absent the influence of any kind of testing whatsoever. But it was still useful for me even though it was extremely slow first.
NOEL: Avdi, how did you learn TDD?
AVDI: I was just thinking about that. I think a lot of the stuff that I learned early on was from kind of browsing the Wikiwiki, the Ward Cunningham's original Wiki.
NOEL: Yeah.
AVDI: Where he and a bunch of other software people were kind of discussing new ways of doing software. And I think I learned it from there or the concept from there. At the time, I was working at a really, really big stodgy company where that kind of thing was just completely off the radar. And I was tasked at one point with implementing a networking protocol from scratch. It was a known protocol.
BETSY: [Laughs]
AVDI: It was a well-documented protocol with a spec, but I needed to implement our own processor for it in C++, which is where all the work was being done in. And I was like, "Okay, I'm going to try this TDD thing." So it's a little bit different from the traditional TDD process because it wasn't like discovering the problem as well as the solution. It was really just discovering the solution. The problem was very well laid down in this example. So, it was probably, in some ways, a nice way to learn. But I was off in a corner, it was just my project. Nobody else is working with me. And I had this spec and I found it a simple like unit test library for C++ that somebody had made. And I just went through the spec, adding a test and making it pass, adding a test, making it pass until I'd gone through the whole spec. And I was like, "Okay, I think I've covered every case that this spec talks about. And we plugged it into the big, the main system, the rest of the system like integrated with the rest of the system is was supposed to be integrated with. And it basically worked the first time, which was unheard of in that environment. And there was like one memory leak or something, which is not something that TDD will get you away from. And yeah, at that point I was like, "Okay, I'm sold." It was nice, it was a very comfortable incremental process. There wasn't a lot of like long periods of time where I wasn't sure if it was going to work or not. It was very much write a test, make it pass, write a test and make it pass all the way up to the end and then it worked. So, I guess that was my first experience with it.
NOEL: That was, I think, kind of similar to my first experience and that I was working more or less on my own and had, I think, kind of a similar experience or something working very close to the first try because I had done the TDD process. Penelope, how did you learn it?
PENELOPE: That's a good question. I think it sort of happens in huge chunks with like a large gap between them. When I was a nascent college student, I think that's when a bunch of the Ruby community, TDD stuff was coming online and there were a bunch of conference talks, some I think by folks on this call that I was starting to watch to try and get a handle on it. And then simultaneously while doing that, I started contributing to the RSpec project. And most of RSpec is not written in a TDD fashion because it's a large gnarly legacy code base and you couldn't, you just couldn't. And then when I graduated college, the company that I joined was like, "We're going to do front to back TDD on every single piece of code we build." It was a Ruby application but it wasn't Rails. It was like an architecture that they'd sort of grown from scratch. At one point, the lead of engineering for this company was like, "I'm trying to force myself to write a test that requires me to change the back end from being in memory to Redis." And I was like, "Well, the only way to do that is to kill and restart the process. But if you really want a test that does that, -- I'm sorry". It was like a very, very extreme. Like we're going to do all TDD all the time environment and we were also doing all pairing all the time. And so, I learned it from people who had been doing it for much, much longer.
In terms of whether I think it was useful, I was actually very resistant to it being useful or faster than taking larger steps and throwing more tests away. I think one of the things we did very wrong in that environment is not throwing enough tests away. And when you do that very pure step at a time TDD, you end up with a bunch of tests that basically completely couple your architecture together and we had to do a big refactoring. And suddenly, all the tests were red and it was like, "Do we fix these one at a time? Do we refactor by breaking and fixing? How do we do this?" Now, I would have been like, "Let's just blow away all the unit tests and as long as these integration tests pass, start again." So yeah, that's my summary.
NOEL: One of the things that I think comes out of this was another thought that I had about TDD and testing in general is that TDD is not a robust process in the sense that a system that has not been written using TDD is very hard to bring under TDD. And even a system that starts using TDD, once you start sort of sliding away from it, it becomes very hard to recover. And I wonder if that makes sense to the assembled panel. And if you think that that's part of why TDD is so hard to transmit to other teams or so often just not useful in practice, or maybe it's just not useful in practice at scale.
PENELOPE: One of the things I've seen is that if you're doing it right, you're doing dependency injection literally everywhere. And this is something Sandi brought up repeatedly in the episode that she was in.
NOEL: What's weird about that is I almost never do dependency injection. So, I kind of push back against that.
PENELOPE: Same
BETSY: I think part of it is that I agree that generally it means doing dependency injection literally everywhere. I think that it doesn't necessarily mean that, but I think that even when it doesn't, the ways that doesn't tend to involve either really useful strategies for managing stubs or ways of setting up your code base so that you don't mean to rely on stubbing to unit test. And both of those are not transmittable via copy and paste.
PENELOPE: Right. Actually, you kind of made my next point for me. I was winding up to that, which is that it led to a propagation of tools that let you like blow holes in your isolation boundaries in order to make testing easier. And like I will put my hat off and say RSpec mocks is like the isolation boundary destroyer. It's basically what it's designed for in principle.
BETSY: And that's difficult because I have a whole talk on this. When you do that, you're learning the wrong thing from the tool. The tool is pointing you in the wrong direction. It's a power users tool. But people go, "Oh, this is advanced stuff, therefore I must use it to be advanced." And they completely shoot themselves in the foot.
PENELOPE: One of the things I've always sort of found is that it makes all the sort of forms of this equivalently easy. So like passing a stub in is made as easy as replacing the implementation of a method which was made as easy as stubbing out an entire class, which was made as easy as using the any instance. There's four things that are equivalently easy with RSpec. So the tool provides you no feedback at all about the idea that you're sort of like going down a path which may indicate that you have a worse design. I've spent years being like, "Please don't do this. I know you can do this. I know it seems like this is what you should be doing, but if we do it this way, then our system will be easier to work with."
NOEL: So, what's the 'this way and then our system will be easier to work with'?
PENELOPE: I guess like anytime I see an any instance or a class stub, I'm immediately like, "We have a serious design issue here that we probably need to address." Unless we're in a legacy code base and we're like just trying to shore up some safeties around something. If this is fresh code, I will be like, "Please, can we find a different way to do this?"
BETSY: I will confess to have used a class stub in a greenfield code base earlier today without any regrets.
[Laughter]
PENELOPE: Can I ask, is that a Rails code base?
BETSY: Yes. But it's actually not the Rails bits. GitHub's GraphQL library is weird. [Laughs]
NOEL: I actually think it's a significant issue here that none of the frameworks that we sort of regularly use on the review side are like built to do TDD. Rails is built to do testing in a very smooth and profound and very valuable way and it was certainly one of the things that drew me to it initially. But it is not really built to do unit testing. You can't really test models without bringing in the entire database. It's nearly impossible to test associations without bringing in the entire database, still at this point. And I think that that also, and Justin's not here, but Justin would say because of Rails, the need to do testing to discover your domain is less because Rails provides such strong pressure on the domain. I think that's what Justin would say. He'd say a lot of things probably, that's one of them.
BETSY: I have a salty and controversial opinion for a minute.
NOEL: No, we don't want to have any salty controversial opinion. Why would we want that?
[Laughter]
PENELOPE: We didn't invite Justin, so salt's not allowed on this podcast.
NOEL: I did invite Justin. He is like 400 hours off of our time zone.
BETSY: The conventional unit test is that they're impossible to truly do because you have to pull in the database and blah, blah blah.
NOEL: Yes.
BETSY: But for me, I would argue that the database is just part of the unit in a Rails model test if you're writing Rails correctly. And so, you should just suck it up and deal.
NOEL: I mean that's basically what I do in practice, right? I think that Rails, yes, within the Rails universe, the database is part of your Active Record model.
PENELOPE: That's a really interesting way of looking at it because like just from a completely different domain, I was once in a distribution systems architecture meeting where they were like, "Cool, so you have your Rails app and you have your database and these are logically separate components in our distributed system." And I was like, "No, there's no way for us to keep that Rails app online if the database isn't there." Or they were like, "Have it return a health check that doesn't require the data." I was like, "The controller will raise an exception regardless of what I put in it if the database isn't there." Like, no, these are all one box. And so, I'm just sort of inclined to agree with Betsy there that like yeah, that makes a lot of sense.
BETSY: That's why I had been spending all that time recently on figuring out ways to make in memory data happen in Active Record tests. I think that we're not going to be able to convince people to stop doing horrible things with stubs to mock out their database in Active Record tests unless we can make that a faster process. And I don't actually think that the problem is with the Rails architecture here. I think it's with us needing test speed and doing horrible things.
NOEL: I do think it's weird and have long thought is weird that nobody has come along and built like in memory Active Record clone for the purpose of testing that lets you in memory manage associations as though they were in a database. And I don't think it's a library I've ever seen and I've never actually been bothered to write, but it's always been a surprise to me that somebody has not gone out and done it.
BETSY: It's because it's a viciously hard problem that involves writing a database from scratch. I had been working on it all summer. [laughs]
NOEL: Hence, the "I've never been bothered to write it myself".
BETSY: [Laughs]
NOEL: There are a lot of questions here, like in practice, is TDD really dead? Like is it worth practicing? Is it worth teaching to Rails developers? Is there a valuable piece of technique or a valuable thought process there at this point?
AVDI: I think the elephant in the room is that TDD, especially once we start talking about like mocks and stubs and things like that as well, once that enters the conversation, TDD is an exercise. It's a way of designing APIs, or very often is.
NOEL: In the larger sense, not just in like the server sense.
AVDI: Yeah. It's determining what your code surface is going to be to your code, to other people's code. It's determining what the interface is going to look like. And it's an exercise and it's a really interesting exercise in experimenting with, what does the usage look like when I design my API this way, when I design my methods this way, when I break the problem down into this set of classes versus that set of classes? The tests say a lot about that you have to write for that say a lot about how that API is, how that interfaces to use. What a lot of TDD advocates will say and unit testing advocates in general will say is that the point of it all is that your tests are the first consumer of your API. They are the original consumer and they give you kind of a sense of what it looks like to consume that API. And API could just mean the three methods that are on the class that you're writing right now. We're not talking about like large scale APIs here, but they give you a fingerprint of what it looks like to use the code that you're writing. And so, when I hear that Dave Thomas throw all his tests away and he and I have talked about this too, when I hear that he threw all his tests away or unit tests or whatever and still like doesn't see a major difference, well I also happened to know that Dave Thomas has an internalized and enormous amount of good taste about interface design over the years. And so, that doesn't surprise me at all because Dave Thomas, he can see in his head what it's going to be like to use the interface that he's creating. And likewise with mocks and stubs, originally that was really supposed to be about, let's see what it's going to look like to use, what is the API that we are going to need even though we don't have that class yet. This is a drum I've been banging for years and years, then I've realized that it's kind of a lost cause at this point that we were never supposed to use. Well, okay. A few years after the creators of mock objects invented them, they looked at their experiences and wrote up a paper that said, "Okay, what we have found is that they're great for simulating objects that don't exist yet, that we need to write because we find out what API those objects are going to need. And they are terrible for isolating the system from already existing other code." They're not so great for like pretending that the database doesn't exist and stuff like that, or pretending that your external API that you're supposed to talk to doesn't exist. They're not great for test performance hacks, but that's basically the only thing that people use them for anymore.
PENELOPE: Avdi, something you said that I'd like to go back in on, and it was right at the beginning where you sort of conceptualized that when someone is exercising, like going through the TDD exercises, they're sort of testing out multiple versions of how they might approach the same problem or editing the interfaces on their objects and that sort of thing. And I think one of the things I've seen is that even up to sort of very experienced mid-level developers, they will just do the TDD, churn through the TDD process and then stop and never actually do that sort of rearchitecting and designing. And even they'll do the refactor step because they'd been told to, but it'd be like, "I'm going to make this method smaller and make a private method," or something like that and never refactor as in move the whole system around. And I think one of the, and this is a question to the panel is like do we think that we've done a bad job of explaining what the refactor step is for? Because I think they'll write a failing test and the make the test pass steps are fairly self explanatory and the refactor step is not.
AVDI: I think there's a missing step because when we talk about the refactor step, we're very explicitly talking about their, according to the definition of refactoring is we take our tests exactly as they are and we muck around with the internals to make them prettier. And that's a really important step because you can write some really ugly code that makes the tests pass. But there is another step which we don't talk about, which is looking at the tests and saying, "Okay, all this works now and we refactored the internals, but this test is telling me that this interface is a pain to use." And at that point, you can't just refactor in a traditional sense. You have to throw the tests away and the code under test and you have to write new tests that are like, "Okay, this is the way I'd really like to use it. This is how I'd like it to work. And now let's make that work."
NOEL: That sounds hard, is the problem.
[Laughter]
BETSY: Being real about why you skip that though. We skip that because two reasons. One of them is that you're typically not inviting earlier career developers, to incorporate slack into their process explicitly. We're doing reverse. We're judging them on how fast they can have the output of a "real developer" because we don't really value them yet. We just view them as bodies and so they feel pressured to keep up by not actually teaching themselves properly. And the other reason is that we train them TDD as a dogmatic process. And that's the thing I think that actually needs to go, not TDD itself, but dogmatism around versions of TDD because dogmatism about versions of TDD discourages people from taking the time to reflect on their own code because they're not in a reflective state of mind. They're in a 'I will execute this mechanistic process' state of mind.
PENELOPE: I completely agree. It's interesting actually. I think Noel originally framed the question 10 minutes ago as like, are we still practicing TDD? I found myself making these much larger steps usually and just being like I could write the five tests that would imply this test or I could just write this test because I know I already like this interface because it's showing me the whole thing in one go. And I think that if you're newer you can't do that, you're looking for anything that lets you sort of not feel scared that something's gone horribly, as Betsy you were saying earlier that something's gone horribly wrong all the time. And I think we kind of hand over TDD and go, "Here you go. Do this." But there's no room to be like, "And this thing is not perfect and here's all the damages done to all of the billions of dollars worth of code bases and that you should find a way to make this work for you."
NOEL: One thing that struck me when I first started learning TDD and has always been a reason why I do it is that TDD reduces my short term cognitive load. It reduces the number of things I need to think about at one time by focusing my efforts on making the current test pass. And especially when I was first learning it, I found that very, very valuable. And I think over time, kind of as you were implying, that's less valuable as I've gotten better and better at keeping large chunks of code in my head at a time, and I've always kind of assumed that DHH, that a lot of Rails designs decisions make a lot of sense if you think of DHH as somebody who is like extremely good at keeping very large amounts of code in his head. A lot of the concerns about like the surface area of Active Record objects and then not having TDD, like all of that is consistent with a design philosophy where you're comfortable keeping a lot of stuff in your head. Does that make sense? Does that imply anything about how we teach this or should teach this?
BETSY: Yes. And I do think that optimizing for a team, because team is always the, even if it's a one person team, a team is always the appropriate unit for understanding a code base. Optimizing for a team that can keep a lot of code in their head at once is a reasonable choice for an organization to make. It's going to be a limiting choice in most cases because it means that you have to have a hiring strategy of a small number of experienced people who stay on the project for literally years. And that's not something the average organization is going to find sustainable, but it's a viable strategy.
AVDI: Yeah. Betsy, what you were saying reminds me that Jessica Kerr has a thing that she says about 10X developers, which is that the 10X developer is not somebody who's better at coding or smarter. The person that seems like the 10X developer on a project is the one that's been there since the project was incepted and just has a mental model of the whole thing in their head. That's not an inherent property. It's a property of context.
BETSY: Oh yeah. And I've worked on the team I just described. I was the new girl on that team and I had a very poor experience but it was in many ways a perfectly high functioning team for everyone else on it.
AVDI: Yeah.
NOEL: I have not spent most of my career on long lived teams. I've spent a lot of my career in small and short consulting projects where the teams turn over a lot. And I think that might affect the way that I approach the testing and design of code because I've never been in a situation where I've had the expectation that there is going to be anybody who has a complete mental model of the system like that.
PENELOPE: I was reflecting on my experiences. I joined this company that had been TDDing everything from scratch from day one. And in the sense that everyone on that team had enough experience that they could be trusted to write code independently without any help whatsoever. I think it actually sort of caused a convergence of people who had been around for a long time and people who were new, were delivering at not wildly varying rates. But that the reason that was true is that I think the actual average of new features being delivered was a lot lower than nearly every other team I've ever worked on. But they were incredibly precise because of all of the tests we were writing. We had three to one, four to one test to code ratio easily.
NOEL: Another thing that I sometimes think about here is sometimes when we talk about metrics or empirical studies of code, and there's actually been empirical studies of TDD and testing and all this stuff, and they tend to be terrible in terms of the design of the studies and they haven't shown a really strong effect. What would convince you that TDD was a bad idea and maybe had always been a bad idea, if anything? What would convince you to stop doing even the amount of it that you are doing now?
PENELOPE: Oh, I'm already there.
NOEL: Okay.
[Laughter]
NOEL: That's one person who has fallen off the train. Okay. So, what convinced you?
PENELOPE: I stopped doing it. Firstly, I didn't feel like my code was substantially worse or better or less reliable or more reliable. But because I have been an authority figure in software testing for essentially my entire career because I've been an Rspec maintainer since I graduated college, people stopped cargo-culting the tests I was writing. The very precise small tests and then doing it in a way where it wasn't actually TDD and ending up with tests that were worse than having larger ones or the ones they didn't implement if they were doing it post hoc.
BETSY: I am also in the 'I'm not currently actually doing a lot of TDD train' that's because the project I'm on right now is a really spiky exploratory year in project. So that's kind of a special case. And I do think that some of the code that I produced in the context of this project has been code that would have been coded differently if I'd used TDD methods. And I do think that some of it is in some ways worse than the code I would have been writing if I'd been test driving. It's a spike, though.
PENELOPE: Can I ask you a question?
BETSY: Yeah.
PENELOPE: Would you think it would be easy for you to get it to a state where it was not like sort of value judgment was and/or that you could retrofit tests on it to improve the design?
BETSY: Yes, but I specifically held myself back from a decision to make that easier early this morning. It was the near thing. I like over fitted to something and then I went, "No, Bad Betsy, slaps self on hand."
PENELOPE: Yeah. The reason I ask that is I think that people who have done it a lot know where to avoid those landmines easily.
BETSY: Yeah. I think that that's reasonable. One thing I do think, kind of riffing off of your question Noel, the evidence that caused me to have a much more complicated position on TDD than I used to earlier in my career was seeing the tests in a lot of modern JavaScript projects, and seeing frameworks in a lot of modern JavaScript projects because until that point, I've largely been working in a Rails context where one is kind of agreed to cross their fingers and say good enough in terms of test isolation because that's all you can really do. But the culture in React particularly but increasingly also Vue is to go for hyper isolation of small pieces. And a lot of the tests that are produced that way feel like virtually useless reifications of something that you were going to do anyway. One of the reasons I'm so hardline about the database as part of a unit in a Rails test now is because I've seen what happens to frontend tests that don't treat the browser as part of the unit and I really don't like it.
NOEL: Yeah. I have a whole separate rant about the DOM.
PENELOPE: I think we all have a rant about this.
AVDI: I have trouble with productivity measures and things like that. Quality measures can be better, not always. Earlier I was talking about how I think TDD unit tests, all this stuff, they're exercises in interface design. And really what that is it's an instance of sense-making. Most of the software project is sense-making. I don't know that we have any kind of quantitative measure of sense-making or whether such a thing even makes sense. Like how well are we doing at sense-making on this project? I don't think that we're going to assign a number to that anytime soon. And so, when I think about the question of what stats would convince me to throw away TDD or unit testing or whatever. Well, I don't think they exist because I don't think that there's a measure of the sense-making portion.
NOEL: No. But there might be a subjective feeling you have on project after project.
AVDI: But I say that as somebody who is kind of moving fluidly between stuff where I do very TDD style design and where I just sort of feel like, "Okay, yeah. This clearly doesn't need a test," whatever. And that's the voice of experience to some degree.
PENELOPE: Yeah. I think if I was to ask myself the inverse question, what stats, metrics, et cetera, would get you back on the TDD train? I think it would be seeing outcomes in code bases worked on by non-experts improve, because the outcomes I've seen in every Rails app where they've tried to do TDD and like the most senior person learned TDD by word of mouth and isn't really in a position to be great at teaching it all the way down, which is nearly every Rails company on the planet. To me, it's just so obviously worse than if they had just done larger tests retrospectively and almost effectively, like mini integration tests that are integrating two or three components of the applications and the database to try something out. And so, if you could plausibly change that effect, which I think is one we've all seen, that's what would do it for me.
NOEL: Okay.
BETSY: For me, it comes back to the sense-making point Avdi was making, I use TDD as a tool for sense-making. I am not convinced that everyone else does. And I think that TDD not used for sense-making has had some really corrosive effects.
NOEL: I would say that it seems to me that a lot of people test without a theory of why they're testing and I think where we're getting at is that there are a number of different theories about why to test that might be effective, but where you really get in trouble is if you just throw tests at the problem without any real sense of why you're writing this test or what your options are, and what options you're choosing.
PENELOPE: And I think some of that could be faulted back to the rhetoric by which a TDD was introduced. Like, "You must do this. This is how you produce quality software. Software not done with this is bad software," which is slight hyperbole, but I don't feel like it's a complete mischaracterization of some of the ways people have presented it.
NOEL: What do I tell these people that are coming to my workshop next week?
AVDI: I mean, for my money, I think it's still very valuable to teach people this skill. There are a lot of skills that are valuable in software, ranging from big O notation, estimating performance of algorithms, to things like TDD. You don't use any of them all the time. But we've found this one to be often useful as an intellectual exercise or as a design exercise. And it's certainly a lot more immediate than a lot of the design exercises that came before it. So, there are days when I just draw a fricking blank about how to move forward on the thing that I need to write. And on those days, TDD has often gotten me out of that rut because it gave me a way to make very small steps and not think about big picture. And if even only for that reason, I think it's a useful tool for new programmers to learn because it's going to give them a tool in their toolbox to get them out of that rut and we all need that sometimes.
NOEL: Okay. I like that. We can leave it there. Where can people find you online if they want to rant at you about testing some more?
BETSY: I'm Betsy. You can find me on the internet @BetsyTheMuffin on Twitter, or you can find my company's Twitter @WeCohere.
AVDI: I'm Avdi. You can find me online at Avdi.codes, and I'm on Twitter as @Avdi.
PENELOPE: You can find me on twitter @penelopezone or you can checkout my blog at http://penelope.zone.
NOEL: Great. Thank you for being on this episode. I really appreciate it and it's great to talk to you. And thank you for being here.
AVDI: Thank you.
PENELOPE: Thank you.
BETSY: Thank you.
NOEL: Tech Done Right is available on the web at TechDoneRight.io where you can learn more about our guests and comment on our episodes. Find us wherever you listen to podcasts. And if you liked the show, tell a friend or your social media network, and leaving a review on Apple Podcasts helps people find the show.
Tech Done Right is hosted by me, Noel Rappin. Our editor is Mandy Moore. You can find us on Twitter @NoelRap and @theRubyRep.
Tech Done Right is produced by Table XI. Table XI is a custom design and software company in Chicago. We've been named one of Inc. Magazine's best workplaces and we are a top-rated custom software development company on clutch.co. You can learn more about working with us or working for us at TableXI.com or follow us on Twitter @TableXI.
And now, an update on the show. I don't know exactly when this episode will be released, but this will probably be my final episode of Tech Done Right. Whether or not it continues in the future, I don't know. If it does, I certainly wish it well and hope that it continues to produce great shows. I've really enjoyed doing this for the past 72 episodes. It's been a thrill and an opportunity to have some great conversations with some wonderful people in and around the technology space. I hope that you've enjoyed it and thank you to all the people who have listened and left feedback, or told me that they liked an episode. It's really been great and I really appreciate all of the time that people have put into listening. And I hope that you have enjoyed the show, too. So, thank you.
I'm going to keep the intro short and sweet this time. I think I set up the situation in the show itself. Thanks for listening and if you listen to the end, there'll be a brief announcement about the show.
So, here's the setup we don't normally do "previously on" on this program. But previously on, we had Dave and Andy on Episode 68 and we spent a long time talking about how Dave Thomas had stopped testing for a while to determine what would happen. And what actually happened was, as far as he could tell, nothing. It didn't really change the code that he was writing at all, he felt. And what he took from that was that having spent 30 years thinking about testing, he had internalized the design requirements that come from TDD. In the episode after that, which is Episode 69, for those of you playing along at home, I had a long conversation with Sandi Metz. We talked about testing to some extent. We talked about Dave's comments, we talked about teaching testing and we both kind of agreed and sort of talked around the idea that those of us who teach testing were not necessarily doing a great job of explaining why testing was a good thing or why people should test. And that as a partial result, Rails developers and a lot of Ruby on Rails code is sort of poorly tested or/and possibly poorly designed. I think Sandi would argue that there's a higher order case of teaching design, which is a problem. So, with the secondary prompt that I am actually, as we record this teaching, a testing and design workshop for a half day for a bunch of relatively junior developers next week, and I'm wondering what do you think I should tell them? Because in response to that, I literally basically sent up the bat signal and got three multiple-time guests on this program who are now all going to introduce themselves. And Betsy, you can start.
BETSY: [Laughs] Hi, I'm Betsy. I am the Co-founder of Cohere, a software education and management training agency. You can check us out at wecohere.com. I also currently work for the LTSE, which stands for Long-Term Stock Exchange and it is exactly what it says on the tin.
NOEL: And Avdi?
AVDI: Hi, I am Avdi Grimm and I run Ruby Tapas, which is a Ruby education screencast service, and do various other things.
PENELOPE: Hi.
NOEL: And...yup. Go ahead.
PENELOPE: Hi. I'm Penelope Phippen, I'm a lead maintainer of the RSpec testing framework and I'm also currently working on a Ruby auto formatter called Ruby Fmt.
NOEL: Okay. Now, what do I tell these poor people who agreed to come listen to me for a couple of hours, talk about testing and design and stuff?
BETSY: One of the things that I immediately thought of when you were talking about Dave and Andy's ideas is something Katrina Owen said when she was promoting 99 Bottles of OOP, which is that she and Sandi were talking about Martin Fowler's Refactoring book. And Katrina kept on insisting that particular techniques and processes and et cetera were things that were in the Refactoring book because she had learned from it and they were things they did and they were things that she did. However, they weren't actually directly in the Refactoring book at all. They were saying that Katrina had kind of folk wisdom added to processes that she had originally derived from that book and that she just kept on iterating on and iterating on in ways where she ascribed them to Fowler. But they were actually entirely her original work by that point. I think about that a lot when I think about how we think about TDD and what it gives us. A lot of us, testing is something where we learned a lot of lessons about design from it. And so we axiomatically think that those design lessons have to have come from testing, but I'm not entirely sure that that's true. I think that might be attribution bias.
PENELOPE: I think that's great. I'm inclined to agree with everything that Betsy just said. One of the things I've sort of found in my own practice is that like I can sort of do what you might describe as the academic TDD. And then that leads to some of the same things that Katrina shows falling out in the book where you have everything dependency injected and all the objects are really tiny and like they are almost functions like that not really even doing what we might think as OOP-y stuff. They end up more looking like pure functions. But on a mostly daily software engineering basis, I don't do that. And the reason I found is that a lot of people will cargo-cult what I'm doing but not actually do the thing as prescribed. Noel, you and I have gone back and forth on this a little bit about there are a whole set of practices that are almost TDD but aren't TDD and that those leave you in a much worse state than if you had just written code on your own without any support from tests, whatsoever. And I have found that without very, very careful guidance, everyone falls somewhere into that space.
NOEL: Yeah. To elaborate on that, yeah, there's a lot of things that I see in people's code or in code that I wind up doing where you have either like a very long test or a very long series of end to end tests that is written sort of around the code. And while it's sort of TDD, it doesn't have the baby step, small tests, small piece of code piece that I've always found to be the most valuable part of the TDD process. Instead, it's a large test and a large chunk of code to make it pass. And that can give you a setup where you have tests and you have the burden of maintaining a test suite. But test suites are not giving you confidence in making changes. So in some sense, you're carrying more code but you're not gaining any confidence about whether you can make a change safely. So that's one of the things that I see. And the prevalence of people falling into that kind of process is the sort of thing that makes me think, and this is another thing that we were kind of going back and forth on, was if I try to teach one person TDD and I fail, that could be on them. If I try to teach a room full of people TDD and they all fail, that's possibly on me. If we all try to teach roomfuls of people TDD and we all fail, that might be on the idea. It might just not fit with what we're trying to do with it or it might, I don't know, all kinds of stuff. I don't really think I have a great answer to that. Avdi, what are you thinking?
AVDI: One thing I've noticed is that we've kind of been using the word testing and then the term TDD interchangeably.
NOEL: Right.
AVDI: So I think this is a thing that has happened in the past, maybe 10 years as a result of the growth in popularity of, I guess, basically the ubiquity of like the extreme programming practice which test driven development was a core part of that. And when we talk about testing, a lot of times what we're really talking about is automated developer testing. And then TDD, I would say, is even in a subset of automated developer testing where you write the test before you actually write the code that you're going to be testing. And that matters to me because I'm not just trying to like get into semantics. But the other day, I was facilitating a group doing a mob programming session and we kind of figured out what we wanted to implement. And so, they dove into the next step to implement that code. And I was like, "Wait a second, hang on. There's nothing broken right now. And that's a problem. There is nothing that is clearly breaking and that will change as we write this code." And the answer to that turned out -- at various points in development process -- the answer to that turned out to be a particular command line, a rake command line invocation that was failing, and that as we wrote more code, failed differently and finally stopped failing as we pressed up arrow and enter over and over again. And in another case, it was a page on their site that we hit the reload button on and it slowly showed more and more stuff. And then eventually was doing the thing that we wanted it to do. And that is testing - fooling around at a REPL is also a testing. And it's not automated testing, but it is testing. And to me, a big part of why TDD was important was just instilling the testing mindset. And that to me as a testing mindset is just the idea that we don't go too long without some kind of feedback from the code, some way for the code to talk to us and say, "Here's what I'm doing now." And that can be a lot of different things. It can be a repeated command, it can be better logging. But something to give us quick feedback. And it's not always TDD, it's not always writing automated developer unit tests beforehand.
NOEL: Right. And on the other side of that, there are all sorts of things we call testing that are not really developer testing, that are regression testing or the kind of things like a QA. I work with a fantastic QA engineer and he is continually pushing me to write more tests because I've written tests to make me comfortable from a developer perspective, but he sees holes from a QA perspective and he's almost always right. There are more expensive tests than I would consider writing from a strictly TDD perspective, but from his perspective, they're almost always something that covers a case that's not covered. To me, that confusion of what is a test and what is an automated testing, and what is TDD is certainly confusion that I see in conversations about this.
BETSY: And what's interesting to me about that is that on the same token, when I'm teaching TDD to newer developers, the thing that I always have to cozy them into is yes, it is okay to throw tests away, it is okay to write a test that gives you the immediate impact on something you want to get to the next step. But that test can have been useful for that moment and not be useful for literally any moment after. And just because you've written the test doesn't mean you need to keep it.
PENELOPE: Yeah, I couldn't agree more. In my teaching, I've never been able to convince anyone, although I'd be interested to hear from the rest of you, is that it's totally okay to delete tests. And sometimes tests have negative value. You can have a unit test or an integration test that is actually hurting you way more than it's helping you and you can delete it. But I find that most teams and particularly Rails teams don't have anywhere near the degree of confidence in their applications to be able to throw any of that away because they don't know what does and doesn't have value.
BETSY: I think a lot of that is that you can't have that idea of what does and doesn't have value until you have good instincts around coupling, and what kinds of couplings are useful or acceptable, and what kinds of coupling are just going to hurt you in the long run.
PENELOPE: Yup.
NOEL: Yeah. And I think that gets to the higher order point of where the object oriented and the test driven stuff sort of intersect, which is how and whether we teach design and that sort of design instinct, and whether there's any way to teach it other than writing a lot of programs and seeing how they fail over time. Another thing that I see people push back on in teaching TDD is the concept of writing what Gary Bernhardt called "sliming" the test, putting just the constant value and to make the immediate test passed even though it's obviously not the final value. I have been in cases where I have been teaching TDD and I say like, "Yeah and we'll just make this test pass by having the method return three," or whatever. And people just start laughing, like they are very resistant to the idea that you would put that temporary code in even if nobody would see it. There's a certain sense, I think especially among entry-level developers, that if it's not perfectly polished all the way through, they don't want to even type it.
BETSY: For me that gets to another thing that happens all the time with entry level and sometimes even late entry level and early mid developers, which is they have a profound discomfort with methods that just return a constant value no matter what. And in my opinion, if you're decomposing code properly and using polymorphism appropriately, you're going to wind up in a case where you have a method that just returns a static value so that you can maintain an interface, especially if you're writing Ruby.
PENELOPE: I send people to Avdi's "prefer methods instead of constants" talk all the time because I like to do it that way.
BETSY: I think at some point, some of it is that they have this definition of a program in their head and a program executing output, and that means you need to write an algorithm which is something that is much more complicated than just returning a static value.
NOEL: Another thing that Sandi asked me, which I actually surprisingly did not have a great answer for, was how and why did I learn TDD? And I guess I would throw that question out to you. How did you learn TDD? Why did you learn it and was it immediately apparent to you that it was a valuable technique?
BETSY: I think I learned TDD for a way that is actually, according to my friend Natalie, incredibly traditional, which was I had an anxiety disorder, the computer freaked me out and it gave me a way of being less freaked out by the computer. There is something about the routine of getting fast feedback is mentally useful for me. I think that that's true of a lot of people. I think that that is a legitimate purpose for certain forms of test driven development. And I think the tension lies in the fact that since it's not okay for us to talk about that as legitimate purpose, we make up all kinds of technical reasons why it has to be better to justify something that is better for us in terms of our work process.
NOEL: Did you find it hard when you first learned it? Did you find it valuable when you first learned it?
BETSY: I found it complicated when I first learned it in large part because I was implementing it as a very confident mid-level developer who had gotten senior put into her title for reasons that surpasseth understanding. The reasons were that I was the only developer on the project and they needed a way to keep me, and dealing with this horrible legacy code base and trying to get any kind of control over it whatsoever. And so, I had an extremely unpleasant experience as one often does trying to impose testing on a code base that has grown up absent the influence of any kind of testing whatsoever. But it was still useful for me even though it was extremely slow first.
NOEL: Avdi, how did you learn TDD?
AVDI: I was just thinking about that. I think a lot of the stuff that I learned early on was from kind of browsing the Wikiwiki, the Ward Cunningham's original Wiki.
NOEL: Yeah.
AVDI: Where he and a bunch of other software people were kind of discussing new ways of doing software. And I think I learned it from there or the concept from there. At the time, I was working at a really, really big stodgy company where that kind of thing was just completely off the radar. And I was tasked at one point with implementing a networking protocol from scratch. It was a known protocol.
BETSY: [Laughs]
AVDI: It was a well-documented protocol with a spec, but I needed to implement our own processor for it in C++, which is where all the work was being done in. And I was like, "Okay, I'm going to try this TDD thing." So it's a little bit different from the traditional TDD process because it wasn't like discovering the problem as well as the solution. It was really just discovering the solution. The problem was very well laid down in this example. So, it was probably, in some ways, a nice way to learn. But I was off in a corner, it was just my project. Nobody else is working with me. And I had this spec and I found it a simple like unit test library for C++ that somebody had made. And I just went through the spec, adding a test and making it pass, adding a test, making it pass until I'd gone through the whole spec. And I was like, "Okay, I think I've covered every case that this spec talks about. And we plugged it into the big, the main system, the rest of the system like integrated with the rest of the system is was supposed to be integrated with. And it basically worked the first time, which was unheard of in that environment. And there was like one memory leak or something, which is not something that TDD will get you away from. And yeah, at that point I was like, "Okay, I'm sold." It was nice, it was a very comfortable incremental process. There wasn't a lot of like long periods of time where I wasn't sure if it was going to work or not. It was very much write a test, make it pass, write a test and make it pass all the way up to the end and then it worked. So, I guess that was my first experience with it.
NOEL: That was, I think, kind of similar to my first experience and that I was working more or less on my own and had, I think, kind of a similar experience or something working very close to the first try because I had done the TDD process. Penelope, how did you learn it?
PENELOPE: That's a good question. I think it sort of happens in huge chunks with like a large gap between them. When I was a nascent college student, I think that's when a bunch of the Ruby community, TDD stuff was coming online and there were a bunch of conference talks, some I think by folks on this call that I was starting to watch to try and get a handle on it. And then simultaneously while doing that, I started contributing to the RSpec project. And most of RSpec is not written in a TDD fashion because it's a large gnarly legacy code base and you couldn't, you just couldn't. And then when I graduated college, the company that I joined was like, "We're going to do front to back TDD on every single piece of code we build." It was a Ruby application but it wasn't Rails. It was like an architecture that they'd sort of grown from scratch. At one point, the lead of engineering for this company was like, "I'm trying to force myself to write a test that requires me to change the back end from being in memory to Redis." And I was like, "Well, the only way to do that is to kill and restart the process. But if you really want a test that does that, -- I'm sorry". It was like a very, very extreme. Like we're going to do all TDD all the time environment and we were also doing all pairing all the time. And so, I learned it from people who had been doing it for much, much longer.
In terms of whether I think it was useful, I was actually very resistant to it being useful or faster than taking larger steps and throwing more tests away. I think one of the things we did very wrong in that environment is not throwing enough tests away. And when you do that very pure step at a time TDD, you end up with a bunch of tests that basically completely couple your architecture together and we had to do a big refactoring. And suddenly, all the tests were red and it was like, "Do we fix these one at a time? Do we refactor by breaking and fixing? How do we do this?" Now, I would have been like, "Let's just blow away all the unit tests and as long as these integration tests pass, start again." So yeah, that's my summary.
NOEL: One of the things that I think comes out of this was another thought that I had about TDD and testing in general is that TDD is not a robust process in the sense that a system that has not been written using TDD is very hard to bring under TDD. And even a system that starts using TDD, once you start sort of sliding away from it, it becomes very hard to recover. And I wonder if that makes sense to the assembled panel. And if you think that that's part of why TDD is so hard to transmit to other teams or so often just not useful in practice, or maybe it's just not useful in practice at scale.
PENELOPE: One of the things I've seen is that if you're doing it right, you're doing dependency injection literally everywhere. And this is something Sandi brought up repeatedly in the episode that she was in.
NOEL: What's weird about that is I almost never do dependency injection. So, I kind of push back against that.
PENELOPE: Same
BETSY: I think part of it is that I agree that generally it means doing dependency injection literally everywhere. I think that it doesn't necessarily mean that, but I think that even when it doesn't, the ways that doesn't tend to involve either really useful strategies for managing stubs or ways of setting up your code base so that you don't mean to rely on stubbing to unit test. And both of those are not transmittable via copy and paste.
PENELOPE: Right. Actually, you kind of made my next point for me. I was winding up to that, which is that it led to a propagation of tools that let you like blow holes in your isolation boundaries in order to make testing easier. And like I will put my hat off and say RSpec mocks is like the isolation boundary destroyer. It's basically what it's designed for in principle.
BETSY: And that's difficult because I have a whole talk on this. When you do that, you're learning the wrong thing from the tool. The tool is pointing you in the wrong direction. It's a power users tool. But people go, "Oh, this is advanced stuff, therefore I must use it to be advanced." And they completely shoot themselves in the foot.
PENELOPE: One of the things I've always sort of found is that it makes all the sort of forms of this equivalently easy. So like passing a stub in is made as easy as replacing the implementation of a method which was made as easy as stubbing out an entire class, which was made as easy as using the any instance. There's four things that are equivalently easy with RSpec. So the tool provides you no feedback at all about the idea that you're sort of like going down a path which may indicate that you have a worse design. I've spent years being like, "Please don't do this. I know you can do this. I know it seems like this is what you should be doing, but if we do it this way, then our system will be easier to work with."
NOEL: So, what's the 'this way and then our system will be easier to work with'?
PENELOPE: I guess like anytime I see an any instance or a class stub, I'm immediately like, "We have a serious design issue here that we probably need to address." Unless we're in a legacy code base and we're like just trying to shore up some safeties around something. If this is fresh code, I will be like, "Please, can we find a different way to do this?"
BETSY: I will confess to have used a class stub in a greenfield code base earlier today without any regrets.
[Laughter]
PENELOPE: Can I ask, is that a Rails code base?
BETSY: Yes. But it's actually not the Rails bits. GitHub's GraphQL library is weird. [Laughs]
NOEL: I actually think it's a significant issue here that none of the frameworks that we sort of regularly use on the review side are like built to do TDD. Rails is built to do testing in a very smooth and profound and very valuable way and it was certainly one of the things that drew me to it initially. But it is not really built to do unit testing. You can't really test models without bringing in the entire database. It's nearly impossible to test associations without bringing in the entire database, still at this point. And I think that that also, and Justin's not here, but Justin would say because of Rails, the need to do testing to discover your domain is less because Rails provides such strong pressure on the domain. I think that's what Justin would say. He'd say a lot of things probably, that's one of them.
BETSY: I have a salty and controversial opinion for a minute.
NOEL: No, we don't want to have any salty controversial opinion. Why would we want that?
[Laughter]
PENELOPE: We didn't invite Justin, so salt's not allowed on this podcast.
NOEL: I did invite Justin. He is like 400 hours off of our time zone.
BETSY: The conventional unit test is that they're impossible to truly do because you have to pull in the database and blah, blah blah.
NOEL: Yes.
BETSY: But for me, I would argue that the database is just part of the unit in a Rails model test if you're writing Rails correctly. And so, you should just suck it up and deal.
NOEL: I mean that's basically what I do in practice, right? I think that Rails, yes, within the Rails universe, the database is part of your Active Record model.
PENELOPE: That's a really interesting way of looking at it because like just from a completely different domain, I was once in a distribution systems architecture meeting where they were like, "Cool, so you have your Rails app and you have your database and these are logically separate components in our distributed system." And I was like, "No, there's no way for us to keep that Rails app online if the database isn't there." Or they were like, "Have it return a health check that doesn't require the data." I was like, "The controller will raise an exception regardless of what I put in it if the database isn't there." Like, no, these are all one box. And so, I'm just sort of inclined to agree with Betsy there that like yeah, that makes a lot of sense.
BETSY: That's why I had been spending all that time recently on figuring out ways to make in memory data happen in Active Record tests. I think that we're not going to be able to convince people to stop doing horrible things with stubs to mock out their database in Active Record tests unless we can make that a faster process. And I don't actually think that the problem is with the Rails architecture here. I think it's with us needing test speed and doing horrible things.
NOEL: I do think it's weird and have long thought is weird that nobody has come along and built like in memory Active Record clone for the purpose of testing that lets you in memory manage associations as though they were in a database. And I don't think it's a library I've ever seen and I've never actually been bothered to write, but it's always been a surprise to me that somebody has not gone out and done it.
BETSY: It's because it's a viciously hard problem that involves writing a database from scratch. I had been working on it all summer. [laughs]
NOEL: Hence, the "I've never been bothered to write it myself".
BETSY: [Laughs]
NOEL: There are a lot of questions here, like in practice, is TDD really dead? Like is it worth practicing? Is it worth teaching to Rails developers? Is there a valuable piece of technique or a valuable thought process there at this point?
AVDI: I think the elephant in the room is that TDD, especially once we start talking about like mocks and stubs and things like that as well, once that enters the conversation, TDD is an exercise. It's a way of designing APIs, or very often is.
NOEL: In the larger sense, not just in like the server sense.
AVDI: Yeah. It's determining what your code surface is going to be to your code, to other people's code. It's determining what the interface is going to look like. And it's an exercise and it's a really interesting exercise in experimenting with, what does the usage look like when I design my API this way, when I design my methods this way, when I break the problem down into this set of classes versus that set of classes? The tests say a lot about that you have to write for that say a lot about how that API is, how that interfaces to use. What a lot of TDD advocates will say and unit testing advocates in general will say is that the point of it all is that your tests are the first consumer of your API. They are the original consumer and they give you kind of a sense of what it looks like to consume that API. And API could just mean the three methods that are on the class that you're writing right now. We're not talking about like large scale APIs here, but they give you a fingerprint of what it looks like to use the code that you're writing. And so, when I hear that Dave Thomas throw all his tests away and he and I have talked about this too, when I hear that he threw all his tests away or unit tests or whatever and still like doesn't see a major difference, well I also happened to know that Dave Thomas has an internalized and enormous amount of good taste about interface design over the years. And so, that doesn't surprise me at all because Dave Thomas, he can see in his head what it's going to be like to use the interface that he's creating. And likewise with mocks and stubs, originally that was really supposed to be about, let's see what it's going to look like to use, what is the API that we are going to need even though we don't have that class yet. This is a drum I've been banging for years and years, then I've realized that it's kind of a lost cause at this point that we were never supposed to use. Well, okay. A few years after the creators of mock objects invented them, they looked at their experiences and wrote up a paper that said, "Okay, what we have found is that they're great for simulating objects that don't exist yet, that we need to write because we find out what API those objects are going to need. And they are terrible for isolating the system from already existing other code." They're not so great for like pretending that the database doesn't exist and stuff like that, or pretending that your external API that you're supposed to talk to doesn't exist. They're not great for test performance hacks, but that's basically the only thing that people use them for anymore.
PENELOPE: Avdi, something you said that I'd like to go back in on, and it was right at the beginning where you sort of conceptualized that when someone is exercising, like going through the TDD exercises, they're sort of testing out multiple versions of how they might approach the same problem or editing the interfaces on their objects and that sort of thing. And I think one of the things I've seen is that even up to sort of very experienced mid-level developers, they will just do the TDD, churn through the TDD process and then stop and never actually do that sort of rearchitecting and designing. And even they'll do the refactor step because they'd been told to, but it'd be like, "I'm going to make this method smaller and make a private method," or something like that and never refactor as in move the whole system around. And I think one of the, and this is a question to the panel is like do we think that we've done a bad job of explaining what the refactor step is for? Because I think they'll write a failing test and the make the test pass steps are fairly self explanatory and the refactor step is not.
AVDI: I think there's a missing step because when we talk about the refactor step, we're very explicitly talking about their, according to the definition of refactoring is we take our tests exactly as they are and we muck around with the internals to make them prettier. And that's a really important step because you can write some really ugly code that makes the tests pass. But there is another step which we don't talk about, which is looking at the tests and saying, "Okay, all this works now and we refactored the internals, but this test is telling me that this interface is a pain to use." And at that point, you can't just refactor in a traditional sense. You have to throw the tests away and the code under test and you have to write new tests that are like, "Okay, this is the way I'd really like to use it. This is how I'd like it to work. And now let's make that work."
NOEL: That sounds hard, is the problem.
[Laughter]
BETSY: Being real about why you skip that though. We skip that because two reasons. One of them is that you're typically not inviting earlier career developers, to incorporate slack into their process explicitly. We're doing reverse. We're judging them on how fast they can have the output of a "real developer" because we don't really value them yet. We just view them as bodies and so they feel pressured to keep up by not actually teaching themselves properly. And the other reason is that we train them TDD as a dogmatic process. And that's the thing I think that actually needs to go, not TDD itself, but dogmatism around versions of TDD because dogmatism about versions of TDD discourages people from taking the time to reflect on their own code because they're not in a reflective state of mind. They're in a 'I will execute this mechanistic process' state of mind.
PENELOPE: I completely agree. It's interesting actually. I think Noel originally framed the question 10 minutes ago as like, are we still practicing TDD? I found myself making these much larger steps usually and just being like I could write the five tests that would imply this test or I could just write this test because I know I already like this interface because it's showing me the whole thing in one go. And I think that if you're newer you can't do that, you're looking for anything that lets you sort of not feel scared that something's gone horribly, as Betsy you were saying earlier that something's gone horribly wrong all the time. And I think we kind of hand over TDD and go, "Here you go. Do this." But there's no room to be like, "And this thing is not perfect and here's all the damages done to all of the billions of dollars worth of code bases and that you should find a way to make this work for you."
NOEL: One thing that struck me when I first started learning TDD and has always been a reason why I do it is that TDD reduces my short term cognitive load. It reduces the number of things I need to think about at one time by focusing my efforts on making the current test pass. And especially when I was first learning it, I found that very, very valuable. And I think over time, kind of as you were implying, that's less valuable as I've gotten better and better at keeping large chunks of code in my head at a time, and I've always kind of assumed that DHH, that a lot of Rails designs decisions make a lot of sense if you think of DHH as somebody who is like extremely good at keeping very large amounts of code in his head. A lot of the concerns about like the surface area of Active Record objects and then not having TDD, like all of that is consistent with a design philosophy where you're comfortable keeping a lot of stuff in your head. Does that make sense? Does that imply anything about how we teach this or should teach this?
BETSY: Yes. And I do think that optimizing for a team, because team is always the, even if it's a one person team, a team is always the appropriate unit for understanding a code base. Optimizing for a team that can keep a lot of code in their head at once is a reasonable choice for an organization to make. It's going to be a limiting choice in most cases because it means that you have to have a hiring strategy of a small number of experienced people who stay on the project for literally years. And that's not something the average organization is going to find sustainable, but it's a viable strategy.
AVDI: Yeah. Betsy, what you were saying reminds me that Jessica Kerr has a thing that she says about 10X developers, which is that the 10X developer is not somebody who's better at coding or smarter. The person that seems like the 10X developer on a project is the one that's been there since the project was incepted and just has a mental model of the whole thing in their head. That's not an inherent property. It's a property of context.
BETSY: Oh yeah. And I've worked on the team I just described. I was the new girl on that team and I had a very poor experience but it was in many ways a perfectly high functioning team for everyone else on it.
AVDI: Yeah.
NOEL: I have not spent most of my career on long lived teams. I've spent a lot of my career in small and short consulting projects where the teams turn over a lot. And I think that might affect the way that I approach the testing and design of code because I've never been in a situation where I've had the expectation that there is going to be anybody who has a complete mental model of the system like that.
PENELOPE: I was reflecting on my experiences. I joined this company that had been TDDing everything from scratch from day one. And in the sense that everyone on that team had enough experience that they could be trusted to write code independently without any help whatsoever. I think it actually sort of caused a convergence of people who had been around for a long time and people who were new, were delivering at not wildly varying rates. But that the reason that was true is that I think the actual average of new features being delivered was a lot lower than nearly every other team I've ever worked on. But they were incredibly precise because of all of the tests we were writing. We had three to one, four to one test to code ratio easily.
NOEL: Another thing that I sometimes think about here is sometimes when we talk about metrics or empirical studies of code, and there's actually been empirical studies of TDD and testing and all this stuff, and they tend to be terrible in terms of the design of the studies and they haven't shown a really strong effect. What would convince you that TDD was a bad idea and maybe had always been a bad idea, if anything? What would convince you to stop doing even the amount of it that you are doing now?
PENELOPE: Oh, I'm already there.
NOEL: Okay.
[Laughter]
NOEL: That's one person who has fallen off the train. Okay. So, what convinced you?
PENELOPE: I stopped doing it. Firstly, I didn't feel like my code was substantially worse or better or less reliable or more reliable. But because I have been an authority figure in software testing for essentially my entire career because I've been an Rspec maintainer since I graduated college, people stopped cargo-culting the tests I was writing. The very precise small tests and then doing it in a way where it wasn't actually TDD and ending up with tests that were worse than having larger ones or the ones they didn't implement if they were doing it post hoc.
BETSY: I am also in the 'I'm not currently actually doing a lot of TDD train' that's because the project I'm on right now is a really spiky exploratory year in project. So that's kind of a special case. And I do think that some of the code that I produced in the context of this project has been code that would have been coded differently if I'd used TDD methods. And I do think that some of it is in some ways worse than the code I would have been writing if I'd been test driving. It's a spike, though.
PENELOPE: Can I ask you a question?
BETSY: Yeah.
PENELOPE: Would you think it would be easy for you to get it to a state where it was not like sort of value judgment was and/or that you could retrofit tests on it to improve the design?
BETSY: Yes, but I specifically held myself back from a decision to make that easier early this morning. It was the near thing. I like over fitted to something and then I went, "No, Bad Betsy, slaps self on hand."
PENELOPE: Yeah. The reason I ask that is I think that people who have done it a lot know where to avoid those landmines easily.
BETSY: Yeah. I think that that's reasonable. One thing I do think, kind of riffing off of your question Noel, the evidence that caused me to have a much more complicated position on TDD than I used to earlier in my career was seeing the tests in a lot of modern JavaScript projects, and seeing frameworks in a lot of modern JavaScript projects because until that point, I've largely been working in a Rails context where one is kind of agreed to cross their fingers and say good enough in terms of test isolation because that's all you can really do. But the culture in React particularly but increasingly also Vue is to go for hyper isolation of small pieces. And a lot of the tests that are produced that way feel like virtually useless reifications of something that you were going to do anyway. One of the reasons I'm so hardline about the database as part of a unit in a Rails test now is because I've seen what happens to frontend tests that don't treat the browser as part of the unit and I really don't like it.
NOEL: Yeah. I have a whole separate rant about the DOM.
PENELOPE: I think we all have a rant about this.
AVDI: I have trouble with productivity measures and things like that. Quality measures can be better, not always. Earlier I was talking about how I think TDD unit tests, all this stuff, they're exercises in interface design. And really what that is it's an instance of sense-making. Most of the software project is sense-making. I don't know that we have any kind of quantitative measure of sense-making or whether such a thing even makes sense. Like how well are we doing at sense-making on this project? I don't think that we're going to assign a number to that anytime soon. And so, when I think about the question of what stats would convince me to throw away TDD or unit testing or whatever. Well, I don't think they exist because I don't think that there's a measure of the sense-making portion.
NOEL: No. But there might be a subjective feeling you have on project after project.
AVDI: But I say that as somebody who is kind of moving fluidly between stuff where I do very TDD style design and where I just sort of feel like, "Okay, yeah. This clearly doesn't need a test," whatever. And that's the voice of experience to some degree.
PENELOPE: Yeah. I think if I was to ask myself the inverse question, what stats, metrics, et cetera, would get you back on the TDD train? I think it would be seeing outcomes in code bases worked on by non-experts improve, because the outcomes I've seen in every Rails app where they've tried to do TDD and like the most senior person learned TDD by word of mouth and isn't really in a position to be great at teaching it all the way down, which is nearly every Rails company on the planet. To me, it's just so obviously worse than if they had just done larger tests retrospectively and almost effectively, like mini integration tests that are integrating two or three components of the applications and the database to try something out. And so, if you could plausibly change that effect, which I think is one we've all seen, that's what would do it for me.
NOEL: Okay.
BETSY: For me, it comes back to the sense-making point Avdi was making, I use TDD as a tool for sense-making. I am not convinced that everyone else does. And I think that TDD not used for sense-making has had some really corrosive effects.
NOEL: I would say that it seems to me that a lot of people test without a theory of why they're testing and I think where we're getting at is that there are a number of different theories about why to test that might be effective, but where you really get in trouble is if you just throw tests at the problem without any real sense of why you're writing this test or what your options are, and what options you're choosing.
PENELOPE: And I think some of that could be faulted back to the rhetoric by which a TDD was introduced. Like, "You must do this. This is how you produce quality software. Software not done with this is bad software," which is slight hyperbole, but I don't feel like it's a complete mischaracterization of some of the ways people have presented it.
NOEL: What do I tell these people that are coming to my workshop next week?
AVDI: I mean, for my money, I think it's still very valuable to teach people this skill. There are a lot of skills that are valuable in software, ranging from big O notation, estimating performance of algorithms, to things like TDD. You don't use any of them all the time. But we've found this one to be often useful as an intellectual exercise or as a design exercise. And it's certainly a lot more immediate than a lot of the design exercises that came before it. So, there are days when I just draw a fricking blank about how to move forward on the thing that I need to write. And on those days, TDD has often gotten me out of that rut because it gave me a way to make very small steps and not think about big picture. And if even only for that reason, I think it's a useful tool for new programmers to learn because it's going to give them a tool in their toolbox to get them out of that rut and we all need that sometimes.
NOEL: Okay. I like that. We can leave it there. Where can people find you online if they want to rant at you about testing some more?
BETSY: I'm Betsy. You can find me on the internet @BetsyTheMuffin on Twitter, or you can find my company's Twitter @WeCohere.
AVDI: I'm Avdi. You can find me online at Avdi.codes, and I'm on Twitter as @Avdi.
PENELOPE: You can find me on twitter @penelopezone or you can checkout my blog at http://penelope.zone.
NOEL: Great. Thank you for being on this episode. I really appreciate it and it's great to talk to you. And thank you for being here.
AVDI: Thank you.
PENELOPE: Thank you.
BETSY: Thank you.
NOEL: Tech Done Right is available on the web at TechDoneRight.io where you can learn more about our guests and comment on our episodes. Find us wherever you listen to podcasts. And if you liked the show, tell a friend or your social media network, and leaving a review on Apple Podcasts helps people find the show.
Tech Done Right is hosted by me, Noel Rappin. Our editor is Mandy Moore. You can find us on Twitter @NoelRap and @theRubyRep.
Tech Done Right is produced by Table XI. Table XI is a custom design and software company in Chicago. We've been named one of Inc. Magazine's best workplaces and we are a top-rated custom software development company on clutch.co. You can learn more about working with us or working for us at TableXI.com or follow us on Twitter @TableXI.
And now, an update on the show. I don't know exactly when this episode will be released, but this will probably be my final episode of Tech Done Right. Whether or not it continues in the future, I don't know. If it does, I certainly wish it well and hope that it continues to produce great shows. I've really enjoyed doing this for the past 72 episodes. It's been a thrill and an opportunity to have some great conversations with some wonderful people in and around the technology space. I hope that you've enjoyed it and thank you to all the people who have listened and left feedback, or told me that they liked an episode. It's really been great and I really appreciate all of the time that people have put into listening. And I hope that you have enjoyed the show, too. So, thank you.
