Amy Phillips 0:02
Welcome to the Humans+Tech podcast. I'm Amy Phillips. And this is Aaron Randall.
Aaron Randall 0:06
Hi.
Amy Phillips 0:07
Today we're joined by the incredible Jon Topper, CTO and CEO of The Scale Factory. Jon founded The Scale Factory way back in 2009, and they now help organisations of all sizes, design, build and manage the foundations of their digital businesses. Jon's a frequent writer and speaker, and in 2017, was named DevOps leader of the year at the DevOps excellence awards. Jon, welcome to the show.
Jon Topper 0:30
Thanks, good to be here. I bought that award, by the way, it was one of those kind of things, publishing companies these days, now that people aren't buying magazines try and find other ways to be relevant. They run these award shows, they sell you like a seat at the table for a grand or something. We bought a couple of those. And so I got an award. It's great. All it all it really did was get me loads of loads of stick from my fellow DevOps people, which, you know, has some value in itself. But yeah, I don't pay a lot of attention to that award. That would that was the claim at the time. I've reined it in since then. And we fired that PR agent as well. Hello.
Amy Phillips 1:15
Well, now we get to award you with something else. Right. So as you may know, every guest on the Humans+Tech podcast gets awarded with a personalised, photorealistic doodle from Aaron. I'm not going to have any part of this oney. Aaron, drew this. And here's your's for your evaluation.
Jon Topper 1:40
That's certainly better and more valuable than the DevOps leader of the year award. Yeah, I guess you've got a bit more hair on there than I actually have now. I went for the Coronavirus chop and actually buzzed my hair off like about a week in because I was pretty sure that we were going to be locked down for a while and my hair needed a certain amount of maintenance. So it all just came off. So that is amazing. If I've got here, this is great for your listeners, obviously. I'm going to show you my Songkick mug. I think one of you gave me when we were together back then with with another Aaron doodle so
Amy Phillips 2:16
I definitely didn't give you that mug Jon because I did not get one of those mugs.
Jon Topper 2:20
Oh wow.
Amy Phillips 2:21
I'm very put out by this.
Jon Topper 2:22
Oh, wow.
Amy Phillips 2:23
I just got shown this mug a number of times. I've not got my own mug.
Jon Topper 2:27
That's unreasonable. I will send you a Scale Factory mug. I'll get my people on that.
Aaron Randall 2:33
Have you got drawings on those mugs?
Jon Topper 2:35
I mean it's a it's a geometric logo. There's no real there's no drawings on there. But I could draw on it for you. It'll come off in the dishwasher though.
Amy Phillips 2:46
Well,
Aaron Randall 2:47
Amy told me I'd got the moustache wrong, by the way.
Amy Phillips 2:49
I mean, my initial thought was that the moustache was quite messy. And I'm not sure I really associate that with you, Jon.
Jon Topper 2:55
Right. Yeah, I mean, I used to do the curled moustache thing. I went for about a year and a half every two years actually properly waxing a handlebar. And it was a massive pain in the ass. It's frankly, like it's getting getting two sides of a moustache symmetrical, would often take sort of five or 10 minutes in the morning to get it quite right. And so yeah, I eventually just grew it out and gave up. So no longer no longer a feature of my my appearance, unfortunately.
Aaron Randall 3:25
Awesome. Jon. So I mean, obviously, Amy and I have known you for a long time many years now. And so for our listeners, we actually met when you were you supported us as we worked at Songkick and building our initial infrastructure there.
Jon Topper 3:40
Yep.
Aaron Randall 3:40
A long, long time ago. I was there before you two. In fact, I think
You were, yeah, that's true. Yeah, I joined and I got to enjoy your infrastructure work. And back then it was all about physical servers in data centres.
Jon Topper 3:56
Yep.
Aaron Randall 3:56
And now obviously, we're pretty much all in the Cloud. How has this changed the type of work that you do?
Jon Topper 4:02
And I mean, there's a lot less time spent sitting around in cold rooms or driving on the M25 with a car full of tin which is, which is a positive, I think we're able to have a much larger impact on people's businesses these days, I think the in the sort of old days when when you were racking hardware and doing that sort of thing, it was quite nobody would hire an external business to do that if they were big enough to have people to do that themselves. And so we were often doing that sort of work for smaller companies. And today, we're able to kind of go into into sort of enterprises and I will not even go into right sit in this very chair where I've been for the last six months of lock down, you know, logging into systems and and deploying them from there. Everything just moves much more quickly, as well as the agility that we have as a result of cloud is It's pretty amazing. And I guess not really something that I would have imagined sort of back then I think, I certainly wouldn't have imagined that the people who sold books would be selling it to. So and that was a bit of a bit of a shift, right. The book and CD company wants to sell us servers that I never see what, what. But yeah, it's, it's had a profound impact. I think on everything, I think I would I take issue a little bit with the idea that we're all in the cloud, I think, the that's definitely true of sort of tech first businesses and the SAS businesses that you know, you guys work for, and a lot of businesses that we work with, and I think Amazon and Google, Microsoft all recognise that there is still a huge opportunity in in, in getting stuff out of data centres, to the point where, you know, Amazon is still throwing, throwing funding programmes at people who want to move, move workloads in, and they are still building new products that are geared up towards making that story easier. And so I think there's still a sort of long, long tail of businesses who are running their own kit, and are not in the cloud. But that's only getting smaller, right, you know, most most, most of those businesses probably are also building things in the cloud as well. And are looking for an excuse to get that stuff out of their buildings and stop employing people to put wires together and sit in cold rooms typing on keyboards and stuff.
Aaron Randall 6:31
When you say that, are you nostalgic for those days at all, as somebody spent a lot of time in those data centres.
Jon Topper 6:37
I so I guess my career trajectory has taken me from practitioner to business owner, right. And so today, my job is spreadsheets and webinars. Right. And so I guess I'm nostalgic for anything that looks like technology practice. I'm not necessarily nostalgic for the for the sort of the data centre days, but there was something certainly in the height of summer, there was something quite pleasing about being able to go and sit in, in an air conditioned room away from everybody. But I used to find that the the background sort of white noise of a data centre would basically cause me to go mad, like you start getting, I start getting audio hallucinations, I imagine my phone ringing, I'd be constantly checking my pocket where the phone would have been vibrating, had it actually been ringing. And so you do get a little bit do lally when you've been in a data centre for a long period of time, and and often provisioning a new new platform, you're there sort of two or three days, crimping cables and slicing your hands open on poorly manufactured rack mount kits. Thanks, Dell. I don't I don't really miss it. I remember that time fondly. But I don't miss it.
Aaron Randall 7:48
That felt like a definitive no.
Jon Topper 7:49
It took a while to get it
Amy Phillips 7:54
I feel you're kind of glossing over they some of the benefits of the sort of offices that had a server room in the office though, like you know, right? Beer fridges are not nearly as big as server rooms, in my experience.
Jon Topper 8:07
Yep. Yeah. But also, server rooms typically not as cold as you would like your beer to be in the large part. And most well run facilities won't allow you to take liquids into them anyway. which is also part of the problem.
Amy Phillips 8:21
Well run might be the key
Jon Topper 8:23
Yes, yeah, I did at one point work for an organisation that had a data floor in a sort of adjunct to the where the office was that we were working. And and I wouldn't necessarily say that was particularly well run. But they didn't let us take our beers in there.
Amy Phillips 8:43
So I mean, it sounds like the sort of work you've been doing over the years that has like, radically changed. Like, from within sort of The Scale Factory, like, I, I'm guessing a decent amount of your work now is actually in the cloud or around the cloud. So yeah, what is it about like, so for someone that's not sort of hands in deep on this stuff? Like, what, why? Why is there work? Why is the cloud not just scaling itself? Like, isn't that kind of what it does?
Jon Topper 9:10
And so after a fashion, yes, there are certain certain parts of cloud that do take away a lot of that pain for you. But there's also a substantial amount of, of the cloud that doesn't, right, I think, we coined the name The Scale Factory originally, because, in part, we sort of recognised that there was we'd be solving a lot of the same problems repeatedly that so this is sort of stamping out of infrastructures following common good practice patterns was where the sort of factory idea came from. And scale was kind of at the time that seemed interesting, right, was that it was an interesting part of the the type of work that we were doing. And so it just sort of stuck. Today, I wouldn't say that we concentrate on scalability as much as we used to. And I mean, you're right, we are all in on the cloud. We're an Amazon Web Services, consultant partner and, and so everything that we do these days is is Amazon focused. But we are equally as likely these days to go and, you know, create or tune a CI/CD pipeline, or to go and do a security audit and improvement piece of work or something like that. So the scaling is a bit less bit less relevant. But at the same time, I think the the assumption that you can just throw your, your app onto the infrastructure and expect it to scale is not correct. still hasn't ever been hasn't ever been correct.
Amy Phillips 10:39
Hang on, but Kubernetes
Jon Topper 10:42
let's not get into the cave, Amy. I mean
Amy Phillips 10:48
For people who are not in there, like why Why? Why is Kubernetes not the answer, right? Because I think this comes up so often, right? I'm going to put it on Kubernetes, so that it just scales right, it will fix all my problems.
Jon Topper 10:58
Right? Yeah. And I mean, No, it won't. Or at least it I think,
Amy Phillips 11:03
You have new problems
Jon Topper 11:04
yeah, you replace your existing problems with with the problems of running Kubernetes. And so Kubernetes, at its heart isn't is sort of designed to solve a scaling problem. But it's also there to solve the problem of we as engineers, we're all doing the same stuff, right? We were all writing bluegreen deploy scripts, and we were all, you know, defining how our apps interacted with our load balancers, and all that kind of stuff. So what Kubernetes offers, which is really powerful, is a consistent interface to all of those problems being solved for you. It still requires you to be thinking about, you know, what, what, what, what is your what resources do you have? Is your application using? You know, how much? How much RAM or compute do I need to give to every instance of this particular part of the application server? Where's my data being held? How available does that need to be? And what is what's my database performance? Who's backing it up, like all of that kind of stuff, that these are still problems that somebody has to solve somewhere? That the story of the cloud is one of commodification, right. So in the old days, when we were working together, at Songkick, we built a lot of this stuff in a data centre. And the sort of highly available MySQL Cluster was something that had been very sort of meticulously put together by hand, using open source tooling and sort of off the shelf monitoring tools and that sort of stuff. But ultimately, someone i.e. me had spent the time plugging those those pieces of Lego together to get to the outcome that we wanted. Today, you get a highly available MySQL Cluster by making an API call and saying Amazon, can I have some, MySQL flavoured Aurora. And then you can just go and talk to it. Some of the scalability problems have gone away there, but you as a developer still have to be choosing good schema shapes, you know, adding indices to the right places. And the cloud lets you get away with a lot more of the awful crap that you used to not be able to get away with, because you have the limitations of the physical tin that you bought, but ultimately, there is always a point where you have to go and tune it or, or interrogate it for statistics to make different decisions. And so that there's always going to be a requirement for, you know, the type of operational thinking and skill that we provide. Because the assumption is that not everybody in your organisation who is good at cutting code, making features, building something that users want to want to use, and that delivers actual business value. The people who have those skills don't always necessarily have have skills in the, you know, algorithmic complexity understanding of their database queries. And so you always have to spend a bit of time on that stuff. And adding Kubernetes to the mix, doesn't make those problems go away. It might might change the way that you think about them, and it might move where they start to become a problem. But the the problems are kind of inherent in what we're doing. I think
Aaron Randall 13:59
so. So I guess if it's not scaling phenomenally is what you're saying, then, like, what are those most common pitfalls that you're observing right now?
Jon Topper 14:07
I mean, today, it's still the same pitfalls, as they were, you know, 10 years ago, my database performed really quickly in production, because there was no fucking data in it. They're, like, as soon as you soon as you move, move, so that database in development performs really well, but you move it into production, and suddenly you've got millions of rows, then the way that that that system performs, looks different. And so the kind of operational thinking that goes into making these decisions about how I use data or how I build my application is still the thing that is missing, I think, for all of the sort of positive sides of the of the DevOps movement as it as it was, I think we're still in a position where the people who are cutting code are not often invested in how that code is going to look when it's running in production. That that was that was the problem that that that DevOps sort of sought to fix. I don't think it was ever really fixed, or at least from my vantage point, as someone who gets a phone call when things are broken, you know, it looks like that's the case. So there's a selection bias here, right? We are working with customers who have called us because things aren't going the way they'd hoped. And so maybe outside of my kind of bubble, everyone's doing a really great job of everything, but I doubt it.
Amy Phillips 15:26
DevOps is an interesting one there, right? Because I think one of the big benefits of DevOps was kind of this sharing of responsibility. You don't get to just throw it over the wall, and it's someone else's problem.
Jon Topper 15:38
Yep.
Amy Phillips 15:39
Do you think it well I suppose? Well, the challenge I always see is the kind of the How do people develop the skills? Or or how many of these skills or what sort of level can they develop these skills to? Because yes, previously, you had someone who, you know, 10 years, actually as a sysadmin, who's been on call and dealing with the all these sort of incidents out of hours or whatever, they, they've got a huge amount of experience. So
Jon Topper 16:06
yes.
Amy Phillips 16:07
Is that a, is that something that a developer, say, frontend developer is now working in DevOps? Like, is that realistic, that they actually can get those skills? Or is it just a matter of time?
Jon Topper 16:22
So I think there's a you can you can learn this stuff, you can actually learn this stuff, right? This, this isn't some secret sauce, I think you're right, that unless you've sort of lived with it for a period of time, you don't, you don't gain the sort of fluency or kind of intuition around it the same as with any other skill, right. And the, I think the, you know, to bring this back to the sort of humans part of the Humans+Tech story, I think, as sysadmins, operators, so forth, we were very used to being lone wolves, we would spend a lot of time in these dark rooms, you know, saying no to people, and sort of preventing change, because change is the thing that causes everything to fall on its ass. And ultimately, we as people who became skilled at those things, and found it very or find it very difficult to teach those skills to others, find it difficult to find the patience to do that, right. And I speak for myself as well, I've, I was not a good sharer for a long period of time. And, and the sort of the, the power of, of knowing all of this stuff, and being able to sort of mete out that information to people as a sort of as a skill that only you had, and I think probably pushed some of my reward buttons. These are all things that I've discovered in, in therapy over the last couple of years, which I recommend to everybody, and especially sysadmins. And, and so I think we made life difficult for ourselves by not being willing to share or assuming that the the the people that we were providing this information to were, were too stupid to understand it. And that does a real disservice to basically everybody. So I think the other the other thing that made that did a difficult story, or maybe less realistic was that the DevOps community didn't really have any say anything like a manifesto, right? The Agile Manifesto is very clear. Sort of statement of intent. And the the DevOps world never really had that sort of clear this is this is what it's about, it had a lot of dogma, and it had a lot of kind of pithy catchphrases, we don't really use the word DevOps these days to describe what we do, partly because I think it's kind of table stakes. Now, I think that the the sort of shared ownership thing is what people are expecting to see. And we work in a DevOps-ey way, but we were no longer calling ourselves DevOps, a DevOps consultancy. And, and people just didn't know what it was. Right. And so a lot of organisations went through their lives, deploying some monitoring, letting the developers have a login on that monitoring platform, and calling that DevOps not really sure that's entirely entirely the full story. So we've picked up a lot of good practice, we've picked up, you know, CI/CD, we've picked up decent monitoring, we've picked up infrastructure as code, and you know, deployment automation, all that kind of good stuff. But all of those were technical solutions. And the the real kind of meat of the problem has with every problem, right? As you go through your career as a technologist, you eventually learned that all the problems are their meat sacks that are pushing the squares, not the things that go on in the machines with the squares on them. And so I think we could have as a community spent a bit more time talking about sharing knowledge and, you know, that sort of culture of practice stuff, then we ultimately did, and whilst devopsdays have has a lot of talking about around that every devopsdays that I've been to, has some variant of the open space session where someone sits and says, so how do I convince everybody to do DevOps? Right. And that's the sort of the extent of the story. And so it was never really clearly defined enough to be picked up by, by people, I think. And whilst ultimately I think it was a force for good and improvement, I think the more actionable stuff around SRE, is perhaps a kind of better, better framing of this stuff today.
Amy Phillips 20:33
Well, it's also a book. I think that's kind of back to the manifesto. Is there's one book that everybody references and that's fine.
Jon Topper 20:40
Yeah.
Amy Phillips 20:41
Like, you can just like, we all have it on our bookshelves, obviously.
Jon Topper 20:44
Yep. Yeah.
Amy Phillips 20:46
Yeah. I mean, it's like people reference it. And you have literally the SRE book. That's fine. We're all on the same page, which we did with Continuous Deployment as well. Like, yeah, I think when you have a single book, like Agile Manifesto had that Manifesto, but yeah, it was like DevOps missed out by just not having the defining book.
Jon Topper 21:03
Yes. Yeah. I think that's probably a fair assessment. I think that there are plenty there's plenty of criticism, you can levy against the the SRE stuff, I think the the sort of Google centricity of it all is, is a little bit grating to some people. And, you know, we we certainly when we're talking about in the consultancy work that we do, we we meet loads of teams who were very, very clearly definitely doing the Spotify model. And you kind of interrogate that and find out what they're doing and sort of realised that what they're doing is cargo culting, something that was not actually what Spotify were doing in the first place, and thinking that's going to help if you cargo cult Google's approach to problems, and you're not Google, and which you come back to the Kubernetes story, I suppose. Then you're you're not going to see the same benefits necessarily, all of these decisions need to be made within the context of your own business, and the business outcomes that you're that you're driving for, rather than just the sort of technology story, which is a bit more common in our landscape, because we are technologists, how we refer to ourselves,
Aaron Randall 22:08
Not meat sacks?
Jon Topper 22:11
Meat sacks when it's other people. A friend of mine who who's been consulting in this for much longer than than I refers to the human beings as the carbon units.
Aaron Randall 22:24
Meat sacks plus tech, we'll go for a rebrand.
Jon Topper 22:28
I look forward to seeing the doodle for that in fact,
Aaron Randall 22:32
Absolutly not. I'm gonna move on from scaling, Jon because you're a fairly prolific conference speaker, I've watched a bunch of conference talks over the years, and they're great. One of my favourite. One of my personal favourites is your Boring is powerful talk, can you tell us what boring tech is? And why we should care?
Jon Topper 22:57
Yeah, sure. So boring tech is the stuff that's been around long enough to work properly, I think it's probably the the really sort of pithy version of this, right. And this is, you know, in, in opposition to the sort of trendy stuff that pops up that gets a big sort of following behind it, because it's new and exciting. But as technologists, we're very much inclined to follow follow the changes in technology, that's what's exciting to us, right, we didn't get into tech, because we wanted to use a BBC Micro for the rest of our lives, we were sort of excited by the new things that that are available, and we sort of follow those and and so we we tend to be inclined to maybe put some of these sorts of technologies into production environments, way before they're ready. And boring is powerful is sort of my way of saying you should be critical about which new technologies you put into production. And particularly because often the people making technology choices are the the developers who are writing the code, they're very close to the user, they're very close to the business domain. And and in many cases, the the things that become readily popular are so because they are they have what I call a low meantime to Hello World. So you can download the thing from Hacker News or wherever it is, you get your up to date tech zeitgeist information from you can install it and you can get the thing apparently running enough to be able to run some of your own code or display your own image or something like that. And that pushes your dopamine buttons, it goes great. This definitely works and now it's in your in your ci pipeline then it shows up in production. And unfortunately the things that are optimised for good meantime to Hello World are often have often traded off against literally everything else. So most newer pieces of technology are not very well put together in terms of how you deploy them, or how you back them up or how they can be secured or whether the default configuration is correct, and so forth. And so what you end up with if you if you sort of follow this fashionable approach to technology choices is that you put something into production, that is then an absolute nightmare to keep up or to keep working or to conform to the SLAs, you've created your customers or what have you. And in in that particular talk, I talked about MongoDB and MongoDB, as a really good example of this. Because at the time, it was exceptionally trendy, it was this kind of like, Hey, you don't need to worry about database schemas anymore, which, of course, is very appealing to the type of people we were talking about earlier, who would just throw data into the databases and assume it would scale. And it got a lot of attention, it was very easy to get up and running, you could download a package, run the run the daemon and then start throwing data into it felt felt really kind of compelling to use. And it was it was released sort of version 1.2 was like December 2009, which is more or less when, when we founded The Scale Factory, in fact, and it wasn't until August 2011, that you could actually authenticate yourself against it as a as a sort of different user until that time, all users were equal. Everybody who could open a TCP port to it could squirt data in there. And and it wasn't until March 2013, that you had any kind of role based access control. So the only sort of security features that that probably
people who are writing software, with user outcomes in mind aren't necessarily thinking about these sorts of things in compliance industries, or we at the time, we were working with a pharmaceuticals business, or at least a SAAS vendor who was selling to the pharmaceuticals industry. And they actually needed all of this stuff. So they got a certain certain way down the line to producing this software using MongoDB. And then they found that they couldn't actually legally put it in production because they couldn't conform to the governance requirements that the the laws that they were following, put in place. And it wasn't until November 2016, that MongoDB actually passed a data safety test. Right, Kyle Kingsbury is Jepsen kind of, you know, cap theory testing mechanism didn't prove MongoDB as actually, it's gonna hold on to your data forever, until, you know, seven years after it was originally originally launched. So there's a real sort of caveat emptor here, right that the newer something is, the less likely it is to have these really important operational characteristics. And unless you are, unless you've got somebody in your organisation advocating for these things, at the time, which you're choosing the technology that you're going to build your apps on, you can easily back yourself into a corner where you've invested all this time and energy building a billing platform on something that ultimately can't work the way that you want it to. And so, boring tech, today, MongoDB is a boring technology, right? It's, it's got it's had enough eyeballs on it that it probably works properly. If you have a problem with it, you go and look it up on StackOverflow. You're not busting out a debugger and like the source code, which if you're if you're adopting things too early, you find yourself doing. And so the the sort of advice that I give around this is that you should really be choosing to, to adopt these kind of exciting technologies only where there is a tangible business benefit to doing so like this is the thing that will differentiate you and your product from your competitors. And because if not, then all you're doing is buying yourself a painful time. And I mean, going back to the Kubernetes story earlier on, like Kubernetes is still a little bit exciting. It's a new technology. And for a long period of time, the only way to run Kubernetes was to install it yourself using just sort of hodgepodge of shell scripts that it was, it was open source. And we've frequently come across customers today who are still running the versions of Kubernetes that they deployed back then because they're too scared of touching it because they don't all their business runs on it, but they don't know how to upgrade it. And they don't know whether it's working for them properly. Today because you can buy a commodified version of a Kubernetes runtime Kubernetes control plane from from a cloud vendor, that is now a more boring choice. And so you can go and do that know that the the cloud vendor is taking care of the the sort of janitorial work that's required to keep keep the thing alive. And so it's a better choice.
Aaron Randall 29:43
Yeah, really interesting. I just how do we, you know, as you mentioned at the beginning about that BBC Micro like, how do we as like humans working in tech teams who do love these dopamine hits, balance that dopamine hit plus a desire to actually keep up to date with the latest tech trends. Yeah, with building with building boring tech from boring?
Jon Topper 30:03
Yes. Yeah. So that's a really insightful question. Because I think if you don't keep keep a check on that stuff, then this exciting tech will appear in your production infrastructure because somebody wants to scratch that itch. And so that I guess the way that we do this, in our organisation, we we have a sort of a budget of development days that the team can draw down on in the same way that they would book a holiday, they can book a development day. And they can use that time for anything that they can, reasonably with a straight face tells me has business value to The Scale Factory. And most, for the most part, that's people doing stuff with tech, but it's occasionally also some will take that day to go and do a course on communication or something like that. And the so that we're sort of channelling that instinct into, into, into a sort of controlled, controlled outlet, which is quite helpful. And the, the other thing that we do with Dev Days is, if the if the outcome of that dev day is in some way valuable to us, such as someone's using it to learn a new tech, so they can write a blog post about it, for the company blog, they get that day back. So there's this sort of, perpetual cycle of if you if you can spend that day valuably in some kind of reputation building manner. So one of my team, Tim, is a contributor to the Kubernetes documentation, special interest group, and he spends some of his Dev Days contributing to to sig docs, I've got a couple of people on the team who are getting better at putting themselves out there as conference speakers, and so they use their days to prepare that material. But yeah, in to bring back to your original question, it's about sort of providing an outlet for that, that stuff. And making sure that design decisions are taken collaboratively rather than individually. Because I think if you have a lot of eyes on this sort of thing, then it can get reeled in a little little easier. And plus, of course, because we're working for customers, rather than building stuff for ourselves, you sort of have to get the buy in from the customers, I think they want to want to do as well. So it's one of the as we've, as we've shifted from being a sort of, I guess, an engineering consultancy, as in that most of the team doing hands on engineering work for customers on a day to day basis, to a more strategic and design and architecture consultancy. And then the sort of the shift in the team has been around becoming more aware of business outcomes and being able to draw a straight line between the technology and what what the what the business is trying to achieve. And that's been difficult to teach actually, it's, it's not always an easy thing to, to rein in I guess the the instinct to, to just go and build something that's fun, or, or justify the thing that you want to build in terms of the the business outcome, which at least is the next step in that direction, I guess.
Amy Phillips 33:14
It's interesting though, like, I've definitely worked in companies where needing to have a little bit of shiny Tech has been quite important from a hiring point of view, right? You know, actually, like, when people are looking for new jobs, it's like, oh, I want to, you know, I want to get into this language, or I want to be using whatever. And so there is also a little bit of pressure that you can't, well for some people, like some teams I've been, it's kind of like, our all of our tech is a bit too boring. We need some more shiny at attract new people.
Jon Topper 33:43
So what one of our long standing larger clients have been going through a, a very long, long term strangler exercise to break a huge Java Oracle monolith into micro services. And they intentionally chose Scala to do that. Not because they thought it was necessarily the exactly the right tool for the job. But because the business domain of this problem is so fucking boring, that they figured the only way they could get people who could who could, the only way that they could interest people come and work on this platform was by doing it in Scala. And so that was that was a major part of their choice to do that. It was the hiring choice. I mean, now they've got a lot of fucking Scala to run so I don't know whether that's, that's worked out exactly the way that they intended. But it seems to be going okay, so far, because at least they're they're microservices, rather than a Scala monolith. But yeah, it's a reasonable it's a reasonable concern. And I would, I would argue a little bit I think, I don't believe this very strongly, I don't think but i i would argue a little bit. You don't want a team full of people who chase shiny new technology. I don't think I don't think that's what you want around like, I I'm biassed towards, you know, wanting people who look at a new tech and go, Oh God, that doesn't look like it's going to work because that's the that is the sysadmin ops mindset, right? That's you want people who can you want your sysadmins should be pessimists, right. That's the only way that you can sit down and enumerate all the things in which a platform will go wrong. So you can go mitigate it. And if all you have are optimists and I think the people that chase the shiny technologies are often optimists, and then that stuff never gets considered. And, and I think that this is borne out a little bit in we were within our Amazon partnership, we're part of a programme called well architected, which is a review framework that Amazon provides, that we go and use this framework to assess somebody's practice on the AWS platform against Amazon standards, and sort of make them some recommendations and say, Oh, you should be doing this differently, or you're doing quite well here, or have you considered this, this technology here, and the thing that we find that the framework is split into five pillars, and one of those pillars is operations. And operations is by far the worst scoring pillar for pretty much everybody. And the majority of that, I think comes down to one, the idea that if you put it in the cloud, Amazon will just take care of it for you. Right, that's a that's definitely a sort of a fallacy. And there's the optimistic idea that nothing ever goes wrong, which anybody who's worked in production systems for any amount of time, will tell you is also a fallacy. And it's under invested in, teams don't spend time thinking about failure modes, because failure dealing with failure modes is sort of its insurance policy. It's, it's not sexy, it doesn't add business value, necessarily. It's it's more that the time that you spend on good operations protects revenue, it doesn't create it. And so it's not seen as important. People don't write down their how things work, people don't share and share amongst the team, how you would recover from the top 10 incident types that you might see. And your teams are just pretty bad at it. I think this is a I think this is a human condition thing, I think we're inclined to think that everything's going to be fine. And occasionally, you need someone to show up and say, Hey, that limit that you've got set really high on this particular part of of your AWS account means that if you accidentally check your Amazon keys into GitHub in a public repo, which happens plenty of times, and then someone could spin up those instances and mine Bitcoin on it, and not only will they be mining Bitcoin, there'll be costing you about 25,000 pounds per hour. While those systems run, like, should we should we tune that down a little bit? So thinking through those sorts of scenarios is a very pessimistic, but pessimistic thing. And and people don't like to think about worst case scenarios. It's not It's not fun.
Amy Phillips 38:06
Yeah, definitely.
Jon Topper 38:07
I would argue that's maybe why our pandemic response has been so terrible. But maybe, maybe politics isn't the topic we want to stray into. In the UK,
Amy Phillips 38:20
We've got we've got people in the audience who are in much more sensible countries, so well, then some people are out and about having a great day, walking in the park listening to this,
Jon Topper 38:30
I've got a number people on the team who are not from the UK originally, and a not inconsequential number of them went back to the countries that they were brought up in for the duration of the first lockdown, because A they wanted to be with their families. And B, they thought that they might do a better job of containing things in Portugal than than we would manage here.
Aaron Randall 38:51
Yeah. Just to just to wrap that, though, I really love that point. You made about like, your hiring plan. And like, do you want to hire those kinds of people? I guess, like, Yeah, do you want to have shiny tech to get those kind of people? I think that's a really interesting way. It's like framing that problem.
Jon Topper 39:05
Right? And I think so. Simon Wardley, who you probably are broadly aware of, because he's spoken a lot of sort of ops and tech related things, and talks about the sort of the evolution of technology and how it goes from being something that you create that you have to build yourself with your bare hands and maybe a soldering iron or two, all the way through to commodification where you can just you know, the power we're using today right 240 volts 50 hertz, comes with a 13 amp plug, which is why that's a commodity, we don't have to worry about it. And he talks about the the evolution of technology along that that spectrum and that you you have different types of people involved at each part of that journey. And so there's this talk of Pioneers, Settlers and Town Planners and the pioneers are the people who are like going out breaking new ground and trying new things. Those are the shiny, shiny technology chases right that Let's let's put this thing together, really exciting kind of, you know, you need people like that to invent things that you can't if you're if you're sort of if you're trying to build disruptive applications, or you're building something brand new that no one's ever seen before you need people with that sort of mindset, they are important individuals. But then you also need the settlers who will take the jagged edges of those solutions and the kind of spit and duct tape adhesives and and polish them up and make them a little bit more more palatable. And sort of get them into a position where they're, they're a bit less dangerous, and that they're a bit better, better known. And most businesses sort of get to there, right at the the far end of the evolutionary spectrum are the town planners, and they're the people who are making it really boring. They're the people who are sitting in rooms deciding what shape a 13 amp plug should be and sort of pulling all of these these sort of strands together and commodifying it Amazon are town planners in the infrastructural sense. So if you look at the the evolution of say MongoDB, the sort of Genesis part of its lifetime, you had the pioneers who were building this thing, right and debugging it and running it in production, you know, keeping both pieces when it fell apart. And And over time, that open source project grows into into a commercial entity that is working on that thing and stabilising it and listening to the pharmaceutical customers of the world who say, actually, would you mind putting a bit of TLS on this, right and securing the thing. And then you've got Amazon at the far end of this, I guess MongoDB is a bad example this because Amazon's MongoDB implementation is a MongoDB compatible layer against their own technology. But they've essentially commodified a MongoDB compatible database service that you can now consume without needing to know anything about how you install and configure MongoDB. And this is a this is all part of the tech lifecycle. And the the argument is that if you're spending time pioneering on something that has no business differentiation value for you. Or if you're if you're doing town planning, but you're not selling commodified things, then you're probably spending your time in the wrong places. And there are there are other sort of related concerns, such as in the Pioneer landscape, you should probably be using agile, to run your delivery because it's, it suits the nature of that work. At the town planning and of the equation, you're probably looking at ITIL, and ISO 27,001, and all that sort of the standards following stuff because the pace of change is a little different, you probably doing Prince II project management and that sort of thing, because the Agile model doesn't fit that world. And so making appropriate choices of technology processes, and how you spend your time. And based on where you are in that lifecycle is a really important thing to do. And that's part of part of the consultancy work we do really is to sort of look at, look at what an organisation is doing holistically and sort of try and understand whether they're trying to, you know, fit fit a pioneer problem into a town planner shaped hole.
Aaron Randall 43:16
And when you talk about that pioneer problem in a town planner shaped hole in your conference talk you you talk about this idea of innovation tokens design is that like somewhat related this idea of like being able to afford a send a few pioneers out to try something out?
Jon Topper 43:29
Yeah. So I can't claim to to have coined the innovation token terminology, that's a guy called Dan McKinley, who wrote about it while he was working at Etsy in 2015. And you're looking at me, like impressed with my recall. I wrote this down earlier, because I knew you were going to ask me about this and I didn't want to take credit for someone else's stuff.
Aaron Randall 43:52
You ruined that one
Jon Topper 43:56
Ditto the MongoDB timeline, I've got my talk slides of it. So So Dan, Dan has a blog post that that's, that's called. It's not boring it's powerful, but it's something about boring, choose boring technology. And yeah, the the idea is that you have a limited number of innovation tokens to spend. And so you should spend them wisely and wisely means in the pursuit of the business differentiation, outcome that you are working on. So at Songkick, for example, your innovation tokens were all very well spent in doing the sort of artists search or the the ticket purchasing or the things that made Songkick essentially Songkick. Spending those innovation tokens on I don't know building your own database, which thankfully you didn't do, because I would have shaken someone to death if they'd suggested it. You do need a little bit of fear as a sysadmin. So we said, The yeah, if you've spent that if you've spent that time kind of doing things that that looked like, you know, building infrastructure components, that wouldn't have been a smart move. And that's why it was sensible at the time for Songkick to bring me in as a as a contractor to do those things. Because there was no need for anybody in Songkick to be good at racking hardware and configuring switches and so forth. There's a bunch of sort of hard work that you do up front to get that stuff running. But then later, the the sort of day to day operation stuff, other people can easily do with a bit of, you know, training and assistance. And that's, that's broadly how we think about the cloud stuff today is, most most of the businesses that we work with don't need sort of high end, Amazon solutions, architect professional level individuals in their organization for the whole time. Because once they've stood something up that broadly works, everything else is iterative from there.
Amy Phillips 45:52
That's interesting. It really strikes me, Jon, when you talk about these things that the so much from your years of experience, like you've seen..
Jon Topper 46:01
I look old and tired. Is that what you're saying?
Amy Phillips 46:07
Like, oh, I've come across this thing, or I've seen this as a consultant. I imagine it's easier to you've seen it 20 time before.
Jon Topper 46:14
Yes. Yeah.
Amy Phillips 46:15
That's also true, it kind of like sysadmin started was thinking of almost with the DevOps is, actually if you haven't got years of experience, you haven't seen the database fail in that way three times previously, it's harder to work out what's happening, right? Do you, when you sort of look back to when you started, like, is there like, Is there a tip you wish you wish to have given yourself like 10 years ago as you started out on this sort of journey?
Jon Topper 46:44
Oh, that's a good question. I think 20 years ago Me was a cocky little fucker really, and and was was cockier than his than his own abilities. And, and in a way that sort of set me an ambition to kind of become as good as I thought I was, there's a bit of that. But I think I've, my professional career, in my personal life and everything else, I've been very independence, I've been very sort of like, untrusting of other people and you know, not not feeling like I can rely on people to do a good job. And again, plus one for therapy, that I'm sort of unraveling why that is and making a good understanding of that. I sort of wish I'd done therapy earlier, I guess the advice is like at 21 22, go see a therapist to to work out who you are. Because I'm now today, having spoken to therapists for sort of two or three years now I think, there abouts, I feel like I have a much better awareness of myself and a better understanding of my, my sort of biases, and enough of an understanding of those biases that I can now correct for them in the moment. So if I feel myself kind of going a little bit too independent on something, I can rein myself in and ask for help from somebody, which I would never have done back then. That said, You know, I think being fiercely independent did mean that I had to learn a lot of stuff, right? In order to sort of live up to the image I portrayed for myself. And, and so, you know, this is very sort of psychology, psychiatry stuff, I suppose. But like every strength is has a weakness and vice versa. Right. There's, there's a sort of a flip side to everything. So my, the flip side of my being sort of untrusting of other people and very independent meant that I had to go and figure out how this stuff worked myself and become become good at it. And so I think it's that I think that the message for me of that era would be you need to lean on other people a bit more often. And, and let them in to help. And and I wouldn't have listened to myself, had I been given that advice at 21. Anyway, so I guess that's kind of moot. I think that the when you talk about the the sort of, you know, I have seen a lot of stuff, right, and I think consultancy is definitely a part of that. The other the other sort of part, I think is that I'm, I'm blessed with a really good memory, like, it's not photographic. It's not, it's not a sort of, you know, sort of superhuman thing. I just remember a lot of things, and coupled with a good understanding of the fundamentals. And I think you can solve any problem, right? So I remember a lot of the stuff, and I kind of remember where the Apache config flags that I haven't touched for, you know, 10 years or something. And there are certain kind of failure conditions that I can go, you know what I think it's x and y based on these two other things that I've seen my understanding of the domain lets me sort of cut off a couple of things that it almost certainly isn't and narrow down on the problem pretty quickly. And so it comes across. It comes across as expertise. And I suppose it is in a way but but to me, it's it's about remembering stuff and having a grasp of the fundamentals. I think that's the the thing that if, if anybody's listening, wants to kind of improve at their operations stuff in the technical sense. And a really good understanding of the foundations of how we build this stuff, I think is is the is the key. I, last year, I think, hired hired someone who was kind of, I don't know, 25 or something. And so they'd never run a BGP session on a router, or had no real concept how the DNS system fits together. And there were a couple of outages that happened that year, where I sort of sat down and walked through the kind of how this actually happened and he grabbed his head, and how does any of this work? And at that point, I think he's achieved enlightenment, because actually, none of this stuff should work at all, because it's all built on on foundations of trust, and, you know, assumptions from the 1970s. And the fact that any of the internet is still working in the slightest is nothing short of miraculous to me to be honest.
Amy Phillips 51:33
I, you moved on quite quickly, but like, I really wanted to say like actually thank you for like highlighting how one of the things we've touched upon in previous episodes, is how it can be quite hard and lonely when you're leading. And you can be quite alone. So yeah, I think you're the first person who's really actually been really quite open about actually, everybody should just go and get therapy go and find out who you are which I think
Jon Topper 51:58
particularly men, particularly men, right, I think
Amy Phillips 52:01
I think everybody,
Jon Topper 52:03
probably everybody, I think men need to hear the message more, right? I think I talk very openly about my therapy experience, partly because it's been really helpful to me, and I like sharing things that help us one of the reasons I talk a lot at conferences is that if I've learned something that's really useful to me, I think other people might want to know it as well. And so I talk about having been through therapy, and partly because of that, and partly because I think we still live in a world today where, you know, asking for help as a man is considered weak. And, and that's bullshit, frankly. And there's, I feel much, much stronger as a result of going through therapy and sort of understanding myself. And I guess I'll put this out there, if anybody's listening and would like to talk about going into therapy, and sort of would like to have some fears about that allayed or otherwise, then I'd be more than happy to sit on a on a zoom or a hangout for 30 minutes or so and sort of talk through it because I think it was therapy's seen as a little bit of a bit of a dark art almost right, you sort of you people don't talk about what they talk about in their therapy very often, right? Because it's very personal and, and quite rightly so. And so from, from an outsider's perspective, it looks like you go and sit in a room and get interrogated for an hour at a time. And it's really not, I mean, it can be like, times when it needs to be, but it's, but it's really not like that at all. And it's, it's also it's quite a technical discipline, there are a lot of different types of therapy. And, you know, for someone who is going into therapy for the first time, it sort of doesn't matter, like what the modality is like the the sort of, is this jungian or freudian? Or is it psychosynthesis? So like, all those kind of words, might as well be like, do you like puppet or chef, like, it's a really kind of religious viewpoint for the people who practice therapy, but for the people who are receiving it, like, it sort of doesn't matter. And, but yeah, I'd be more than happy to kind of, I guess, offer some words of comfort to anybody who was thinking about going into therapy, particularly as a as a man. And, and I would seek to encourage them to do that, because I think it's a it's an important powerful thing. And particularly, as you say, like, as a lead or a leader. For me, I own the business. You know, I'm the CEO, the CTO, every everybody, the buck for everything stops with me. And I don't have a one to one with anybody, like, Who am I going to have a one to one with? It turns out that therapy and external coaching are the ways to do that if you're that sort of high up in the organization. And I've sort of I've solved the therapy problem. I've done a little bit of coaching as well. I think there's more coaching in my future around around certain things. But asking for help is the thing that I've learned to do in later life that, I think is the thing that marks me out just having finally grown up along with, you know, the gray hairs and my beard, which are expected of someone who knows Unix as well as I do.
Aaron Randall 55:12
Amazing, Jon, thank you so much for sharing that sentiment that was like, such a profound and an insightful answer. And it's really nice to hear that.
Jon Topper 55:21
I'm glad. I'm glad it comes across that way. It was not my intent. But I do feel pretty strongly about it. So
Aaron Randall 15:27
Yeah, no, it's great. It's really great. And we, I mean, we should wrap here. And that's a really great, great place to stop. Yeah. An incredible high, we have lots of learnings. And I do want to wrap with our usual three quickfire questions. If that's okay.
Jon Topper 55:41
Yes, yeah.
Aaron Randall 55:27
And we like to ask these to all of our Humans+ Tech podcast guests, as a question for you is, what's your top book recommendation?
Jon Topper 55:50
So it would be remiss of me to not to mention the book that I talked about incessantly in every presentation that I give recently, and that is Accelerate, which is the the sort of I can't remember what the subtitle of it is, but it's the it's the DevOps book that has statistics in it. Right? It's the only is the only sort of numerical study, or quantitative study of the things that we've been preaching dogmatically for a decade. And so for me, that's really important, because it allows me to point out some science and say, you know, that the science says you should do this. And I think that the authors are Nicole Forsgren, who is Dr. Nicole Foster, and in fact, who's the primary author, and the person who's done the actual science. Gene Kim and Jez Humble are there as well. I don't know whether that's whether they had major input into the into the stats, but their names are on the front, at least.
Amy Phillips 56:45
Jez Humble had a reasonable input.
Jon Topper 56:48
Right, that's a that's yes. I think that i think that's likely. And and, yeah, I recommend it in in most of my most of my talks, I think the only if I'm going to levy criticism against it, it's that the out the sort of outcomes, that there's a, there's a sort of assumption that you're measuring performance. And the performance is all about rapid delivery of software and low, low error delivery of software, it doesn't make any attempts to draw those parallels with business outcomes or sort of commercial goals. And so it falls a little short there for me, but as a, as a tool to offer a bit of quantative proof for the dogma we've been spouting like, very valuable. Awesome, it's the it's the it's the questions that a quick fire right, not the answers.
Aaron Randall 57:41
Quick fire question number two is who inspires you in tech?
Jon Topper 57:46
Well, that's an interesting one, because I guess my my lone wolf tendencies have led me to kind of not really sort of identify with the idea of being inspired by others, which is interesting and a topic for Tuesday morning, maybe. And but I guess, in the, in the realm of, of what we do today in the cloud industry, I mean, Corey Quinn is someone that I think is, is pretty inspiring, if only for the sheer volume of tweets that he manages to churn out. And so I guess he's inspiring from a sort of personal brand perspective, I think he does it extremely well into the promotion and so forth. And, you know, it's the sort of broad reach that I would quite like, for the consultancy work that we do, but there is no possible way that his approach would work for us. Which is why it's inspiring I think, to go, you found your you found your audience, you found your, your sort of tone, and it's really working. I don't want to sound like that. Thanks.
Aaron Randall 58:54
Nice. And finally, what's the most surprisingly non tech thing about you?
Jon Topper 59:00
That's a different question to the other ones that you've been asking.
Aaron Randall 59:02
You can't prep for this one my friend.
Jon Topper 59:05
I definitely had an answer for the other one that you're asking. Ask that again the most surprisingly non tech thing about me?
Aaron Randall 59:13
Yeah.
Jon Topper 59:14
And so something that that would surprise other people if I told them it is that what you're looking for?
Aaron Randall 59:24
something surprising that is non tech about your life. Oh,
Jon Topper 59:27
yeah. Okay. I mean, I'm polyamorous. I mean, that's, that's not that shouldn't be surprising to anyone who's looked at my Twitter bio, but yeah, I'm a person who enjoys multiple romantic relationships and doesn't really believe in monogamy as a as a concept or, or a structural norm. So I guess that's, that does surprise some people. I'm not as vocal about it as I used to be. I used to sort of because it was new and shiny to me, I guess I was, I was often quite vocal because it would differentiate me and maybe shock people a little bit. But today, it's just a really normal part of my life. So I'm married, my wife and I've been together for over 20 years. My other partner, we've been together for 10 years. And it's all quite normal and boring these days and, and as domestic as you can be in a pandemic and between two houses. But yeah, it's, it can be surprising. I think it's it as a lifestyle choice it's becoming more, if not more popular, then certainly more visible these days. And it's I think, my my ethos in life, career, everything else has always been to sort of question the norm and sort of ask yourself, does this apply to me? Is that a, is this norm a rule that I want to follow? And many times the answer is no. And so I've sort of chosen my own path. And that's, that's one more way in which I've done that. And the other way is, you know, self employment, um, sort of just just a control freak, really just like to
Aaron Randall 01:01:01
Are you saying you're a pioneer, then Jon, in some ways.
Jon Topper 01:01:03
I, yeah, probably. In some ways, yeah, yeah.
Aaron Randall 01:01:10
Awesome. Thanks for sharing. Amazing. And finally, where can people find out more about you?
Jon Topper 01:01:16
Erm so I'm reasonably active on Twitter as @jtopper, where I tend to complain about things mostly these days, to be honest. I'm not as active on Twitter as I as I have been in the past. So Twitter is somewhere I go for tech news, but I don't don't tend to be quite as vocal myself there. I should probably turn that around, actually. And do a bit more kind of thought leadering there.
Aaron Randall 01:01:44
Awesome. Jon, thank you so much for taking the time to talk to us today. It has been so much fun.
Amy Phillips 01:01:50
Thanks so much.
Aaron Randall 01:01:56
We'll be sharing all the links and show notes plus the all important doodle over on HumansPlus.tech. I'm Aaron Randall. This is Amy Phillips and you've been listening to the Humans+Tech podcast.
Welcome to the Humans+Tech podcast. I'm Amy Phillips. And this is Aaron Randall.
Aaron Randall 0:06
Hi.
Amy Phillips 0:07
Today we're joined by the incredible Jon Topper, CTO and CEO of The Scale Factory. Jon founded The Scale Factory way back in 2009, and they now help organisations of all sizes, design, build and manage the foundations of their digital businesses. Jon's a frequent writer and speaker, and in 2017, was named DevOps leader of the year at the DevOps excellence awards. Jon, welcome to the show.
Jon Topper 0:30
Thanks, good to be here. I bought that award, by the way, it was one of those kind of things, publishing companies these days, now that people aren't buying magazines try and find other ways to be relevant. They run these award shows, they sell you like a seat at the table for a grand or something. We bought a couple of those. And so I got an award. It's great. All it all it really did was get me loads of loads of stick from my fellow DevOps people, which, you know, has some value in itself. But yeah, I don't pay a lot of attention to that award. That would that was the claim at the time. I've reined it in since then. And we fired that PR agent as well. Hello.
Amy Phillips 1:15
Well, now we get to award you with something else. Right. So as you may know, every guest on the Humans+Tech podcast gets awarded with a personalised, photorealistic doodle from Aaron. I'm not going to have any part of this oney. Aaron, drew this. And here's your's for your evaluation.
Jon Topper 1:40
That's certainly better and more valuable than the DevOps leader of the year award. Yeah, I guess you've got a bit more hair on there than I actually have now. I went for the Coronavirus chop and actually buzzed my hair off like about a week in because I was pretty sure that we were going to be locked down for a while and my hair needed a certain amount of maintenance. So it all just came off. So that is amazing. If I've got here, this is great for your listeners, obviously. I'm going to show you my Songkick mug. I think one of you gave me when we were together back then with with another Aaron doodle so
Amy Phillips 2:16
I definitely didn't give you that mug Jon because I did not get one of those mugs.
Jon Topper 2:20
Oh wow.
Amy Phillips 2:21
I'm very put out by this.
Jon Topper 2:22
Oh, wow.
Amy Phillips 2:23
I just got shown this mug a number of times. I've not got my own mug.
Jon Topper 2:27
That's unreasonable. I will send you a Scale Factory mug. I'll get my people on that.
Aaron Randall 2:33
Have you got drawings on those mugs?
Jon Topper 2:35
I mean it's a it's a geometric logo. There's no real there's no drawings on there. But I could draw on it for you. It'll come off in the dishwasher though.
Amy Phillips 2:46
Well,
Aaron Randall 2:47
Amy told me I'd got the moustache wrong, by the way.
Amy Phillips 2:49
I mean, my initial thought was that the moustache was quite messy. And I'm not sure I really associate that with you, Jon.
Jon Topper 2:55
Right. Yeah, I mean, I used to do the curled moustache thing. I went for about a year and a half every two years actually properly waxing a handlebar. And it was a massive pain in the ass. It's frankly, like it's getting getting two sides of a moustache symmetrical, would often take sort of five or 10 minutes in the morning to get it quite right. And so yeah, I eventually just grew it out and gave up. So no longer no longer a feature of my my appearance, unfortunately.
Aaron Randall 3:25
Awesome. Jon. So I mean, obviously, Amy and I have known you for a long time many years now. And so for our listeners, we actually met when you were you supported us as we worked at Songkick and building our initial infrastructure there.
Jon Topper 3:40
Yep.
Aaron Randall 3:40
A long, long time ago. I was there before you two. In fact, I think
You were, yeah, that's true. Yeah, I joined and I got to enjoy your infrastructure work. And back then it was all about physical servers in data centres.
Jon Topper 3:56
Yep.
Aaron Randall 3:56
And now obviously, we're pretty much all in the Cloud. How has this changed the type of work that you do?
Jon Topper 4:02
And I mean, there's a lot less time spent sitting around in cold rooms or driving on the M25 with a car full of tin which is, which is a positive, I think we're able to have a much larger impact on people's businesses these days, I think the in the sort of old days when when you were racking hardware and doing that sort of thing, it was quite nobody would hire an external business to do that if they were big enough to have people to do that themselves. And so we were often doing that sort of work for smaller companies. And today, we're able to kind of go into into sort of enterprises and I will not even go into right sit in this very chair where I've been for the last six months of lock down, you know, logging into systems and and deploying them from there. Everything just moves much more quickly, as well as the agility that we have as a result of cloud is It's pretty amazing. And I guess not really something that I would have imagined sort of back then I think, I certainly wouldn't have imagined that the people who sold books would be selling it to. So and that was a bit of a bit of a shift, right. The book and CD company wants to sell us servers that I never see what, what. But yeah, it's, it's had a profound impact. I think on everything, I think I would I take issue a little bit with the idea that we're all in the cloud, I think, the that's definitely true of sort of tech first businesses and the SAS businesses that you know, you guys work for, and a lot of businesses that we work with, and I think Amazon and Google, Microsoft all recognise that there is still a huge opportunity in in, in getting stuff out of data centres, to the point where, you know, Amazon is still throwing, throwing funding programmes at people who want to move, move workloads in, and they are still building new products that are geared up towards making that story easier. And so I think there's still a sort of long, long tail of businesses who are running their own kit, and are not in the cloud. But that's only getting smaller, right, you know, most most, most of those businesses probably are also building things in the cloud as well. And are looking for an excuse to get that stuff out of their buildings and stop employing people to put wires together and sit in cold rooms typing on keyboards and stuff.
Aaron Randall 6:31
When you say that, are you nostalgic for those days at all, as somebody spent a lot of time in those data centres.
Jon Topper 6:37
I so I guess my career trajectory has taken me from practitioner to business owner, right. And so today, my job is spreadsheets and webinars. Right. And so I guess I'm nostalgic for anything that looks like technology practice. I'm not necessarily nostalgic for the for the sort of the data centre days, but there was something certainly in the height of summer, there was something quite pleasing about being able to go and sit in, in an air conditioned room away from everybody. But I used to find that the the background sort of white noise of a data centre would basically cause me to go mad, like you start getting, I start getting audio hallucinations, I imagine my phone ringing, I'd be constantly checking my pocket where the phone would have been vibrating, had it actually been ringing. And so you do get a little bit do lally when you've been in a data centre for a long period of time, and and often provisioning a new new platform, you're there sort of two or three days, crimping cables and slicing your hands open on poorly manufactured rack mount kits. Thanks, Dell. I don't I don't really miss it. I remember that time fondly. But I don't miss it.
Aaron Randall 7:48
That felt like a definitive no.
Jon Topper 7:49
It took a while to get it
Amy Phillips 7:54
I feel you're kind of glossing over they some of the benefits of the sort of offices that had a server room in the office though, like you know, right? Beer fridges are not nearly as big as server rooms, in my experience.
Jon Topper 8:07
Yep. Yeah. But also, server rooms typically not as cold as you would like your beer to be in the large part. And most well run facilities won't allow you to take liquids into them anyway. which is also part of the problem.
Amy Phillips 8:21
Well run might be the key
Jon Topper 8:23
Yes, yeah, I did at one point work for an organisation that had a data floor in a sort of adjunct to the where the office was that we were working. And and I wouldn't necessarily say that was particularly well run. But they didn't let us take our beers in there.
Amy Phillips 8:43
So I mean, it sounds like the sort of work you've been doing over the years that has like, radically changed. Like, from within sort of The Scale Factory, like, I, I'm guessing a decent amount of your work now is actually in the cloud or around the cloud. So yeah, what is it about like, so for someone that's not sort of hands in deep on this stuff? Like, what, why? Why is there work? Why is the cloud not just scaling itself? Like, isn't that kind of what it does?
Jon Topper 9:10
And so after a fashion, yes, there are certain certain parts of cloud that do take away a lot of that pain for you. But there's also a substantial amount of, of the cloud that doesn't, right, I think, we coined the name The Scale Factory originally, because, in part, we sort of recognised that there was we'd be solving a lot of the same problems repeatedly that so this is sort of stamping out of infrastructures following common good practice patterns was where the sort of factory idea came from. And scale was kind of at the time that seemed interesting, right, was that it was an interesting part of the the type of work that we were doing. And so it just sort of stuck. Today, I wouldn't say that we concentrate on scalability as much as we used to. And I mean, you're right, we are all in on the cloud. We're an Amazon Web Services, consultant partner and, and so everything that we do these days is is Amazon focused. But we are equally as likely these days to go and, you know, create or tune a CI/CD pipeline, or to go and do a security audit and improvement piece of work or something like that. So the scaling is a bit less bit less relevant. But at the same time, I think the the assumption that you can just throw your, your app onto the infrastructure and expect it to scale is not correct. still hasn't ever been hasn't ever been correct.
Amy Phillips 10:39
Hang on, but Kubernetes
Jon Topper 10:42
let's not get into the cave, Amy. I mean
Amy Phillips 10:48
For people who are not in there, like why Why? Why is Kubernetes not the answer, right? Because I think this comes up so often, right? I'm going to put it on Kubernetes, so that it just scales right, it will fix all my problems.
Jon Topper 10:58
Right? Yeah. And I mean, No, it won't. Or at least it I think,
Amy Phillips 11:03
You have new problems
Jon Topper 11:04
yeah, you replace your existing problems with with the problems of running Kubernetes. And so Kubernetes, at its heart isn't is sort of designed to solve a scaling problem. But it's also there to solve the problem of we as engineers, we're all doing the same stuff, right? We were all writing bluegreen deploy scripts, and we were all, you know, defining how our apps interacted with our load balancers, and all that kind of stuff. So what Kubernetes offers, which is really powerful, is a consistent interface to all of those problems being solved for you. It still requires you to be thinking about, you know, what, what, what, what is your what resources do you have? Is your application using? You know, how much? How much RAM or compute do I need to give to every instance of this particular part of the application server? Where's my data being held? How available does that need to be? And what is what's my database performance? Who's backing it up, like all of that kind of stuff, that these are still problems that somebody has to solve somewhere? That the story of the cloud is one of commodification, right. So in the old days, when we were working together, at Songkick, we built a lot of this stuff in a data centre. And the sort of highly available MySQL Cluster was something that had been very sort of meticulously put together by hand, using open source tooling and sort of off the shelf monitoring tools and that sort of stuff. But ultimately, someone i.e. me had spent the time plugging those those pieces of Lego together to get to the outcome that we wanted. Today, you get a highly available MySQL Cluster by making an API call and saying Amazon, can I have some, MySQL flavoured Aurora. And then you can just go and talk to it. Some of the scalability problems have gone away there, but you as a developer still have to be choosing good schema shapes, you know, adding indices to the right places. And the cloud lets you get away with a lot more of the awful crap that you used to not be able to get away with, because you have the limitations of the physical tin that you bought, but ultimately, there is always a point where you have to go and tune it or, or interrogate it for statistics to make different decisions. And so that there's always going to be a requirement for, you know, the type of operational thinking and skill that we provide. Because the assumption is that not everybody in your organisation who is good at cutting code, making features, building something that users want to want to use, and that delivers actual business value. The people who have those skills don't always necessarily have have skills in the, you know, algorithmic complexity understanding of their database queries. And so you always have to spend a bit of time on that stuff. And adding Kubernetes to the mix, doesn't make those problems go away. It might might change the way that you think about them, and it might move where they start to become a problem. But the the problems are kind of inherent in what we're doing. I think
Aaron Randall 13:59
so. So I guess if it's not scaling phenomenally is what you're saying, then, like, what are those most common pitfalls that you're observing right now?
Jon Topper 14:07
I mean, today, it's still the same pitfalls, as they were, you know, 10 years ago, my database performed really quickly in production, because there was no fucking data in it. They're, like, as soon as you soon as you move, move, so that database in development performs really well, but you move it into production, and suddenly you've got millions of rows, then the way that that that system performs, looks different. And so the kind of operational thinking that goes into making these decisions about how I use data or how I build my application is still the thing that is missing, I think, for all of the sort of positive sides of the of the DevOps movement as it as it was, I think we're still in a position where the people who are cutting code are not often invested in how that code is going to look when it's running in production. That that was that was the problem that that that DevOps sort of sought to fix. I don't think it was ever really fixed, or at least from my vantage point, as someone who gets a phone call when things are broken, you know, it looks like that's the case. So there's a selection bias here, right? We are working with customers who have called us because things aren't going the way they'd hoped. And so maybe outside of my kind of bubble, everyone's doing a really great job of everything, but I doubt it.
Amy Phillips 15:26
DevOps is an interesting one there, right? Because I think one of the big benefits of DevOps was kind of this sharing of responsibility. You don't get to just throw it over the wall, and it's someone else's problem.
Jon Topper 15:38
Yep.
Amy Phillips 15:39
Do you think it well I suppose? Well, the challenge I always see is the kind of the How do people develop the skills? Or or how many of these skills or what sort of level can they develop these skills to? Because yes, previously, you had someone who, you know, 10 years, actually as a sysadmin, who's been on call and dealing with the all these sort of incidents out of hours or whatever, they, they've got a huge amount of experience. So
Jon Topper 16:06
yes.
Amy Phillips 16:07
Is that a, is that something that a developer, say, frontend developer is now working in DevOps? Like, is that realistic, that they actually can get those skills? Or is it just a matter of time?
Jon Topper 16:22
So I think there's a you can you can learn this stuff, you can actually learn this stuff, right? This, this isn't some secret sauce, I think you're right, that unless you've sort of lived with it for a period of time, you don't, you don't gain the sort of fluency or kind of intuition around it the same as with any other skill, right. And the, I think the, you know, to bring this back to the sort of humans part of the Humans+Tech story, I think, as sysadmins, operators, so forth, we were very used to being lone wolves, we would spend a lot of time in these dark rooms, you know, saying no to people, and sort of preventing change, because change is the thing that causes everything to fall on its ass. And ultimately, we as people who became skilled at those things, and found it very or find it very difficult to teach those skills to others, find it difficult to find the patience to do that, right. And I speak for myself as well, I've, I was not a good sharer for a long period of time. And, and the sort of the, the power of, of knowing all of this stuff, and being able to sort of mete out that information to people as a sort of as a skill that only you had, and I think probably pushed some of my reward buttons. These are all things that I've discovered in, in therapy over the last couple of years, which I recommend to everybody, and especially sysadmins. And, and so I think we made life difficult for ourselves by not being willing to share or assuming that the the the people that we were providing this information to were, were too stupid to understand it. And that does a real disservice to basically everybody. So I think the other the other thing that made that did a difficult story, or maybe less realistic was that the DevOps community didn't really have any say anything like a manifesto, right? The Agile Manifesto is very clear. Sort of statement of intent. And the the DevOps world never really had that sort of clear this is this is what it's about, it had a lot of dogma, and it had a lot of kind of pithy catchphrases, we don't really use the word DevOps these days to describe what we do, partly because I think it's kind of table stakes. Now, I think that the the sort of shared ownership thing is what people are expecting to see. And we work in a DevOps-ey way, but we were no longer calling ourselves DevOps, a DevOps consultancy. And, and people just didn't know what it was. Right. And so a lot of organisations went through their lives, deploying some monitoring, letting the developers have a login on that monitoring platform, and calling that DevOps not really sure that's entirely entirely the full story. So we've picked up a lot of good practice, we've picked up, you know, CI/CD, we've picked up decent monitoring, we've picked up infrastructure as code, and you know, deployment automation, all that kind of good stuff. But all of those were technical solutions. And the the real kind of meat of the problem has with every problem, right? As you go through your career as a technologist, you eventually learned that all the problems are their meat sacks that are pushing the squares, not the things that go on in the machines with the squares on them. And so I think we could have as a community spent a bit more time talking about sharing knowledge and, you know, that sort of culture of practice stuff, then we ultimately did, and whilst devopsdays have has a lot of talking about around that every devopsdays that I've been to, has some variant of the open space session where someone sits and says, so how do I convince everybody to do DevOps? Right. And that's the sort of the extent of the story. And so it was never really clearly defined enough to be picked up by, by people, I think. And whilst ultimately I think it was a force for good and improvement, I think the more actionable stuff around SRE, is perhaps a kind of better, better framing of this stuff today.
Amy Phillips 20:33
Well, it's also a book. I think that's kind of back to the manifesto. Is there's one book that everybody references and that's fine.
Jon Topper 20:40
Yeah.
Amy Phillips 20:41
Like, you can just like, we all have it on our bookshelves, obviously.
Jon Topper 20:44
Yep. Yeah.
Amy Phillips 20:46
Yeah. I mean, it's like people reference it. And you have literally the SRE book. That's fine. We're all on the same page, which we did with Continuous Deployment as well. Like, yeah, I think when you have a single book, like Agile Manifesto had that Manifesto, but yeah, it was like DevOps missed out by just not having the defining book.
Jon Topper 21:03
Yes. Yeah. I think that's probably a fair assessment. I think that there are plenty there's plenty of criticism, you can levy against the the SRE stuff, I think the the sort of Google centricity of it all is, is a little bit grating to some people. And, you know, we we certainly when we're talking about in the consultancy work that we do, we we meet loads of teams who were very, very clearly definitely doing the Spotify model. And you kind of interrogate that and find out what they're doing and sort of realised that what they're doing is cargo culting, something that was not actually what Spotify were doing in the first place, and thinking that's going to help if you cargo cult Google's approach to problems, and you're not Google, and which you come back to the Kubernetes story, I suppose. Then you're you're not going to see the same benefits necessarily, all of these decisions need to be made within the context of your own business, and the business outcomes that you're that you're driving for, rather than just the sort of technology story, which is a bit more common in our landscape, because we are technologists, how we refer to ourselves,
Aaron Randall 22:08
Not meat sacks?
Jon Topper 22:11
Meat sacks when it's other people. A friend of mine who who's been consulting in this for much longer than than I refers to the human beings as the carbon units.
Aaron Randall 22:24
Meat sacks plus tech, we'll go for a rebrand.
Jon Topper 22:28
I look forward to seeing the doodle for that in fact,
Aaron Randall 22:32
Absolutly not. I'm gonna move on from scaling, Jon because you're a fairly prolific conference speaker, I've watched a bunch of conference talks over the years, and they're great. One of my favourite. One of my personal favourites is your Boring is powerful talk, can you tell us what boring tech is? And why we should care?
Jon Topper 22:57
Yeah, sure. So boring tech is the stuff that's been around long enough to work properly, I think it's probably the the really sort of pithy version of this, right. And this is, you know, in, in opposition to the sort of trendy stuff that pops up that gets a big sort of following behind it, because it's new and exciting. But as technologists, we're very much inclined to follow follow the changes in technology, that's what's exciting to us, right, we didn't get into tech, because we wanted to use a BBC Micro for the rest of our lives, we were sort of excited by the new things that that are available, and we sort of follow those and and so we we tend to be inclined to maybe put some of these sorts of technologies into production environments, way before they're ready. And boring is powerful is sort of my way of saying you should be critical about which new technologies you put into production. And particularly because often the people making technology choices are the the developers who are writing the code, they're very close to the user, they're very close to the business domain. And and in many cases, the the things that become readily popular are so because they are they have what I call a low meantime to Hello World. So you can download the thing from Hacker News or wherever it is, you get your up to date tech zeitgeist information from you can install it and you can get the thing apparently running enough to be able to run some of your own code or display your own image or something like that. And that pushes your dopamine buttons, it goes great. This definitely works and now it's in your in your ci pipeline then it shows up in production. And unfortunately the things that are optimised for good meantime to Hello World are often have often traded off against literally everything else. So most newer pieces of technology are not very well put together in terms of how you deploy them, or how you back them up or how they can be secured or whether the default configuration is correct, and so forth. And so what you end up with if you if you sort of follow this fashionable approach to technology choices is that you put something into production, that is then an absolute nightmare to keep up or to keep working or to conform to the SLAs, you've created your customers or what have you. And in in that particular talk, I talked about MongoDB and MongoDB, as a really good example of this. Because at the time, it was exceptionally trendy, it was this kind of like, Hey, you don't need to worry about database schemas anymore, which, of course, is very appealing to the type of people we were talking about earlier, who would just throw data into the databases and assume it would scale. And it got a lot of attention, it was very easy to get up and running, you could download a package, run the run the daemon and then start throwing data into it felt felt really kind of compelling to use. And it was it was released sort of version 1.2 was like December 2009, which is more or less when, when we founded The Scale Factory, in fact, and it wasn't until August 2011, that you could actually authenticate yourself against it as a as a sort of different user until that time, all users were equal. Everybody who could open a TCP port to it could squirt data in there. And and it wasn't until March 2013, that you had any kind of role based access control. So the only sort of security features that that probably
people who are writing software, with user outcomes in mind aren't necessarily thinking about these sorts of things in compliance industries, or we at the time, we were working with a pharmaceuticals business, or at least a SAAS vendor who was selling to the pharmaceuticals industry. And they actually needed all of this stuff. So they got a certain certain way down the line to producing this software using MongoDB. And then they found that they couldn't actually legally put it in production because they couldn't conform to the governance requirements that the the laws that they were following, put in place. And it wasn't until November 2016, that MongoDB actually passed a data safety test. Right, Kyle Kingsbury is Jepsen kind of, you know, cap theory testing mechanism didn't prove MongoDB as actually, it's gonna hold on to your data forever, until, you know, seven years after it was originally originally launched. So there's a real sort of caveat emptor here, right that the newer something is, the less likely it is to have these really important operational characteristics. And unless you are, unless you've got somebody in your organisation advocating for these things, at the time, which you're choosing the technology that you're going to build your apps on, you can easily back yourself into a corner where you've invested all this time and energy building a billing platform on something that ultimately can't work the way that you want it to. And so, boring tech, today, MongoDB is a boring technology, right? It's, it's got it's had enough eyeballs on it that it probably works properly. If you have a problem with it, you go and look it up on StackOverflow. You're not busting out a debugger and like the source code, which if you're if you're adopting things too early, you find yourself doing. And so the the sort of advice that I give around this is that you should really be choosing to, to adopt these kind of exciting technologies only where there is a tangible business benefit to doing so like this is the thing that will differentiate you and your product from your competitors. And because if not, then all you're doing is buying yourself a painful time. And I mean, going back to the Kubernetes story earlier on, like Kubernetes is still a little bit exciting. It's a new technology. And for a long period of time, the only way to run Kubernetes was to install it yourself using just sort of hodgepodge of shell scripts that it was, it was open source. And we've frequently come across customers today who are still running the versions of Kubernetes that they deployed back then because they're too scared of touching it because they don't all their business runs on it, but they don't know how to upgrade it. And they don't know whether it's working for them properly. Today because you can buy a commodified version of a Kubernetes runtime Kubernetes control plane from from a cloud vendor, that is now a more boring choice. And so you can go and do that know that the the cloud vendor is taking care of the the sort of janitorial work that's required to keep keep the thing alive. And so it's a better choice.
Aaron Randall 29:43
Yeah, really interesting. I just how do we, you know, as you mentioned at the beginning about that BBC Micro like, how do we as like humans working in tech teams who do love these dopamine hits, balance that dopamine hit plus a desire to actually keep up to date with the latest tech trends. Yeah, with building with building boring tech from boring?
Jon Topper 30:03
Yes. Yeah. So that's a really insightful question. Because I think if you don't keep keep a check on that stuff, then this exciting tech will appear in your production infrastructure because somebody wants to scratch that itch. And so that I guess the way that we do this, in our organisation, we we have a sort of a budget of development days that the team can draw down on in the same way that they would book a holiday, they can book a development day. And they can use that time for anything that they can, reasonably with a straight face tells me has business value to The Scale Factory. And most, for the most part, that's people doing stuff with tech, but it's occasionally also some will take that day to go and do a course on communication or something like that. And the so that we're sort of channelling that instinct into, into, into a sort of controlled, controlled outlet, which is quite helpful. And the, the other thing that we do with Dev Days is, if the if the outcome of that dev day is in some way valuable to us, such as someone's using it to learn a new tech, so they can write a blog post about it, for the company blog, they get that day back. So there's this sort of, perpetual cycle of if you if you can spend that day valuably in some kind of reputation building manner. So one of my team, Tim, is a contributor to the Kubernetes documentation, special interest group, and he spends some of his Dev Days contributing to to sig docs, I've got a couple of people on the team who are getting better at putting themselves out there as conference speakers, and so they use their days to prepare that material. But yeah, in to bring back to your original question, it's about sort of providing an outlet for that, that stuff. And making sure that design decisions are taken collaboratively rather than individually. Because I think if you have a lot of eyes on this sort of thing, then it can get reeled in a little little easier. And plus, of course, because we're working for customers, rather than building stuff for ourselves, you sort of have to get the buy in from the customers, I think they want to want to do as well. So it's one of the as we've, as we've shifted from being a sort of, I guess, an engineering consultancy, as in that most of the team doing hands on engineering work for customers on a day to day basis, to a more strategic and design and architecture consultancy. And then the sort of the shift in the team has been around becoming more aware of business outcomes and being able to draw a straight line between the technology and what what the what the business is trying to achieve. And that's been difficult to teach actually, it's, it's not always an easy thing to, to rein in I guess the the instinct to, to just go and build something that's fun, or, or justify the thing that you want to build in terms of the the business outcome, which at least is the next step in that direction, I guess.
Amy Phillips 33:14
It's interesting though, like, I've definitely worked in companies where needing to have a little bit of shiny Tech has been quite important from a hiring point of view, right? You know, actually, like, when people are looking for new jobs, it's like, oh, I want to, you know, I want to get into this language, or I want to be using whatever. And so there is also a little bit of pressure that you can't, well for some people, like some teams I've been, it's kind of like, our all of our tech is a bit too boring. We need some more shiny at attract new people.
Jon Topper 33:43
So what one of our long standing larger clients have been going through a, a very long, long term strangler exercise to break a huge Java Oracle monolith into micro services. And they intentionally chose Scala to do that. Not because they thought it was necessarily the exactly the right tool for the job. But because the business domain of this problem is so fucking boring, that they figured the only way they could get people who could who could, the only way that they could interest people come and work on this platform was by doing it in Scala. And so that was that was a major part of their choice to do that. It was the hiring choice. I mean, now they've got a lot of fucking Scala to run so I don't know whether that's, that's worked out exactly the way that they intended. But it seems to be going okay, so far, because at least they're they're microservices, rather than a Scala monolith. But yeah, it's a reasonable it's a reasonable concern. And I would, I would argue a little bit I think, I don't believe this very strongly, I don't think but i i would argue a little bit. You don't want a team full of people who chase shiny new technology. I don't think I don't think that's what you want around like, I I'm biassed towards, you know, wanting people who look at a new tech and go, Oh God, that doesn't look like it's going to work because that's the that is the sysadmin ops mindset, right? That's you want people who can you want your sysadmins should be pessimists, right. That's the only way that you can sit down and enumerate all the things in which a platform will go wrong. So you can go mitigate it. And if all you have are optimists and I think the people that chase the shiny technologies are often optimists, and then that stuff never gets considered. And, and I think that this is borne out a little bit in we were within our Amazon partnership, we're part of a programme called well architected, which is a review framework that Amazon provides, that we go and use this framework to assess somebody's practice on the AWS platform against Amazon standards, and sort of make them some recommendations and say, Oh, you should be doing this differently, or you're doing quite well here, or have you considered this, this technology here, and the thing that we find that the framework is split into five pillars, and one of those pillars is operations. And operations is by far the worst scoring pillar for pretty much everybody. And the majority of that, I think comes down to one, the idea that if you put it in the cloud, Amazon will just take care of it for you. Right, that's a that's definitely a sort of a fallacy. And there's the optimistic idea that nothing ever goes wrong, which anybody who's worked in production systems for any amount of time, will tell you is also a fallacy. And it's under invested in, teams don't spend time thinking about failure modes, because failure dealing with failure modes is sort of its insurance policy. It's, it's not sexy, it doesn't add business value, necessarily. It's it's more that the time that you spend on good operations protects revenue, it doesn't create it. And so it's not seen as important. People don't write down their how things work, people don't share and share amongst the team, how you would recover from the top 10 incident types that you might see. And your teams are just pretty bad at it. I think this is a I think this is a human condition thing, I think we're inclined to think that everything's going to be fine. And occasionally, you need someone to show up and say, Hey, that limit that you've got set really high on this particular part of of your AWS account means that if you accidentally check your Amazon keys into GitHub in a public repo, which happens plenty of times, and then someone could spin up those instances and mine Bitcoin on it, and not only will they be mining Bitcoin, there'll be costing you about 25,000 pounds per hour. While those systems run, like, should we should we tune that down a little bit? So thinking through those sorts of scenarios is a very pessimistic, but pessimistic thing. And and people don't like to think about worst case scenarios. It's not It's not fun.
Amy Phillips 38:06
Yeah, definitely.
Jon Topper 38:07
I would argue that's maybe why our pandemic response has been so terrible. But maybe, maybe politics isn't the topic we want to stray into. In the UK,
Amy Phillips 38:20
We've got we've got people in the audience who are in much more sensible countries, so well, then some people are out and about having a great day, walking in the park listening to this,
Jon Topper 38:30
I've got a number people on the team who are not from the UK originally, and a not inconsequential number of them went back to the countries that they were brought up in for the duration of the first lockdown, because A they wanted to be with their families. And B, they thought that they might do a better job of containing things in Portugal than than we would manage here.
Aaron Randall 38:51
Yeah. Just to just to wrap that, though, I really love that point. You made about like, your hiring plan. And like, do you want to hire those kinds of people? I guess, like, Yeah, do you want to have shiny tech to get those kind of people? I think that's a really interesting way. It's like framing that problem.
Jon Topper 39:05
Right? And I think so. Simon Wardley, who you probably are broadly aware of, because he's spoken a lot of sort of ops and tech related things, and talks about the sort of the evolution of technology and how it goes from being something that you create that you have to build yourself with your bare hands and maybe a soldering iron or two, all the way through to commodification where you can just you know, the power we're using today right 240 volts 50 hertz, comes with a 13 amp plug, which is why that's a commodity, we don't have to worry about it. And he talks about the the evolution of technology along that that spectrum and that you you have different types of people involved at each part of that journey. And so there's this talk of Pioneers, Settlers and Town Planners and the pioneers are the people who are like going out breaking new ground and trying new things. Those are the shiny, shiny technology chases right that Let's let's put this thing together, really exciting kind of, you know, you need people like that to invent things that you can't if you're if you're sort of if you're trying to build disruptive applications, or you're building something brand new that no one's ever seen before you need people with that sort of mindset, they are important individuals. But then you also need the settlers who will take the jagged edges of those solutions and the kind of spit and duct tape adhesives and and polish them up and make them a little bit more more palatable. And sort of get them into a position where they're, they're a bit less dangerous, and that they're a bit better, better known. And most businesses sort of get to there, right at the the far end of the evolutionary spectrum are the town planners, and they're the people who are making it really boring. They're the people who are sitting in rooms deciding what shape a 13 amp plug should be and sort of pulling all of these these sort of strands together and commodifying it Amazon are town planners in the infrastructural sense. So if you look at the the evolution of say MongoDB, the sort of Genesis part of its lifetime, you had the pioneers who were building this thing, right and debugging it and running it in production, you know, keeping both pieces when it fell apart. And And over time, that open source project grows into into a commercial entity that is working on that thing and stabilising it and listening to the pharmaceutical customers of the world who say, actually, would you mind putting a bit of TLS on this, right and securing the thing. And then you've got Amazon at the far end of this, I guess MongoDB is a bad example this because Amazon's MongoDB implementation is a MongoDB compatible layer against their own technology. But they've essentially commodified a MongoDB compatible database service that you can now consume without needing to know anything about how you install and configure MongoDB. And this is a this is all part of the tech lifecycle. And the the argument is that if you're spending time pioneering on something that has no business differentiation value for you. Or if you're if you're doing town planning, but you're not selling commodified things, then you're probably spending your time in the wrong places. And there are there are other sort of related concerns, such as in the Pioneer landscape, you should probably be using agile, to run your delivery because it's, it suits the nature of that work. At the town planning and of the equation, you're probably looking at ITIL, and ISO 27,001, and all that sort of the standards following stuff because the pace of change is a little different, you probably doing Prince II project management and that sort of thing, because the Agile model doesn't fit that world. And so making appropriate choices of technology processes, and how you spend your time. And based on where you are in that lifecycle is a really important thing to do. And that's part of part of the consultancy work we do really is to sort of look at, look at what an organisation is doing holistically and sort of try and understand whether they're trying to, you know, fit fit a pioneer problem into a town planner shaped hole.
Aaron Randall 43:16
And when you talk about that pioneer problem in a town planner shaped hole in your conference talk you you talk about this idea of innovation tokens design is that like somewhat related this idea of like being able to afford a send a few pioneers out to try something out?
Jon Topper 43:29
Yeah. So I can't claim to to have coined the innovation token terminology, that's a guy called Dan McKinley, who wrote about it while he was working at Etsy in 2015. And you're looking at me, like impressed with my recall. I wrote this down earlier, because I knew you were going to ask me about this and I didn't want to take credit for someone else's stuff.
Aaron Randall 43:52
You ruined that one
Jon Topper 43:56
Ditto the MongoDB timeline, I've got my talk slides of it. So So Dan, Dan has a blog post that that's, that's called. It's not boring it's powerful, but it's something about boring, choose boring technology. And yeah, the the idea is that you have a limited number of innovation tokens to spend. And so you should spend them wisely and wisely means in the pursuit of the business differentiation, outcome that you are working on. So at Songkick, for example, your innovation tokens were all very well spent in doing the sort of artists search or the the ticket purchasing or the things that made Songkick essentially Songkick. Spending those innovation tokens on I don't know building your own database, which thankfully you didn't do, because I would have shaken someone to death if they'd suggested it. You do need a little bit of fear as a sysadmin. So we said, The yeah, if you've spent that if you've spent that time kind of doing things that that looked like, you know, building infrastructure components, that wouldn't have been a smart move. And that's why it was sensible at the time for Songkick to bring me in as a as a contractor to do those things. Because there was no need for anybody in Songkick to be good at racking hardware and configuring switches and so forth. There's a bunch of sort of hard work that you do up front to get that stuff running. But then later, the the sort of day to day operation stuff, other people can easily do with a bit of, you know, training and assistance. And that's, that's broadly how we think about the cloud stuff today is, most most of the businesses that we work with don't need sort of high end, Amazon solutions, architect professional level individuals in their organization for the whole time. Because once they've stood something up that broadly works, everything else is iterative from there.
Amy Phillips 45:52
That's interesting. It really strikes me, Jon, when you talk about these things that the so much from your years of experience, like you've seen..
Jon Topper 46:01
I look old and tired. Is that what you're saying?
Amy Phillips 46:07
Like, oh, I've come across this thing, or I've seen this as a consultant. I imagine it's easier to you've seen it 20 time before.
Jon Topper 46:14
Yes. Yeah.
Amy Phillips 46:15
That's also true, it kind of like sysadmin started was thinking of almost with the DevOps is, actually if you haven't got years of experience, you haven't seen the database fail in that way three times previously, it's harder to work out what's happening, right? Do you, when you sort of look back to when you started, like, is there like, Is there a tip you wish you wish to have given yourself like 10 years ago as you started out on this sort of journey?
Jon Topper 46:44
Oh, that's a good question. I think 20 years ago Me was a cocky little fucker really, and and was was cockier than his than his own abilities. And, and in a way that sort of set me an ambition to kind of become as good as I thought I was, there's a bit of that. But I think I've, my professional career, in my personal life and everything else, I've been very independence, I've been very sort of like, untrusting of other people and you know, not not feeling like I can rely on people to do a good job. And again, plus one for therapy, that I'm sort of unraveling why that is and making a good understanding of that. I sort of wish I'd done therapy earlier, I guess the advice is like at 21 22, go see a therapist to to work out who you are. Because I'm now today, having spoken to therapists for sort of two or three years now I think, there abouts, I feel like I have a much better awareness of myself and a better understanding of my, my sort of biases, and enough of an understanding of those biases that I can now correct for them in the moment. So if I feel myself kind of going a little bit too independent on something, I can rein myself in and ask for help from somebody, which I would never have done back then. That said, You know, I think being fiercely independent did mean that I had to learn a lot of stuff, right? In order to sort of live up to the image I portrayed for myself. And, and so, you know, this is very sort of psychology, psychiatry stuff, I suppose. But like every strength is has a weakness and vice versa. Right. There's, there's a sort of a flip side to everything. So my, the flip side of my being sort of untrusting of other people and very independent meant that I had to go and figure out how this stuff worked myself and become become good at it. And so I think it's that I think that the message for me of that era would be you need to lean on other people a bit more often. And, and let them in to help. And and I wouldn't have listened to myself, had I been given that advice at 21. Anyway, so I guess that's kind of moot. I think that the when you talk about the the sort of, you know, I have seen a lot of stuff, right, and I think consultancy is definitely a part of that. The other the other sort of part, I think is that I'm, I'm blessed with a really good memory, like, it's not photographic. It's not, it's not a sort of, you know, sort of superhuman thing. I just remember a lot of things, and coupled with a good understanding of the fundamentals. And I think you can solve any problem, right? So I remember a lot of the stuff, and I kind of remember where the Apache config flags that I haven't touched for, you know, 10 years or something. And there are certain kind of failure conditions that I can go, you know what I think it's x and y based on these two other things that I've seen my understanding of the domain lets me sort of cut off a couple of things that it almost certainly isn't and narrow down on the problem pretty quickly. And so it comes across. It comes across as expertise. And I suppose it is in a way but but to me, it's it's about remembering stuff and having a grasp of the fundamentals. I think that's the the thing that if, if anybody's listening, wants to kind of improve at their operations stuff in the technical sense. And a really good understanding of the foundations of how we build this stuff, I think is is the is the key. I, last year, I think, hired hired someone who was kind of, I don't know, 25 or something. And so they'd never run a BGP session on a router, or had no real concept how the DNS system fits together. And there were a couple of outages that happened that year, where I sort of sat down and walked through the kind of how this actually happened and he grabbed his head, and how does any of this work? And at that point, I think he's achieved enlightenment, because actually, none of this stuff should work at all, because it's all built on on foundations of trust, and, you know, assumptions from the 1970s. And the fact that any of the internet is still working in the slightest is nothing short of miraculous to me to be honest.
Amy Phillips 51:33
I, you moved on quite quickly, but like, I really wanted to say like actually thank you for like highlighting how one of the things we've touched upon in previous episodes, is how it can be quite hard and lonely when you're leading. And you can be quite alone. So yeah, I think you're the first person who's really actually been really quite open about actually, everybody should just go and get therapy go and find out who you are which I think
Jon Topper 51:58
particularly men, particularly men, right, I think
Amy Phillips 52:01
I think everybody,
Jon Topper 52:03
probably everybody, I think men need to hear the message more, right? I think I talk very openly about my therapy experience, partly because it's been really helpful to me, and I like sharing things that help us one of the reasons I talk a lot at conferences is that if I've learned something that's really useful to me, I think other people might want to know it as well. And so I talk about having been through therapy, and partly because of that, and partly because I think we still live in a world today where, you know, asking for help as a man is considered weak. And, and that's bullshit, frankly. And there's, I feel much, much stronger as a result of going through therapy and sort of understanding myself. And I guess I'll put this out there, if anybody's listening and would like to talk about going into therapy, and sort of would like to have some fears about that allayed or otherwise, then I'd be more than happy to sit on a on a zoom or a hangout for 30 minutes or so and sort of talk through it because I think it was therapy's seen as a little bit of a bit of a dark art almost right, you sort of you people don't talk about what they talk about in their therapy very often, right? Because it's very personal and, and quite rightly so. And so from, from an outsider's perspective, it looks like you go and sit in a room and get interrogated for an hour at a time. And it's really not, I mean, it can be like, times when it needs to be, but it's, but it's really not like that at all. And it's, it's also it's quite a technical discipline, there are a lot of different types of therapy. And, you know, for someone who is going into therapy for the first time, it sort of doesn't matter, like what the modality is like the the sort of, is this jungian or freudian? Or is it psychosynthesis? So like, all those kind of words, might as well be like, do you like puppet or chef, like, it's a really kind of religious viewpoint for the people who practice therapy, but for the people who are receiving it, like, it sort of doesn't matter. And, but yeah, I'd be more than happy to kind of, I guess, offer some words of comfort to anybody who was thinking about going into therapy, particularly as a as a man. And, and I would seek to encourage them to do that, because I think it's a it's an important powerful thing. And particularly, as you say, like, as a lead or a leader. For me, I own the business. You know, I'm the CEO, the CTO, every everybody, the buck for everything stops with me. And I don't have a one to one with anybody, like, Who am I going to have a one to one with? It turns out that therapy and external coaching are the ways to do that if you're that sort of high up in the organization. And I've sort of I've solved the therapy problem. I've done a little bit of coaching as well. I think there's more coaching in my future around around certain things. But asking for help is the thing that I've learned to do in later life that, I think is the thing that marks me out just having finally grown up along with, you know, the gray hairs and my beard, which are expected of someone who knows Unix as well as I do.
Aaron Randall 55:12
Amazing, Jon, thank you so much for sharing that sentiment that was like, such a profound and an insightful answer. And it's really nice to hear that.
Jon Topper 55:21
I'm glad. I'm glad it comes across that way. It was not my intent. But I do feel pretty strongly about it. So
Aaron Randall 15:27
Yeah, no, it's great. It's really great. And we, I mean, we should wrap here. And that's a really great, great place to stop. Yeah. An incredible high, we have lots of learnings. And I do want to wrap with our usual three quickfire questions. If that's okay.
Jon Topper 55:41
Yes, yeah.
Aaron Randall 55:27
And we like to ask these to all of our Humans+ Tech podcast guests, as a question for you is, what's your top book recommendation?
Jon Topper 55:50
So it would be remiss of me to not to mention the book that I talked about incessantly in every presentation that I give recently, and that is Accelerate, which is the the sort of I can't remember what the subtitle of it is, but it's the it's the DevOps book that has statistics in it. Right? It's the only is the only sort of numerical study, or quantitative study of the things that we've been preaching dogmatically for a decade. And so for me, that's really important, because it allows me to point out some science and say, you know, that the science says you should do this. And I think that the authors are Nicole Forsgren, who is Dr. Nicole Foster, and in fact, who's the primary author, and the person who's done the actual science. Gene Kim and Jez Humble are there as well. I don't know whether that's whether they had major input into the into the stats, but their names are on the front, at least.
Amy Phillips 56:45
Jez Humble had a reasonable input.
Jon Topper 56:48
Right, that's a that's yes. I think that i think that's likely. And and, yeah, I recommend it in in most of my most of my talks, I think the only if I'm going to levy criticism against it, it's that the out the sort of outcomes, that there's a, there's a sort of assumption that you're measuring performance. And the performance is all about rapid delivery of software and low, low error delivery of software, it doesn't make any attempts to draw those parallels with business outcomes or sort of commercial goals. And so it falls a little short there for me, but as a, as a tool to offer a bit of quantative proof for the dogma we've been spouting like, very valuable. Awesome, it's the it's the it's the questions that a quick fire right, not the answers.
Aaron Randall 57:41
Quick fire question number two is who inspires you in tech?
Jon Topper 57:46
Well, that's an interesting one, because I guess my my lone wolf tendencies have led me to kind of not really sort of identify with the idea of being inspired by others, which is interesting and a topic for Tuesday morning, maybe. And but I guess, in the, in the realm of, of what we do today in the cloud industry, I mean, Corey Quinn is someone that I think is, is pretty inspiring, if only for the sheer volume of tweets that he manages to churn out. And so I guess he's inspiring from a sort of personal brand perspective, I think he does it extremely well into the promotion and so forth. And, you know, it's the sort of broad reach that I would quite like, for the consultancy work that we do, but there is no possible way that his approach would work for us. Which is why it's inspiring I think, to go, you found your you found your audience, you found your, your sort of tone, and it's really working. I don't want to sound like that. Thanks.
Aaron Randall 58:54
Nice. And finally, what's the most surprisingly non tech thing about you?
Jon Topper 59:00
That's a different question to the other ones that you've been asking.
Aaron Randall 59:02
You can't prep for this one my friend.
Jon Topper 59:05
I definitely had an answer for the other one that you're asking. Ask that again the most surprisingly non tech thing about me?
Aaron Randall 59:13
Yeah.
Jon Topper 59:14
And so something that that would surprise other people if I told them it is that what you're looking for?
Aaron Randall 59:24
something surprising that is non tech about your life. Oh,
Jon Topper 59:27
yeah. Okay. I mean, I'm polyamorous. I mean, that's, that's not that shouldn't be surprising to anyone who's looked at my Twitter bio, but yeah, I'm a person who enjoys multiple romantic relationships and doesn't really believe in monogamy as a as a concept or, or a structural norm. So I guess that's, that does surprise some people. I'm not as vocal about it as I used to be. I used to sort of because it was new and shiny to me, I guess I was, I was often quite vocal because it would differentiate me and maybe shock people a little bit. But today, it's just a really normal part of my life. So I'm married, my wife and I've been together for over 20 years. My other partner, we've been together for 10 years. And it's all quite normal and boring these days and, and as domestic as you can be in a pandemic and between two houses. But yeah, it's, it can be surprising. I think it's it as a lifestyle choice it's becoming more, if not more popular, then certainly more visible these days. And it's I think, my my ethos in life, career, everything else has always been to sort of question the norm and sort of ask yourself, does this apply to me? Is that a, is this norm a rule that I want to follow? And many times the answer is no. And so I've sort of chosen my own path. And that's, that's one more way in which I've done that. And the other way is, you know, self employment, um, sort of just just a control freak, really just like to
Aaron Randall 01:01:01
Are you saying you're a pioneer, then Jon, in some ways.
Jon Topper 01:01:03
I, yeah, probably. In some ways, yeah, yeah.
Aaron Randall 01:01:10
Awesome. Thanks for sharing. Amazing. And finally, where can people find out more about you?
Jon Topper 01:01:16
Erm so I'm reasonably active on Twitter as @jtopper, where I tend to complain about things mostly these days, to be honest. I'm not as active on Twitter as I as I have been in the past. So Twitter is somewhere I go for tech news, but I don't don't tend to be quite as vocal myself there. I should probably turn that around, actually. And do a bit more kind of thought leadering there.
Aaron Randall 01:01:44
Awesome. Jon, thank you so much for taking the time to talk to us today. It has been so much fun.
Amy Phillips 01:01:50
Thanks so much.
Aaron Randall 01:01:56
We'll be sharing all the links and show notes plus the all important doodle over on HumansPlus.tech. I'm Aaron Randall. This is Amy Phillips and you've been listening to the Humans+Tech podcast.
