התמלול המלא של 274: Managing People Versus Servers with Arpit Mohan

about 12 years now and so, Appsmith is actually officially the third company that we are working on and maybe I think the 30th, or 40th product. I just lost count of number of products that we've built, we've launched, we've failed at miserably for a large part of them and seen a lot of success with some of them like the mobile game in the past. So a lot of startup founders tend to start up once, or twice and then give up and maybe move on to a corporate job. But that's when I realized that if you keep at something, if you keep continue to do something, you start to fractured luck and at some point, lady luck does smile at you. So I think just the startup journey is when I realize that tenacity is something that a lot of people lack. MAE: I love it. Arpit, I keep – are you familiar with Tenacious D? Yeah, absolutely. Tenacious D, a fantastic movie. Love the music especially at the end where he kind of sings with the devil, I think. It's a really, really good song. [laughter] Although, I wouldn't probably tattoo on myself, but yeah, I love the movie and the actor. Jack Black, right? MAE: Yeah, yeah, yeah. I kept wanting to call you Tenacious A so, that's your name for me now. ARPIT: Absolutely. MANDY: So Arpit, we invited you on the show today because you wanted to talk about managing people versus servers and I'm interested in this topic because, because I want to figure out what do you mean by that? ARPIT: Yeah. So as engineers, there's always a lot of mind space and thought that goes into how we write, or manage a code, manage our infrastructure, and manage our server. But managing servers is actually the easy part of up because if a server does not work, or it's not to your liking, you can always reboot the server. You can get a different server and with AWS, or Azure, or any other cloud, it happens with the click of a button. Unfortunately, or fortunately, people are much trickier. You can't just reboot a person and that's what actually makes managing people, working with people, leading people a much more interesting experience and it also is a lot of learning that happens with everybody that you work with because the same person evolves over time. So even if I'm working with you, Mandy, over here, we may be working together for 5 years, but the Mandy of 5 years ago is a very different person from what she is today. An Ubuntu system 16.04, you keep it on for the next 30 years, that is exactly what you will have. So the amount of learning that you have when you're managing a server is constant, or it plateaus up after a while. But the interesting part about people is that there's always something new to learn about your colleague, your partner, or just humans in general. That's what I find very, very interesting about the difference between servers and people and why they might be two slightly different sides of the coin. But I think there is a lot to be learned, or a lot to be derived from engineering principles when we deal with people. For example, there's a lot of literature around how to manage a distributed system. A distributed system is nothing but a cluster of servers, or a lot of servers that form a cohesive unit and operate as one. So when you do a google.com search, you are actually hitting some large cluster of servers posted by Google, but is presented to you as a single Google search. All these servers are operating as a cohesive unit. We can derive a lot of learnings from how a company, or engineers manage these large clusters of servers and how we can cohesively manage a large group of people to act as one towards a common goal, towards a common outcome, and that is something that I find very fascinating. MAE: I agree completely. I love and am fascinated people. And I would add to your list of always new things to learn as also about one's self. Like we too are changing and/or most people don't have a good lock on exactly [chuckles] who they're, what they're doing. So a lot of constantly changing variables is super fun place to be. Have you, Arpit, taken this analogy any steps further of like – and so, there's this system upgrade that we apply, or I don't know, have you explored any deeper into this analogy? ARPIT: Yeah, absolutely. This is something I've thought about a lot and something that I try to practice in our day-to-day job. Appsmith is an open source project. We deal with a lot of people, a lot of contributors, and a lot of community users as well on a daily basis and we are globally distributed across the planet. So a lot of learnings that I've had as a distributed systems engineer, I've tried to apply it to Appsmith the project and to the work that we do on a day-to-day basis. One of the immediate examples is that whenever you have a distributed system, a very important aspect of it is having a consistent language, or an interface where two microservices can talk to each other. So if I have a service, a microservice A and a microservice B, for them to communicate with each other, there is a predefined interface that is well defined. This applies to people as well, that whenever you are interacting with multiple other folks in the team, if you don't have a shared language, you don't have a shared understanding of the topic at hand, the problem at hand, that's when things start to go awry. So one of the first things to actually do, whenever you start working with anybody for that matter, is establish a level, a consistent language and a consistent interface so that both the parties are always on the same page. They're always if I say X, you understand exactly what I mean and it’s not like you have a different version of X in your head as compared to what I have. A lot of times, for humans especially, the illusion of communication is that it has happened and that's what I do whenever a new contributor, or a new person joins the team, that's the first thing that we kind of sit and do is, “Oh, what's our terminology. What's

the level of understanding that we have with each other?” And we just spent the first few days, or weeks just establishing the shared level of understanding with anybody new in the team. It begins with something as basic as having a consistent language. So that's one of the principles. MAE: I suspected that you are going to have some more detail on that. [laughter] I'm so glad I asked and honestly, I would love to hear the outline of how to translate distributed server management to very human focused, human serving management practices. This is super cool. JOHN: Yeah. Actually, before we dive into that, there's one follow-up I'd love to get before we jump off is you have any techniques, or things that you do on a basis to verify that everyone's gotten onto that same page, that the two either individuals, or teams, or groups, or whatever have gotten to that point, or have maintained that sync on their understanding of what they're working on? ARPIT: Yeah, actually, there are a few tactical things, or signals that you pick up on. The first is going back to the basis of all human knowledge, which is the written word. At the end of a particular discussion, or a meeting that you have with somebody, if you share the meeting notes, or minutes of the meeting with the other person. Immediately, the words that are used in the minutes are typically a translation of what the writer has understood, or what the writer is taking away. If there are differences, or discrepancies in the keywords that are being used. For example, let's say, we are talking about an authentication system. In an authentication system, a user role, or a user policy in different systems might be interchangeably used. They might mean the same thing, or different things to you, or me, or whatever, but inherently, if wherever we are talking about roles—like in Google Drive, you have an editor role and a commentator role, and a viewer—versus a policy like a commenter policy, or a viewer policy. So immediately, if the writer is using the word policy, then you know that oh, the keywords that are being used are not the same. That means we're not yet on the same page. That means the writer is probably thinking of something else. They may, or may not be on the same page. We don't know yet. You kind of have to dig deeper and be like, “Oh, why did you use the word policy? What did you actually understand when we talked about the roles when we were talking about?” So the written word is the first thing that I would probably look out for. JOHN: Hmm. ARPIT: The other one is following up where you constantly have, again, it goes back a little bit with distributed systems. So a TCP handshake, the HTTP protocol that we have, a TCP handshake, it's literally a handshake. So there's a SYN and then there's an ACK. The client establishes a connection and the server then acknowledges that the connection has happened. During a conversation, you always try to get an acknowledgement from the

other person that, did you understand what I just said, or am I making sense to you, or is this too complicated? So on and so forth. So you look for other signals like a head nod, a confused expression, wide eyes, or just a verbal confirmation that hey, am I making sense? Like right now, did this make sense to you guys? JOHN: Yeah. It immediately struck me that you're talking about an active listening behavior there that's verifying that the right things are being communicated. But it also struck me that meeting minutes are a larger form of active listening for the meeting and not just for an individual set of exchanges and in between two people. Thinking about it that way is actually really helpful to me because it's something I'm practicing to get better at and lumping that meeting summary under that category makes it feel better to me as a thing that I can do. ARPIT: Yeah. Meeting minutes are typically treated with a little bit of – are given, for lack of a better word, a very step treatment, step sisterly sort of treatment where it's just like, oh, this is what we talked about. But I think just if you go through a bunch of the past X meetings, you'll start to see your trend show up and overall, it represents the shared understanding of the entire group and how that understanding has evolved over a period of time. So we go back and read a lot of fast reviews just to see that oh, how have we evolved in our thought, or in our language. MAE: Another one on the initial and maybe this is going to show up in some of your later examples. I apologize if [chuckles] that's the case, but one thing I've tried to adopt, or encourage to some success is a user manual. Even though a lot of people are still figuring themselves out, generally do have an awareness of here are some ticks I have, here are some things that I respond well to, when I hear this, it makes me think that. Those sorts of translations also can go a long way. So it's very similar to your suggestion, just instead of an interpretation of what just got said, a meta level of it. ARPIT: Yeah, absolutely. And that's still the beauty of interacting with other people is that nobody gives you that user manual and with every person, it's an investigative exercise to figure your version of that person's user manual, or the persona that you are to me, like what me is to Arpit is very different from what me would be to her partner, or to her colleague, or to her parent, for example. So everybody has very different user manuals of me, or of a person and they're very different persons and for everybody that you meet, it's literally an investigation. It's a detective investigation to figure this person out and that's why I absolutely love meeting new people because it presents a new challenge. It presents like oh, what do I know of this person, or what side of this person do I know of, or are being presented to, and how can I deeper that understanding of this persona, or this person? JOHN: Yeah. That's fantastic. So you were about to go into of your overall schema for this. I'd love to hear about the rest of it. MAE: Yeah, yeah. We want to hear! We want to hear! [laughter] ARPIT: Okay. So the other one is whenever you have a group of services together, you have something called a consensus algorithm that is at play. For example, you have the Raft consensus algorithm, or the Paxos algorithm that is used. Different systems like Redis, or Zookeeper, or et cetera. Depending on the system, they choose a consensus algorithm. The one thing that is common across all these consensus algorithms is that there is basically, a leader and a follower, or a main, a node and you have other backup nodes, or follower nodes, which basically translates to, there is always a single source of truth in the system. So regardless of how many people that you are organizing, or working with, there needs to be a single source of truth. That's why I am very anti-anarchy as a governance model because I don't think that works. What works really, really well is a democracy where the backup nodes, or the followers, they elect a leader and when the leader no longer exists, maybe at the end of their term, or et cetera, the backup nodes, or the follower nodes choose to elect another leader from amongst themselves. So one of the followers gets promoted to the post of a leader and this happens in literally every single organization and every single governance model, if you will. The leader plays very critical role because the leader is literally answerable to the external system about the happenings that are happening inside the cluster. So if you try to persist data into a cluster of servers and one of the backup nodes does not perform well, or let's say, it's a rogue server, or a rogue person. You don't hold the server, or the follower node responsible. You actually hold the leader responsible and you say that, “Oh, as a leader, you didn't do your job and ensure that the data got persisted correctly. So I'm going to replace you as a leader because you no longer a good leader.” That is true of teams as well that regardless of what everybody else is doing in the team, if you are leading a team, you are responsible for their outcome. You need to take responsibility for their output and their outcome and act like that. And you are the interface for the team to the external world, because the external world will only talk to you. They're not going to talk to a hundred other people that stand beside you; they'll only talk to you. So that's the other consensus algorithms and how we elect a new leader and that's why democracy rocks. Most successful nations have democracy because it's literally the best consensus that we've come up with. The third one is the concept of independent failures. In a distributed system when you have multiple services doing different parts of the workflow. For example, if I try to initiate a refund flow. Now a refund flow needs to go to

the order system, cancel the order, go to my payment system, initiate a payment refund, and then maybe talk to the notification system and notify the user that the refund is complete. So there are three different subsystems that are at play in order for the refund to actually successfully happen and this is where the concept of independent failures in a distributed system crop up. Wherein, an order system will actually be focused on accomplishing its job and it relies on the notification system to have done its job. It's not like if the notification system fails, we are going to not cancel the order. The order is canceled. So there's this shared sense of trust that a distributed system must take doing its services so that each server does its job properly and independently without necessarily having to rely on another server. This is true of people as well and this is what I tell a lot of folks within the team. The top priority is to get your job done first, without worrying about what somebody else is doing. So once you've got your job done, you've delivered what you needed to deliver, then focus on what everybody else is doing. Don't try to unnecessarily distract yourself and focus on what is X doing, or what is Y doing unless you've completed what your deliverable is, because there are other people in this shared trust model who are relying on you to do your job well. This is again, why high trust teams operate a lot more efficiently and are able to move a lot faster rather than a trustless system, or a trustless team, because of the shared sense of trust and the corollary to this obviously is the independent failure is the concept of failing early. For example, if I expect somebody else to do their job and I'm dependent on their output for me to complete my job successfully, I better have a backup plan. What happens if the other person does not, or the other team does not deliver on time? Do we have a backup? In the engineering world, people use mock data, or mock APIs so that they can get their job done and while they wait for somebody else, or some other team to deliver. Similarly for teams, or human teams as well is invariably most software projects, I think a ridiculously large number of software projects are delayed, which means that there are some teams which are almost always behind schedule. The way to kind of move forward is to always have, or formulate a backup plan for yourself, your team so that even if the other team does not deliver on time, you are still able to move forward and still have your team's output, or your personal output ready and deliverable so that you can then quickly integrate with the other team once they've given, or they've delivered their output. So you shouldn't be beholden to them and say that, “Oh, team X and delivered. So therefore, I'm stuck.” That's not an excuse. That's how we, again, large teams work well with each other. By the way, as an interesting aside, little bit tangential is something that I learned from airplane engines, or seeing how airplane engines are maintained. It's a different engineering discipline. The beauty of an airplane engine, if you ever kind of look at it, is they're geared for quick maintenance. When a flight lands, because there's a dollar cost attached to how long the aircraft is parked in the bay and the maintenance needs to happen really, really quickly, every part that can potentially fail is literally within one arm's length inside the engine of a human. So they can literally put their arm in, reach for a part, take it out, and replace it for maintenance. The deep maintenance only happens every once in a while. My learning from this was for teams is that part, or everything that can potentially fail in the entire ecosystem should literally be within one arm's length of us as a team being able to change it, or fix it. So instead of having deep processes and deep overarching fixes, as a team, always try to focus on, it's called inversion of control, or dependency, is inverting that control and saying that, “Oh, if there's something that's changing a lot, how do we ensure that we can actually change something really, really quickly and adapt to the situation versus having to necessarily fall back to large overarching code refactors, or large overarching processes that need to be hauled over versus, “Hey, what's the quick thing that we can do, what's the smallest unit of work that we can do in order to actually improve our entire system, or our process as a team”? So that was my takeaway from literally sitting in tons of flights and seeing the maintenance happen outside. JOHN: So it sounds to me like microservices are a part of that because you've got sort of smaller pieces of code that you don't have to go 15 class layers deep in defined where the thing is going wrong on You have the small encapsulated service that it's easy to get to in metaphorically. ARPIT: Yeah, absolutely. And that's why if you look at frameworks like Spring, or I think React also, I think does some parts of this is the whole dependency injection and in version of control where it surfaces. Even if there is a class that is like 15 layers deep, they actually allow you to expose it at the top level so you can just swap out your class at the top which again, sort of maps to what you're seeing in the microservices also. That you can have a cluster of lots of microservices talking in a mesh network to each other a lot, but you’re literally one git push away from changing, or improving something in one microservice, or one workflow. JOHN: Do you have anything in the same toolbox that helps manage once you have a large collection of microservices with their complex interactions, then you get that sort of next order complexity arises out of the system with emergent behavior and whatnot? Do you have anything that helps you, or the team deal with that layer of complexity? ARPIT: Sort-ish of, yes. I think this is something

that was actually popularized by Spotify, the Spotify Squads, which is essentially a source code, or as an Appsmith, we call them quads, is a self-contained unit of frontend engineers, mobile engineers, backend engineers, product managers, designers, and QAs put together so that they are able to accomplish a unit of work, or they're able to take a feature from idea to production largely on their own without necessarily having to talk to other people. So if you have a lot of microservices, or a lot of different services that are talking to each other in a complex mesh network, one of the ways basically follow Conway’s law and divide the entire team, or your group of people as well to be responsible for a smaller set of microservices. Sorry, you're familiar with Conway law, right? MAE: I was going to say, I know a lot of people have done Conway's game of life and that, but in case there are people who didn't, I was going to ask if you'd be willing to go into it more. ARPIT: Yeah. Okay. So Conway’s law is actually very interesting. What Conway’s law states is that if you want to find out about the organization hierarchy, or the organization structure of a company, you actually don't need to talk to the people. You actually go look at the code that they've written, or produced because the code is a representation of the organization structure. So if you see a code base that has a lot of microservices with a lot of interfaces between them, then you can be rest assured that the entire, this company, or this organization is organized into a set of smaller teams that are communicating with each other through a shared interface, or through a shared understanding, again. On the other hand, if you see a very monolithic code base with classes randomly talking to each other, or randomly imported, or pulled into each other, then you can be rest assured that this is a much more fluid organization. It's a more cohesive large block where people are just talking to each other a lot more. A very good example of this is Jeff Bezos back in the day. This is early, early days of AWS. There's this famous memo of his wherein he went in and he is literally told everybody in AWS, or Amazon, the two large pizza rule and he said, “No team is going to be larger than what two large pieces can feed,” which essentially limited it to anywhere between 10 to 20 people, depending on their appetite. So this ended up with AWS where now every service in AWS, whether it's EC2, IAM, S3, et cetera, they are actually smaller teams that communicate with each other only via the API and he said that, “Oh, don't you dare talk to this other team without an API.” Eventually, the organization reformed itself in order to meet that code structure that Jeff Bezos wanted. So Jeff Bezos didn't say, “Oh, I'm going to change the organization structure.” He basically wanted the code, or the services in such a manner and he basically sort of – you move one lever and the other one will automatically move. So if you want to make a code change, change your teams, or you change your teams and you'll change the code, whichever you think is the lesser of the evil, or easier thing to do and just do that because the other one will automatically happen. That's the Conway’s law. So to your point about the microservices is again, organize the people and the services will automatically happen where one team handles two, or three services and just get those two, three services running with their uptime with 5 nines, 6 nines, whatever you want. MAE: I have to admit, Arpit when I read some of this description, I have this very visceral reaction to people trying to treat humans like systems and vice versa like, there's certainly things to be learned and applied across, but this is the best most humane human application of these things that I have ever heard and I'm very excited about it. Especially for how you opened up in saying your opener, I should say it more clearly, was about how people are complex and it's hard and it's more of a complexity than figuring out code and for many reasons and layers. So seeing a self-learning algorithm application of how to deal with human systems, love it. ARPIT: Yeah. No, no, absolutely and most of these things are cross-disciplinary in nature and this basically stems from the idea that the history of software is the history of teams. There is no good, or great software that is ever built by a single person, it's always built by teams of people. So it's important that the code, or the server is just the tool and you never hear, I don't know, like da Vinci ever praise his paint brush for Mona Lisa, right? The paint brush is just a tool. That's why I think focusing on the team and how teams collaborate, or how people collaborate with each other is how we will get to better and better software, to better abstraction level, layers, et cetera. Better team organization, or better people organization is how we are going to get to better software. But that in no way means that we can't learn from our engineering disciplines, or what other people have done, whether it's the aircraft industry, or what F-1 is doing to make their cars faster, or what surgeons are doing to make surgeon be safer and apply them and their processes and principles to actually running our own software teams, or tech teams itself. MAE: I have thought often about how the energy with which we build things, it does show up not only structurally, but just where it finds itself directionally. So to put that into practice is pretty cool. You're reminding me also of Jess Kerr’s RubyConf keynote, I think it's 2019. It's a collective problem solving and she's come up with this term “somathesis” and I think that you would really love her work and anybody enjoying hearing about innovation and team dynamics, and how to apply our own discipline to the human aspects back so that there really is two-way communication. We really do have an API

with what we're building to ourselves. ARPIT: Nice. That sounds like a very interesting – I haven't heard her keynote yet, but I'm definitely going to give it a watch after this show. So thank you so much for the pointer. The last thing that I wanted to mention as well was when we look at again distributed systems and clusters of server, the reliability and uptime is almost always given a higher preference to simply speed, or speed of response, or speed of your SLA on how quickly did the service respond to my ACP request, or TCP request. That correlates again, very, very highly with teams and people as well, is that regardless of how good coder you are, a good an artist you are, or et cetera, if you are unreliable, or your uptime so to speak is not really there, it's very hard again, to establish that human trust. There's this this book I was reading about how SEAL teams operate because the Navy SEALs are literally the highest functioning team that exists out there and they operate in some very, very adverse conditions and there was this book about how do seals organize themselves and how do they operate. The way the SEAL team members select who's going to be on their team is where they basically draw a quadrant. If you have performance on one axis and trust on the other axis, how much do I trust this person in the middle of a firefight versus performance wise how good of a sharpshooter are you, or something like that. They'll prefer somebody who is of mediocre performance, but higher trust over somebody who is of higher performance and mediocre trust, or low trust. Because at the end of the day, when you're in an adverse situation, or when things go south, what you really want next to you is not the best sharpshooter, but what you want is somebody you can actually trust and software teams are a far cry from what the Navy SEALs do. We have very, very cushy jobs and thank God for it. But you don't want to go into a meeting with a customer, or you're presenting to your CEO in the all-hands and you have your partner who throws you under the bus when something doesn't work. There's a demo effect and things don't work and you don't want to be thrown under the bus in front of everybody. So what you'd rather do is work with somebody who is more reliable, who has “greater uptime”, again for a person, it's how quickly are they responding to you, do they communicate back that hey, I'm getting delayed on X, can we push this meeting forward? Even something as small as, “Hey, I'm running late by 5 minutes on this meeting. Can we push the meeting by 5 minutes, or 10 minutes?” That's a very, very, a small signal of reliability and trust and are invariably the people, correlation, or causation, who will actually end up getting much better performance reviews. They're the ones actually get promoted because again, consensus algorithm, they are the ones who will get elected to be the next leader. They're the people that the team trusts. So what do you want to do in any team is, if you are an engineer in a team, or a new engineer in a team, the first thing to do is not show off how good a coder you are, or how good your code is, but what you want to actually do when you're a new person in a group is establish trust. Like, can you trust me as a human? We have a lizard brain right from Neanderthals to homo sapiens to whatever is that we'd much rather be with people that we trust more than anything else in this world. Reliability and uptime and how do you be the reliable person that, “Oh, I've told Arpit to do X, we know that X will be delivered” so they can then move on and live their lives. So you want to be that person in the team. MAE: Yes! Everyone who's about to be, or is new to a team. We'll hear you say that one again. Arpit, what would you do? What would you want? ARPIT: Build trust, establish trust, forget everything else. JOHN: Definitely. MANDY: This seems like a great time to move on to reflections. Who'd like to start us off? JOHN: I can go first. I've got two things that are takeaways. One is the realization I had earlier about how meeting minutes and clear communication after some group conversation is a form of active listening and ensuring that communication has happened the way you thought it did, which is great in increasing the fidelity of the understanding throughout the team. I love that. And then the other one was, I've been familiar with Conway's law for a long time, of course. It's been around for since 70s, I think at least, maybe even the 60s. But the way you phrased it, where you were saying that you can start with the structure of the code and have that influence the teams, or you could start with the teams and have that influence the structure of the code. Just that idea that you could say, “Okay, well, I kind of think the architecture's going to end up looking like this. That means I'm going to build these teams that do these things and then that will just naturally flow out of the fact that the teams are structured that way is such a fascinating flip to the normal way of thinking about how software is going to get built in multi-team environment. So that's definitely going to stick with me for a while. ARPIT: Awesome. MAE: John, you stole mine! [laughter] Those are definitely on my list, but so many of them. I'm definitely going to re-listen to this and Arpit, I want you to write a book, or I don't know, make a movie. This is great! I love it. So I think the only thing I could add to what you were saying, John, is I loved thinking about trust in terms of reliability and uptime. [laughter] That was really well done. ARPIT: Yeah. Thank you. Yeah, I really, really want to write a book at some point. That's on my bucket list to kind of do short eBook someday. I hope I can do that. For me, the takeaway is literally this RubyConf talk that you shared because to be honest, I haven't heard a lot about people speaking about the confluence between engineering disciplines and team management principles and human disciplines itself. So I think that's definitely a takeaway for me is to listen to what she said and learn from what she's talking about in the keynote. MAE: And I've been saving up a reading list, too. So I’ll write you after this and we can compare some notes. Also, when you write the book, Arpit, you need to sign it as Tenacious A. [laughter] ARPIT: Oh yeah, absolutely, absolutely. At that point, I'll probably tattoo it. [laughter] Not just sign it. [laughs] MANDY: And then you'll have to come back on the show to talk about it again. JOHN: Yes, yes. ARPIT: Oh, absolutely, absolutely. This has been a really, really fun, very free-flowing, very casual conversation. So thank you so much to all of you for doing that and for having me on the show. It has been a true pleasure. MANDY: Thank you. I just want to say my reflection was just in the beginning when you described your superpower being tenacity, I haven’t thought of that word in such a long time. [laughter] And it’s such a great word. Determination and tenacity, I love the words so that was a great takeaway for me. But again, thank you so much for coming on the show. It was wonderful to have you and we will see everyone next week.